In the previous article, everything happened in the inner loop. We connected a laptop to a live Kubernetes cluster, had a coding agent build and validate a feature against real dependencies, and debugged a cross-service bug with traffic recording and local overrides. All of that was you (or your agent), working from your laptop before the commit.
But code ships from a pull request. The moment you open a PR, you want the same guarantees applied automatically, for every PR and every developer on the team.
The outer loop applies those same guarantees after you push. CI creates a sandbox for every GitHub PR, with an optional live preview for reviewers. Even traditionally scary PRs with database schema changes get a disposable database branch instead of a dedicated environment. Tests run inside the cluster against the PR's code, and closing or merging the PR cleans everything up.
From the inner loop to the outer loop
The inner loop ends at the commit; the outer loop starts when the pull request opens.
Rendering diagram…
There are three building blocks that make the outer loop work at scale.
Sandbox template
A sandbox template is a sandbox spec with variables, checked into the repo right next to the code. CI fills in the name, image, and PR number at apply time.
Scope the sandbox to the service the PR changed. Other microservices, databases, and message queues stay shared from the baseline. With one changed service per PR, fifty sandboxes add about fifty pods to one cluster. Fifty full environments duplicate the entire stack fifty times.
This walkthrough focuses on one changed service, but a sandbox can contain multiple forks. In a monorepo, CI can detect which services changed, build each image, and include all affected services in the same PR sandbox. The cost then follows the number of changed services, not the size of the full environment.
The GitHub Actions lifecycle
One GitHub Actions workflow owns the PR lifecycle. On open, synchronize, or reopen, it builds the PR image, creates or updates the sandbox, runs the in-cluster test, and posts one sticky comment with the preview details. On close, it deletes the PR-derived sandbox and updates that comment. There is no separate GitHub App in this setup.
Jobs
Jobs put the tests inside the cluster, where they can use normal Kubernetes DNS names with sandbox routing applied. A Job combines a script, a pool of runner pods in your cluster, and a routing context that adds the sandbox's routing key to every request.
Prerequisites
This builds on the same environment as the previous article: HotROD in the
hotrod namespace, the Signadot operator installed, and the CLI authenticated. On top of that, you'll need:- A Signadot account and the Signadot CLI (
signadot --version) - A GitHub account and a fork of the HotROD demo repository you control
- A Docker Hub account with a personal access token (CI pushes a per-PR image)
- For the database section, a Neon account,
neonctlinstalled and authenticated, andpsql jqfor inspecting output
Part 1: A sandbox for every pull request
Open a PR and a sandbox running that PR's code appears in the shared cluster, with its status and routing details attached to the PR for reviewers. A preview URL already includes the routing context, but reviewers can instead activate the sandbox with the browser extension. The complete workflow, sandbox template, Job template, and Job Runner Group spec are checked into the repository.
File 1: the sandbox template
The HotROD repo already ships the sandbox template at
.signadot/sbx-gh-template.yaml:name: "@{name}"
spec:
description: PR sandbox for the HotROD route service
cluster: "@{cluster}"
labels:
signadot/github-repo: "@{github-repo}"
signadot/github-pull-request: "@{github-pr}"
branch: "@{branch}"
ttl:
duration: 2d
offsetFrom: updatedAt
forks:
- forkOf:
kind: Deployment
name: route
namespace: "@{namespace}"
customizations:
images:
- container: hotrod
image: "@{image}"
defaultRouteGroup:
endpoints:
- name: hotrod
target: "http://frontend.@{namespace}.svc:8080"
The sandbox spec forks the route deployment and overrides its image. The
@{...} placeholders are template variables that CI fills in, including the cluster and namespace.File 2: the GitHub Actions workflow
The complete GitHub Actions workflow is checked in at
.github/workflows/sandbox.yaml. It reads the Signadot organization, API key, and Docker Hub credentials from repository secrets instead of hardcoding them. Its lifecycle is:on:
pull_request:
types: [opened, synchronize, reopened, closed]
permissions:
contents: read
pull-requests: write
env:
SANDBOX_NAME: hotrod-pr-${{ github.event.pull_request.number }}
jobs:
create-or-update:
if: github.event.action != 'closed'
# Build route-pr-<sha>, apply sandbox, run Job, post sticky comment.
cleanup:
if: github.event.action == 'closed'
# Run: signadot sandbox delete "${SANDBOX_NAME}"
The workflow installs the Signadot CLI, builds and pushes a per-commit image, applies the sandbox template, submits the route Job, and posts the result. The sandbox runs the per-commit image built from this PR.
The image tag includes the commit SHA, but the sandbox name comes from the PR number. Every push updates the same sandbox for the lifetime of the PR instead of creating another sandbox for every commit.
hotrod-pr-<number> identifies the long-lived PR sandbox, while route-pr-<sha> identifies the specific route-service image running inside it.Sandbox names cap out at 30 characters. This repository fits comfortably as
hotrod-pr-<number>. For longer repository names, truncate the readable prefix and append a short deterministic hash so the name remains stable and collision-resistant for the lifetime of the PR.The Signadot API key and registry credentials are repository secrets. The cluster and HotROD namespace are repository variables. The workflow itself is the integration; no separate App installation is required.
Open a real PR
Push a small, visible change to a branch (I tweaked the route service response), then run:
gh pr create --head demo/route-tweak \
--title "Tweak route service response" \
--body "Demo PR for sandbox-per-PR workflow"
The workflow kicks off and builds the image. The Signadot dashboard then shows a sandbox named
hotrod-pr-<number>, with a forked route service running the commit-specific route-pr-<sha> image and wired into the shared cluster. Fifty open PRs create fifty sandboxes, each about one pod.The workflow follows this lifecycle:
Rendering diagram…
Back on the PR page, the workflow's sticky comment reports the sandbox name, commit-specific image, preview URL, routing key, and passing Job. Whether reviewers use the preview or activate the sandbox with the browser extension, they see this PR's version of the app running against real upstream and downstream services.
Cleanup is automatic
Once you merge or close the PR, the workflow receives the
closed event and runs signadot sandbox delete for the stable PR-derived name. It should disappear from the dashboard shortly afterward; the exact delay depends on event delivery and cleanup time. The workflow updates the same sticky comment to show that cleanup completed.Database isolation with Neon branching
So far, every sandbox has shared the baseline database, which works for most changes. A schema-changing PR can't run a migration there without breaking the baseline and every other sandbox. Spinning up a full environment avoids the collision but duplicates the rest of the stack.
For this example, I'll use Neon, a serverless Postgres service with copy-on-write branching that's similar to Git branches, but for your database.
A Neon branch is a fully isolated Postgres environment. It starts with the parent's schema and data, along with its databases, roles, and extensions. Changes made within the branch remain isolated from the parent and every other branch. Branch creation takes seconds, and the branch shares storage with its parent until you write to it. A Signadot resource plugin can create a branch when a sandbox starts and destroy it when the sandbox is deleted.
Rendering diagram…
Let's build a small service to demonstrate how this works. The service is a deliberately tiny users API with about 50 lines of Express, one table, and three endpoints. It reads its Postgres connection string from a single
DATABASE_URL environment variable, which the sandbox will override later. The Signadot Neon branching example contains a complete version of the service, Docker build, manifests, resource plugin, and sandbox spec; the snippets below keep the project fixed to main and neondb to make the lifecycle easier to follow.One table, two seed rows:
-- schema.sql
CREATE TABLE IF NOT EXISTS users (
id SERIAL PRIMARY KEY,
name TEXT NOT NULL,
email TEXT UNIQUE NOT NULL
);
INSERT INTO users (name, email) VALUES
('Ada Lovelace', 'ada@example.com'),
('Grace Hopper', 'grace@example.com')
ON CONFLICT (email) DO NOTHING;
Create a Neon project and load the schema into its main branch:
neonctl projects create --name users-demo
psql "$(neonctl connection-string main \
--project-id <project-id> \
--database-name neondb)" \
-f schema.sql
Note the project ID from the output.
Deploy it to the cluster
The Kubernetes side has one Deployment and one Service:
# k8s/users-service.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: users-service
namespace: default
spec:
replicas: 1
selector:
matchLabels:
app: users-service
template:
metadata:
labels:
app: users-service
annotations:
sidecar.signadot.com/inject: "true"
spec:
containers:
- name: users-service
image: users-service:demo
imagePullPolicy: Never # The image is side-loaded into kind.
ports:
- containerPort: 3000
env:
- name: DATABASE_URL
valueFrom:
secretKeyRef:
name: users-db-credentials
key: DATABASE_URL
---
apiVersion: v1
kind: Service
metadata:
name: users-service
namespace: default
spec:
selector:
app: users-service
ports:
- port: 3000
targetPort: 3000
Build the image and load it into the cluster. I'm using kind, but you can alternatively push it to a registry. Next, create the two secrets and deploy:
docker build -t users-service:demo .
kind load docker-image users-service:demo --name <kind-cluster-name>
# The baseline's connection string (Neon main branch).
kubectl create secret generic users-db-credentials \
--from-literal=DATABASE_URL="$(neonctl connection-string main \
--project-id <project-id> \
--database-name neondb)"
# Neon API key for the plugin. Plugin runners execute in the signadot namespace.
kubectl -n signadot create secret generic neon-api-credentials \
--from-literal=NEON_API_KEY=<your-neon-api-key>
kubectl apply -f k8s/users-service.yaml
kubectl get pods -l app=users-service # 2/2 Running: app + devmesh sidecar
The resource plugin
A resource plugin has two lifecycle scripts with typed inputs and outputs. One runs when a sandbox that requests the resource is created, and one runs when it's deleted. The full plugin is:
# neon-branch-plugin.yaml
name: neon-branch
spec:
description: Creates and deletes a Neon database branch per sandbox
runner:
image: node:20-alpine
namespace: signadot
podTemplateOverlay: |
spec:
containers:
- name: main
env:
- name: NEON_API_KEY
valueFrom:
secretKeyRef:
name: neon-api-credentials
key: NEON_API_KEY
create:
- name: createbranch
inputs:
- name: project-id
valueFromSandbox: true
as:
env: NEON_PROJECT_ID
script: |
#!/bin/sh
set -e
npm install -g neonctl
# Branch named after the sandbox (Neon branch names: no hyphens).
SAFE_NAME=$(echo "${SIGNADOT_SANDBOX_NAME}" | tr -d '-')
BRANCH_NAME="sandbox${SAFE_NAME}"
neonctl branches create \
--project-id "${NEON_PROJECT_ID}" \
--name "${BRANCH_NAME}" \
--parent main
CONNECTION_STRING=$(neonctl connection-string "${BRANCH_NAME}" \
--project-id "${NEON_PROJECT_ID}" \
--database-name neondb)
mkdir -p /outputs
echo -n "${BRANCH_NAME}" > /outputs/branch-name
echo -n "${CONNECTION_STRING}" > /outputs/connection-string
outputs:
- name: branch-name
valueFromPath: /outputs/branch-name
- name: connection-string
valueFromPath: /outputs/connection-string
delete:
- name: deletebranch
inputs:
- name: project-id
valueFromSandbox: true
as:
env: NEON_PROJECT_ID
- name: branch-name
valueFromStep:
name: createbranch
output: branch-name
as:
env: BRANCH_NAME
script: |
#!/bin/sh
set -e
npm install -g neonctl
neonctl branches delete "${BRANCH_NAME}" --project-id "${NEON_PROJECT_ID}"
The runner is a plain
node:20-alpine pod in the signadot namespace with the Neon API key injected from the secret we created. The create step takes the project ID from the sandbox spec, names a branch after the sandbox (Signadot injects SIGNADOT_SANDBOX_NAME automatically), creates it with neonctl, and writes the branch name and connection string to files under /outputs. Writing those files publishes the step outputs. The delete step reads the branch name from the create step's output via valueFromStep and removes it.Install it:
signadot resourceplugin apply -f neon-branch-plugin.yaml
The sandbox spec
The sandbox spec requests the branch and injects its connection string into the fork:
# users-sandbox.yaml
name: "@{sandbox-name}"
spec:
description: users-service fork with an isolated Neon database branch
cluster: "@{cluster}"
resources:
- name: usersDb
plugin: neon-branch
params:
project-id: "@{neon-project-id}"
forks:
- forkOf:
kind: Deployment
namespace: default
name: users-service
customizations:
env:
- name: DATABASE_URL
valueFrom:
resource:
name: usersDb
outputKey: createbranch.connection-string
defaultRouteGroup:
endpoints:
- name: users-api
target: http://users-service.default.svc:3000
The
resources block requests a neon-branch from the plugin, so the branch gets created before the fork starts. The fork's DATABASE_URL comes from the plugin's createbranch.connection-string output, overriding the secret-backed value used by the baseline. The application still reads the same environment variable; only the value changes.Create the sandbox and watch the branch appear
signadot sandbox apply -f users-sandbox.yaml \
--set sandbox-name=schema-change \
--set cluster=<your-cluster-name> \
--set neon-project-id=<project-id>
Immediately afterward:
neonctl branches list --project-id <project-id>
Within seconds, the
sandboxschemachange branch exists as a copy-on-write clone with production-shaped data, without waiting for a nightly dump restore. In the dashboard, the sandbox's Resources tab shows the plugin run steps and their outputs.Prove the isolation
Write a user through the sandbox. The preview URL routes to the forked service, which talks to the branch. Grab an API key from the dashboard:
curl -s -X POST \
-H "signadot-api-key: <key>" \
-H "Content-Type: application/json" \
-d '{"name":"Sandbox-Only User","email":"sandbox@example.com"}' \
"https://users-api--schema-change.preview.signadot.com/users" | jq
Read it back through the sandbox to confirm it's there:
curl -s \
-H "signadot-api-key: <key>" \
"https://users-api--schema-change.preview.signadot.com/users" | jq
Now read from the baseline via
signadot local connect:curl -s "http://users-service.default.svc:3000/users" | jq
It's not there. Same cluster, same service name, two completely isolated databases. The PR can run its migration, mutate data, or drop tables; the baseline never notices.
Cleanup is automatic here too
signadot sandbox delete schema-change
neonctl branches list --project-id <project-id>
The plugin's delete step removed the branch, leaving just
main. Nothing to remember and no cost leaking from forgotten branches.Jobs running tests inside the cluster
CI created the sandbox from outside the cluster. The tests still need to run inside, use normal service names, and carry the sandbox routing context. Signadot Jobs handle that.
Create a Job Runner Group
A Job Runner Group is a pool of runner pods in your own cluster. You pick the image, namespace, and number of pods. Tests run on your infrastructure, next to your services, instead of on a GitHub-hosted VM reaching in from the internet. The repository includes the complete Job Runner Group spec.
# .signadot/jobrunnergroup.yaml
name: hotrod-tests
spec:
cluster: "@{cluster}"
namespace: signadot-jobs
image: golang:1.22-bookworm
jobTimeout: 15m
scaling:
manual:
desiredPods: 1
kubectl create namespace signadot-jobs
signadot jobrunnergroup apply \
--set cluster=<your-cluster-name> \
-f .signadot/jobrunnergroup.yaml
signadot jobrunnergroup list # Wait until ready.
The Job template
The repository also includes the Job template at
.signadot/job-route-test.yaml:spec:
namePrefix: route-api-test-
runnerGroup: hotrod-tests
routingContext:
sandbox: "@{sandbox}"
script: |
#!/bin/bash
set -euo pipefail
git clone --depth 1 --single-branch \
--branch "@{branch}" \
"https://github.com/@{repo}.git" \
/tmp/hotrod
cd /tmp/hotrod
TEST_ROUTE_ADDR="route.@{namespace}.svc:8083" \
go test -run '^TestRouteClient$' -v ./services/route 2>&1 \
| tee /tmp/route-test.log
grep -q -- '--- PASS: TestRouteClient' /tmp/route-test.log
echo "PASS: route gRPC test reached sandbox @{sandbox}"
uploadArtifact:
- path: /tmp/route-test.log
routingContext.sandbox adds the sandbox routing key to requests from the Job. HotROD's existing Go test connects to the ordinary gRPC address route.<namespace>.svc:8083, and routing sends that connection to the forked version. The test does not need a per-PR hostname.Submit and attach
With a target sandbox running (reuse a PR sandbox from Part 1, or create one from the template), run:
signadot job submit -f .signadot/job-route-test.yaml \
--set sandbox=hotrod-pr-<number> \
--set branch=demo/route-tweak \
--set repo=<owner>/hotrod \
--set namespace=<hotrod-namespace> \
--attach
With
--attach, the logs stream straight to your terminal until the Job finishes, and the Job's exit code propagates to the caller. Logs and artifacts land in the dashboard, attached to the run. The test's captured response body is available there as a downloadable artifact.signadot job list
Slot it into CI
Add this step to the
create-or-update job from earlier:- name: Run API test against sandbox
run: |
signadot job submit --attach \
--set sandbox=${SANDBOX_NAME} \
--set branch=${{ github.head_ref }} \
--set repo=${{ github.repository }} \
--set namespace=${HOTROD_NAMESPACE} \
-f .signadot/job-route-test.yaml
signadot job submit --attach propagates the Job's exit code to GitHub Actions, so a failed in-cluster test fails the PR check. When the PR closes, the workflow deletes the sandbox, and resource plugins delete any database branches they created.Conclusion
Every PR now gets the same live-cluster validation as the inner loop: one stable sandbox, a commit-specific image, in-cluster tests, and a preview reviewers can open before merge. Schema-changing PRs get their own Neon branch, so migrations and test data never touch the baseline. With one changed service per PR, fifty open PRs add roughly fifty pods to the shared cluster; closing a PR removes its pod and database branch.
