Ephemeral Preview Environments: Namespace-per-PR

A shared staging cluster with a single mutable branch deployed on merge is the single largest source of false-negative QA sign-offs in mid-sized engineering organisations. Two feature branches touching the same feature flag, the same database migration, or the same cache key will silently overwrite each other’s state, and nobody notices until a release manager asks why staging is “broken again”. The fix that has become standard practice across platform teams is ephemeral preview environments: a fully isolated, disposable deployment per pull request, provisioned automatically on push and torn down automatically on merge or close. The architectural challenge is not spinning up a namespace — that part is trivial. The challenge is doing it without leaking cluster capacity, without exhausting your DNS and certificate issuance quotas, and without turning your CI pipeline into a distributed systems problem in its own right.

#The Shared Staging Bottleneck

In a conventional pipeline, every merge to a long-lived integration branch triggers a redeploy of a single staging namespace. This works acceptably at low merge velocity. Past roughly 15–20 merges per day across a team of more than a dozen engineers, the staging environment becomes a queue, not an environment. QA engineers wait for a deploy slot. Two PRs with conflicting schema migrations corrupt each other’s state. Load testing against staging produces noise because an unrelated deploy just rolled a pod mid-run.

Ephemeral preview environments remove the queue entirely by giving every open pull request its own Kubernetes namespace, its own subdomain, and its own resource envelope. The trade-off is operational complexity: you are now managing N concurrent environments instead of one, and N is a function of how many PRs are open at any given time, not a number you control directly.

#Architectural Breakdown: Namespace-per-PR

The core pattern rests on four cooperating subsystems:

• A GitOps controller (Argo CD ApplicationSet or Flux with a Git generator) that watches for open pull requests and materialises a Kubernetes Application object per PR.
• A templating layer (Kustomize overlays or Helm value overrides) that parameterises image tags, subdomain, and resource limits per PR number.
• An ingress and TLS layer using wildcard DNS and a wildcard certificate, so no per-PR certificate issuance is required against a rate-limited CA.
• A TTL reaper that deletes namespaces on PR close, on merge, or after a maximum lifetime, independent of whether the webhook that should have triggered cleanup actually fired.

The reason this differs meaningfully from a standard architectural pattern for multi-tenant SaaS provisioning is lifecycle volatility. A SaaS tenant namespace lives for years. An ephemeral preview environment lives for a median of 18–36 hours and must be provisioned and destroyed with near-zero manual intervention, at a rate that can spike to dozens of concurrent creates during a release freeze followed by a merge storm.

#Why ApplicationSet Over a Bespoke Controller

Teams frequently attempt to hand-roll a controller that listens to GitHub webhooks and calls the Kubernetes API directly. This works until the webhook delivery fails silently (GitHub’s webhook retry window is finite) and a namespace is orphaned. Argo CD’s ApplicationSet Pull Request generator instead polls the SCM API on an interval, reconciling desired state against actual state continuously. This shifts the ephemeral preview environments lifecycle from an event-driven model with silent failure modes to a level-triggered reconciliation

#Implementation Logic

The provisioning flow, end to end, looks like this:

1. A developer opens or updates a pull request against the main branch.
2. The ApplicationSet controller polls the SCM API, detects the open PR, and generates a templated Application manifest using the PR number as the primary key.
3. Argo CD syncs the manifest, creating a namespace labelled preview.kbytech.io/pr=<number> and preview.kbytech.io/ttl-expiry=<unix-timestamp>.
4. A ResourceQuota and LimitRange are applied to the namespace before workload manifests, preventing an oversized preview environment from starving the node pool.
5. An ingress resource is created using a wildcard host pattern, routing pr-<number>.preview.kbytech.io to the namespace’s service, terminating TLS against a pre-issued wildcard certificate.
6. On PR close or merge, the ApplicationSet generator no longer lists the PR, and Argo CD’s pruning behaviour deletes the associated resources.
7. A separate CronJob acts as a backstop, deleting any namespace whose ttl-expiry label has passed, regardless of Argo CD’s reconciliation state.

#Code and Configuration

The ApplicationSet definition below generates one Argo CD Application per open pull request, templating the namespace and subdomain from the PR number and branch SHA:

1apiVersion: argoproj.io/v1alpha1
2kind: ApplicationSet
3metadata:
4  name: preview-environments
5  namespace: argocd
6spec:
7  generators:
8  - pullRequest:
9      github:
10        owner: kby-technologies
11        repo: platform-app
12        tokenRef:
13          secretName: github-pr-token
14          key: token
15      requeueAfterSeconds: 60
16  template:
17    metadata:
18      name: 'preview-pr-{{number}}'
19    spec:
20      project: previews
21      source:
22        repoURL: https://github.com/kby-technologies/platform-app.git
23        targetRevision: '{{head_sha}}'
24        path: deploy/overlays/preview
25        kustomize:
26          namePrefix: 'pr-{{number}}-'
27      destination:
28        server: https://kubernetes.default.svc
29        namespace: 'preview-pr-{{number}}'
30      syncPolicy:
31        automated:
32          prune: true
33          selfHeal: true
34        syncOptions:
35        - CreateNamespace=true

Every ephemeral preview environment namespace must carry a hard resource ceiling. Without this, a single memory-leaking PR build can starve the entire preview node pool:

1apiVersion: v1
2kind: ResourceQuota
3metadata:
4  name: pr-preview-quota
5spec:
6  hard:
7    requests.cpu: "2"
8    requests.memory: 4Gi
9    limits.cpu: "4"
10    limits.memory: 8Gi
11    pods: "12"
12    services.loadbalancers: "0"

The TTL reaper runs as a scheduled Kubernetes CronJob rather than relying purely on GitOps pruning, since Argo CD’s reconciliation depends on the SCM API remaining reachable and the generator config remaining correct. This script provides an independent, defence-in-depth cleanup path:

1#!/usr/bin/env bash
2set -euo pipefail
3
4NOW=$(date +%s)
5MAX_TTL_SECONDS=$((36 * 3600))
6
7kubectl get ns -l 'preview.kbytech.io/pr' -o json | jq -c '.items[]' | while read -r ns; do
8  NAME=$(echo "$ns" | jq -r '.metadata.name')
9  CREATED=$(echo "$ns" | jq -r '.metadata.creationTimestamp')
10  CREATED_EPOCH=$(date -d "$CREATED" +%s)
11  AGE=$((NOW - CREATED_EPOCH))
12
13  if [ "$AGE" -gt "$MAX_TTL_SECONDS" ]; then
14    echo "Reaping stale preview namespace: $NAME (age: ${AGE}s)"
15    kubectl delete namespace "$NAME" --wait=false
16  fi
17done

#Wildcard Ingress and Certificate Strategy

Issuing a discrete TLS certificate per ephemeral preview environment against a public CA is the single fastest way to hit Let’s Encrypt’s rate limits during a busy merge day. The correct approach is a single wildcard certificate for *.preview.kbytech.io, issued once via cert-manager using DNS-01 validation, and referenced by every PR’s ingress object as a shared TLS secret. Per-PR ingress objects then only need a distinct host field, not a distinct tls block, which removes certificate issuance entirely from the per-PR provisioning path.

#Failure Modes and Edge Cases

Ephemeral preview environments introduce failure classes that a static staging environment does not have to contend with:

Namespace leak on webhook or polling failure. If the SCM API is unreachable during a polling interval, the ApplicationSet generator simply does not see a closed PR and the namespace persists. This is precisely why the standalone TTL reaper CronJob must exist independently of GitOps state — it is the only cleanup path that does not depend on the SCM API being reachable.

Shared database contamination. Many teams provision ephemeral preview environments against application containers but point them at a shared upstream database to save cost. This defeats the isolation guarantee the pattern exists to provide: two PRs running conflicting migrations against the same schema will corrupt each other’s data exactly as they did on shared staging. The correct pattern is a per-namespace ephemeral Postgres instance (via a lightweight operator such as CloudNativePG) seeded from a nightly anonymised snapshot, not a shared database with row-level tenancy hacks.

DNS propagation lag on wildcard records. If the wildcard A/CNAME record has a low TTL cached upstream at a corporate resolver, newly provisioned preview subdomains can resolve inconsistently for the first few minutes. This is usually invisible because the wildcard record itself does not change, only the ingress routing behind it, but it is worth validating that your ingress controller’s default backend does not return a bare 404 that QA engineers mistake for a broken deploy.

Resource quota starvation causing pending pods. A ResourceQuota that is too conservative will leave pods in Pending indefinitely with no obvious error surfaced to the developer, because the scheduler failure is silent from the CI pipeline’s perspective. Pipe kubectl describe pod events back into the PR comment via your CI bot rather than assuming a green deploy step means a healthy pod.

Cluster autoscaler thrash. A burst of merges at end-of-sprint can create dozens of ephemeral preview environments simultaneously, each requesting CPU and memory against the quota above. If your preview workloads share a node pool with production-adjacent workloads, autoscaler churn from this burst can introduce scheduling latency for unrelated pods. Isolate preview namespaces onto a dedicated, aggressively-scaled node pool with a taint, and label preview workloads with a matching toleration.

#Scaling and Security Trade-offs

Running ephemeral preview environments at meaningful pull request volume forces several concrete trade-off decisions:

• Node pool isolation vs cost — a dedicated spot-instance node pool for preview workloads avoids autoscaler contention with production-adjacent traffic, but idle capacity during low-PR periods is pure cost overhead unless the pool scales to zero.
• Namespace-level NetworkPolicy vs debugging friction —ephemeral preview environments should default-deny cross-namespace traffic to prevent PR-A’s workload from reaching PR-B’s database, but this adds friction when engineers need to debug shared dependency calls during triage.
• Per-namespace secrets provisioning vs credential sprawl —ephemeral preview environments need scoped API keys and database credentials, ideally minted per namespace via External Secrets Operator against Vault, rather than a single long-lived secret copied across every PR namespace, which multiplies blast radius on leak.
• RBAC scoping vs onboarding overhead — restricting each engineer’s kubeconfig context to only their own PR’s namespace via a RoleBinding is the correct least-privilege posture, but requires automation to issue and revoke bindings in step with namespace creation and deletion.
• TTL aggressiveness vs QA availability — a short TTL (e.g. 12 hours) reclaims capacity faster but risks tearing down an environment mid-review if QA works across time zones; a longer TTL (36–48 hours) is safer for review continuity but increases steady-state cluster load.

None of these trade-offs have a universally correct answer — they depend on merge velocity, team distribution across time zones, and whether preview infrastructure is billed against the same cost centre as production. What is consistent across deployments of this pattern is that ephemeral preview environments only deliver their promised isolation benefit when the TTL reaper, resource quotas, and network policies are treated as first-class, independently tested components of the platform, not optional hardening applied after the first incident.

Teams that treat namespace-per-PR as “just another Helm release” tend to rediscover, at cost, that ephemeral infrastructure without a defence-in-depth cleanup path degrades into exactly the shared-state mess it was built to replace.