The Internal Developer Platform – Part 3: Implementing the IDP Reconciliation Loop

Provisioning a developer platform via a service catalog is the easy 20% of the problem. The remaining 80% is keeping the declared state of every workspace, database, and IAM binding in sync with what actually exists in the underlying cloud accounts. Once your Internal Developer Platform (IDP) crosses roughly 200 provisioned resources, imperative scripts running on a cron schedule stop being viable: drift accumulates, partial failures leave orphaned infrastructure, and nobody can answer “what is the current source of truth” with confidence. The fix is not more automation scripts — it is a proper reconciliation loop, the same control-theory pattern that underpins Kubernetes’ own controllers.

This is Part 3 of our IDP series. Part 1 covered the golden-path abstraction layer and the self-service API contract; Part 2 implemented an IDP control plane concretely, using Kubernetes CRDs and an Operator running a level-triggered reconciliation loop. Here we zoom out from that specific Kubernetes implementation to the reconciliation pattern itself as general architecture — the control-theory model of level-triggered convergence that applies whether your control plane is Kubernetes-native, built on Crossplane, or a bespoke system layered over Terraform state — and address the hardest engineering problem in platform design: making the system self-healing by continuously converging actual state toward desired state, without a human in the loop and without a stampede of API calls when things go wrong.

#Problem Statement: Why Cron-Based Sync Fails

Most first-generation internal platforms implement “sync” as a scheduled job: pull desired state from a Git repo or database, diff against a snapshot of cloud resources, and apply changes every 15 minutes. This model has three structural failures at scale:

• Edge-triggered blindness — if the job crashes mid-run, or a webhook is dropped, the system has no memory of the missed event. State silently diverges until the next full scan.
• No backpressure — a burst of 500 simultaneous changes (e.g., a bulk onboarding event) triggers 500 concurrent API calls against a cloud provider with a hard rate limit, resulting in cascading 429s.
• No idempotency guarantee — re-running the same sync twice can create duplicate resources if the “check before create” logic races against itself.

A reconciliation loop solves all three by treating infrastructure state the way a control system treats a physical process: continuously measure the error between desired and actual state, and apply a bounded correction, repeatedly, forever.

#Architectural Breakdown

The core theoretical model is level-triggered reconciliation rather than edge-triggered — the same principle introduced against a concrete Kubernetes Operator in Part 2. In an edge-triggered system, you react to individual events (“user updated the manifest”). In a level-triggered system, you react to the current observed level of the system regardless of history: it does not matter whether three events or thirty were missed, the reconciler always re-derives the correct action from the current diff. This is precisely the pattern documented in the Kubernetes controller architecture, and it generalises cleanly to any IDP built on a custom resource graph, whether backed by Kubernetes CRDs, Crossplane, or a bespoke control plane on top of Terraform state.

#The Three Core Components

• Informer/Watcher — maintains a local, eventually-consistent cache of desired state objects via a watch stream (not polling), reducing API server load.
• Work Queue — a rate-limited, deduplicating queue keyed by resource identity. Multiple change events for the same object collapse into a single queued key.
• Reconciler Function — a pure function: f(desired, actual) -> actions. It must be idempotent, side-effect-safe on retry, and stateless between invocations (all state lives in the desired/actual stores, never in the reconciler’s memory).

Applying these architectural patterns consistently across every resource type in the platform (compute, DNS, IAM, secrets) is what separates a mature IDP control plane from a collection of glue scripts.

#Level-Triggered vs Edge-Triggered: Why It Matters for IDPs

Consider a developer submitting a manifest that requests an RDS instance. An edge-triggered system fires “create RDS” once, on submission. If that call times out, the developer’s request is lost unless something explicitly retries it. A level-triggered reconciliation loop instead observes: “desired state says RDS instance X should exist; actual state says it does not” on every pass, and keeps re-queuing the object until the observed actual state matches. The reconciler doesn’t care that the timeout happened — it just sees the diff persist and tries again, with backoff.

#Implementation Logic

The execution flow for a single reconciliation pass should follow this sequence strictly, in order, with no shortcuts:

• Fetch the desired-state object from the local informer cache (never hit the API server directly inside the hot loop).
• Fetch the actual-state object from the cloud provider or downstream API, using a cached or batched read where the provider supports it.
• Diff the two structures using a deterministic, field-level comparison — not a whole-object hash, so unrelated field changes (e.g., provider-managed timestamps) don’t trigger false-positive drift.
• Generate a patch representing the minimal set of API calls required to close the gap.
• Apply the patch, capturing partial failure per sub-resource.
• Update status on the desired-state object (observed generation, last-reconciled timestamp, condition list) so downstream consumers (dashboards, CLI tools) can query convergence state without re-running the diff themselves.
• Requeue — either immediately (if the patch failed transiently), after a fixed re-sync interval (to catch out-of-band drift), or not at all (if fully converged and no re-sync policy is set).

Every reconciliation loop needs a hard upper bound on requeue frequency. Without this, a permanently broken resource (e.g., invalid credentials) will burn through your API rate limit indefinitely. This is where exponential backoff with jitter becomes non-negotiable, not optional.

#Code & Configurations

Below is a representative custom resource definition for a platform-managed “Workspace” object — the desired-state schema the reconciliation loop watches.

1apiVersion: platform.kby.io/v1alpha1
2kind: Workspace
3metadata:
4 name: team-payments-prod
5 annotations:
6 platform.kby.io/owner: "payments-team"
7spec:
8 environment: production
9 resources:
10 database:
11 engine: postgres
12 version: "15"
13 sizeClass: db.r6g.large
14 network:
15 vpcRef: shared-prod-vpc
16 reconcilePolicy:
17 resyncIntervalSeconds: 300
18 maxBackoffSeconds: 900
19status:
20 observedGeneration: 14
21 conditions:
22 - type: Reconciled
23 status: "True"
24 lastTransitionTime: "2024-03-11T09:14:02Z"

The reconciler itself, expressed as Go-style pseudocode against a controller-runtime-shaped interface, illustrates the idempotent structure required:

1func (r *WorkspaceReconciler) Reconcile(ctx context.Context, req Request) (Result, error) {
2 desired, err := r.Cache.Get(req.NamespacedName)
3 if errors.IsNotFound(err) {
4 return Result{}, nil // object deleted, nothing to do
5 }
6
7 actual, err := r.CloudClient.Describe(ctx, desired.Spec.Resources)
8 if err != nil {
9 return Result{RequeueAfter: 15 * time.Second}, err
10 }
11
12 patch := ComputeDiff(desired.Spec, actual)
13 if patch.IsEmpty() {
14 r.setCondition(desired, "Reconciled", true)
15 return Result{RequeueAfter: desired.Spec.ReconcilePolicy.ResyncInterval}, nil
16 }
17
18 if err := r.CloudClient.Apply(ctx, patch); err != nil {
19 r.setCondition(desired, "Reconciled", false)
20 return Result{RequeueAfter: r.backoff.Next(req)}, err
21 }
22
23 r.backoff.Reset(req)
24 return Result{RequeueAfter: 30 * time.Second}, nil
25}

Finally, a drift-detection sanity check that platform SREs can run out-of-band against the reconciliation loop’s own metrics endpoint, useful during incident triage:

1curl -s http://controller.internal:8080/metrics 
2 | grep 'reconcile_errors_total|workqueue_depth|reconcile_duration_seconds'
3
4# workqueue_depth > 0 for > 5 minutes indicates a stuck reconciler,
5# usually caused by a poison-pill object failing every attempt.

#Failure Modes & Edge Cases

A reconciliation loop that hasn’t been stress-tested against the following failure modes will eventually take down the platform’s control plane, not just an individual workspace:

• Hot-loop / thundering requeue — a bug in the diff function that never converges (e.g., comparing a timestamp field that always differs) causes infinite requeue at maximum frequency, saturating the work queue and starving unrelated objects of processing time.
• Poison-pill resources — a single object with permanently invalid configuration (bad credentials, quota exceeded) will retry forever unless the reconciler enforces a max retry ceiling after which it flips the object into a terminal Failed condition requiring human intervention.
• Stale informer cache — if the watch connection silently drops and the informer doesn’t detect it, the reconciler operates against outdated desired state. Mitigate with periodic full-list re-syncs (typically every 10–30 minutes) independent of the watch stream.
• Out-of-band mutation — an engineer manually changes a resource via the cloud console, bypassing the platform. The reconciliation loop must treat this as drift and revert it on the next pass, which is correct behaviour but frequently surprises teams unfamiliar with GitOps-style enforcement.
• Split-brain during leader election — if two controller replicas both believe they hold the lock (due to a lease renewal race), you get double-application of patches. This is why leader election must use a strongly consistent lock store (etcd lease, or a database with row-level locking and TTL), never an eventually-consistent one.
• Cascading dependency drift — reconciling a VPC object before the security groups it depends on have converged produces a transient invalid state. The reconciler must either encode dependency ordering explicitly or be tolerant of “not ready yet” errors and simply requeue rather than treating them as failures.

#Scaling & Security Trade-offs

Once the reconciliation loop is correct, the remaining decisions are about how far to push concurrency and how tightly to scope credentials. These are genuine trade-offs, not defaults to copy blindly:

• Single sharded controller vs per-tenant controllers — a single controller with sharded work queues scales more efficiently (shared cache, lower memory overhead) but a bug affects every tenant simultaneously; per-tenant controller instances isolate blast radius at the cost of higher baseline compute.
• Concurrent workers per reconciler — increasing worker count from 1 to 10 improves throughput roughly linearly until you hit the cloud provider’s API rate limit, at which point additional concurrency only increases 429 error rates; tune MaxConcurrentReconciles against measured provider limits, not arbitrary defaults.
• Broad IAM role vs scoped per-resource-type credentials — a single controller identity with account-wide permissions simplifies operations but means a reconciler bug can mutate any resource type; scoping credentials per resource controller (database controller can only touch RDS APIs) limits damage from a compromised or buggy reconciler at the cost of credential-management overhead.
• Full re-sync interval length — short intervals (60s) catch drift quickly but increase read-API load and cost on providers that bill per API call; long intervals (30–60 min) reduce cost but widen the window during which out-of-band drift goes undetected.
• Optimistic concurrency vs pessimistic locking — using resource-version checks (optimistic) scales better under high write concurrency but requires careful retry-on-conflict logic; pessimistic locking simplifies reasoning but serialises writes to the same object, which becomes a bottleneck for frequently-updated shared resources like a shared VPC.

The reconciliation loop is the mechanism that turns a static provisioning tool into a genuinely self-healing platform. Get the level-triggered model, idempotency guarantees, and backoff ceilings right, and drift becomes a metric you observe and correct automatically rather than an incident you discover during an audit. Part 4 of this series applies this same convergence model to a concrete decision every platform team eventually faces: whether the resources being reconciled are modelled as Crossplane Compositions or as layered Terraform modules.