Fixing State Drift in GraphQL Gateways

A federated GraphQL layer is supposed to be stateless at the edge — that is the entire pitch of putting a gateway in front of your subgraphs. In practice, once you scale beyond a single gateway replica, you inherit a distributed systems problem that most teams do not budget for: state distribution anomalies in distributed GraphQL gateways. Query plan caches diverge, Automatic Persisted Query (APQ) hashes resolve to different documents depending on which pod answers the request, and entity-level response caches desynchronise across nodes during rolling deployments. The symptom is almost always the same — intermittent 400s, phantom cache misses, or a subscription that silently stops delivering events after a canary rollout. The root cause is rarely the schema. It is the gateway’s local state model colliding with horizontal scaling.

This article dissects the mechanics of that failure class in Apollo Router / Federation-style architectures (the pattern generalises to Hasura, WunderGraph, and hand-rolled schema-stitching gateways), and lays out a concrete remediation architecture using externalised, versioned state.

#Problem Statement: Why Stateless Gateways Aren’t Actually Stateless

A GraphQL gateway performs three operations that are, by design, cacheable and therefore stateful in practice:

• Query planning — compiling a client operation against the supergraph schema into a distributed execution plan across subgraphs.
• Automatic Persisted Queries — mapping a SHA-256 hash to a full query document so clients can send hashes instead of payloads.
• Entity and field-level response caching — memoising resolver output keyed by entity `id` and field arguments.

Each of these caches is, by default, held in-process. When you run a single gateway instance, this is fine — the cache is always consistent with itself. The moment you run N replicas of a distributed GraphQL gateway behind a load balancer without shared cache backing, you introduce three independent sources of truth that are only eventually consistent with each other, and often not consistent at all during a deployment window.

The most common trigger is a rolling schema update. Replica A pulls the new supergraph SDL from the uplink before Replica B. For the several seconds (sometimes minutes, depending on your uplink poll interval) that the fleet is mixed-version, a client operation routed to Replica A produces a different query plan than the identical operation routed to Replica B. If the client has APQ enabled and the hash was first registered against Replica A’s plan, a subsequent request landing on Replica B — which has never seen that hash — returns PersistedQueryNotFound, forcing a client-side retry with the full document. At scale, this looks like a retry storm correlated exactly with deploy windows, and it is almost always misdiagnosed as a network blip.

#Architectural Breakdown

The correct mental model is to treat the gateway tier as a stateless compute layer and externalise every piece of derived state — query plans, APQ hashes, and entity cache entries — into a shared, versioned store. This mirrors the same reasoning applied to session state in traditional web tiers, just moved up a layer into the GraphQL execution engine. If you are auditing broader architectural patterns for stateless compute versus shared state, this is a direct application of that same trade-off at the API gateway tier.

The reference architecture below separates the gateway replicas from a shared Redis cluster that owns APQ hash mappings and entity cache entries, while schema version propagation is handled through a monotonic version token embedded in every query plan cache key — preventing cross-version cache pollution entirely, rather than merely tolerating it.

The critical design decision is the schema_version namespace prefix on the query plan cache key. Without it, a stale plan computed against SDL v41 can be served alongside fresh entity data computed against SDL v42, producing type mismatches that surface as opaque INTERNAL_SERVER_ERROR responses at the client. With it, a version mismatch simply forces a cache miss and a fresh plan compile — slower, but correct.

#Where the Anomaly Actually Lives

In distributed GraphQL gateways, the anomaly is never in the subgraphs themselves — subgraphs are typically stateless resolvers over a database and recover cleanly. The anomaly lives at the boundary between the gateway’s in-memory LRU (query plan cache, APQ cache) and the assumption, baked into most reverse proxy configurations, that any replica can answer any request identically. That assumption is false the instant you have asymmetric SDL propagation, asymmetric TTL expiry, or a sticky-session load balancer that pins a client to a replica which then gets terminated mid-rollout.

#Implementation Logic

The remediation sequence, in the order it should be implemented:

• Move APQ hash storage from in-process LRU to a shared Redis instance with a consistent TTL (recommend 24 hours minimum to match typical client cache lifetimes).
• Namespace every cached artefact — query plans and entity responses — by a monotonically increasing schema version token, sourced from the uplink’s launchId or a content hash of the composed supergraph SDL.
• Reduce the schema uplink poll interval to the smallest value your registry rate limit tolerates (typically 10 seconds), to shrink the mixed-version window during rollout.
• Disable sticky sessions at the load balancer for the GraphQL gateway route specifically — the gateway tier should be treated as fungible compute, not session-affine.
• For subscriptions, externalise WebSocket connection registries to a pub/sub backbone (Redis Streams or NATS) so a subscription survives the termination of the specific gateway pod that originated it.

#Code & Configurations

The Apollo Router configuration below wires distributed APQ and entity caching to a Redis cluster, and sets an aggressive uplink poll interval to minimise version skew across a fleet of distributed GraphQL gateways.

1supergraph:
2 listen: 0.0.0.0:4000
3
4uplink:
5 poll_interval: 10s
6 fallback_urls:
7 - https://uplink.api.apollographql.com/
8
9apq:
10 router:
11 cache:
12 redis:
13 urls: ["redis://redis-cluster.internal:6379"]
14 timeout: 2ms
15 ttl: 86400s
16
17preview_entity_cache:
18 enabled: true
19 redis:
20 urls: ["redis://redis-cluster.internal:6379"]
21 namespace: "gw-entity-v1"
22 subgraph:
23 all:
24 enabled: true
25 ttl: 30s
26
27traffic_shaping:
28 router:
29 timeout: 30s
30 all:
31 timeout: 10s
32 global_rate_limit:
33 capacity: 5000
34 interval: 1s

To verify version skew across replicas during a rollout, poll the debug endpoint on each pod directly rather than through the load balancer:

1#!/usr/bin/env bash
2# Compare schema hash across all gateway pods to detect propagation lag
3for pod in $(kubectl get pods -l app=graphql-gateway -o jsonpath='{.items[*].metadata.name}'); do
4 hash=$(kubectl exec "$pod" -- curl -s localhost:4000/.well-known/apollo/server-health 
5 | jq -r '.schema.hash // "unknown"')
6 echo "$pod -> $hash"
7done
8
9# A healthy rollout converges to a single hash within one uplink poll interval (~10s).
10# If hashes remain split for >60s, the uplink fallback path is likely being triggered.

Entity cache keys must be namespaced by schema version to prevent cross-version pollution. The following Rhai script snippet (used by Apollo Router’s scripting hooks) demonstrates the key-derivation logic conceptually:

1fn build_cache_key(request, schema_version) {
2 let entity_id = request.body.variables["id"];
3 let field_set = request.query_hash;
4 // Namespacing prevents a v41-plan cache hit against v42 entity data
5 return `entity:${schema_version}:${entity_id}:${field_set}`;
6}

For teams building a custom gateway rather than using Apollo Router, the same key-derivation discipline applies regardless of runtime — the RFC on HTTP caching semantics (RFC 7234) is a useful baseline for reasoning about validator-based invalidation when you extend this pattern to CDN-fronted GraphQL POST caching.

#Failure Modes & Edge Cases

Even with externalised state, distributed GraphQL gateways fail in specific, predictable ways that are worth instrumenting for explicitly:

• Redis latency spikes cascading into request latency — if the entity cache lookup is on the synchronous request path with no circuit breaker, a Redis GC pause or network partition turns a 5ms cache hit into a 2s timeout multiplied across every field resolver. Mitigate with a strict 2ms Redis timeout and a fail-open policy that falls through to subgraph execution.
• Thundering herd on cache invalidation — when a schema version increments, every replica simultaneously experiences 100% query plan cache misses. Without request coalescing (single-flight locking per query hash), this produces a synchronised compile storm that can double CPU load across the fleet for the first 10–15 seconds post-deploy.
• Subscription orphaning — a WebSocket connection pinned to a gateway pod that gets evicted during a scale-down event loses its subscription silently unless the pub/sub backbone re-establishes the topic subscription on reconnect. Clients must implement exponential backoff reconnect with subscription re-registration, not just transport-level reconnect.
• APQ hash collision across schema versions — extremely rare (SHA-256), but if you namespace APQ storage per-tenant without also namespacing per schema version in a multi-tenant supergraph, a hash registered under an old schema can resolve against a renamed field in the new schema, producing a plan compile error rather than a clean cache miss.
• Clock skew between gateway pods — TTL-based entity cache expiry assumes synchronised clocks. In practice, NTP drift beyond a few hundred milliseconds is rarely material, but it becomes visible when debugging why one replica appears to expire cache entries “early” relative to another during forensic log correlation.

#Scaling & Security Trade-offs

Externalising state to Redis solves the consistency problem for distributed GraphQL gateways but introduces a new dependency with its own failure surface. The trade-offs need to be made explicit to whoever owns the SLO:

• Consistency vs. latency — a shared Redis-backed query plan cache guarantees identical plans across all replicas, at the cost of a network round-trip (~1-3ms intra-VPC) on every cache lookup versus sub-microsecond in-process LRU access.
• Blast radius — centralising APQ and entity cache state in one Redis cluster means a Redis outage degrades every gateway replica simultaneously rather than in isolation. Run Redis in a clustered, multi-AZ topology and treat it as tier-0 infrastructure, not an optional cache layer.
• Cache poisoning surface — a shared cache is a larger attack surface than N independent in-process caches. Entity cache keys derived from unsanitised user input (e.g., raw JWT claims embedded in cache keys) can enable cross-tenant cache poisoning in multi-tenant supergraphs. Always hash and namespace tenant identifiers before using them in cache keys.
• Operational overhead vs. correctness — teams running fewer than 3 gateway replicas with infrequent deploys may find the added Redis dependency not worth the operational cost; the anomaly rate scales with replica count and deploy frequency, not absolute traffic volume.
• Version-skew window — reducing the uplink poll interval below 10 seconds shrinks the mixed-version blast radius but increases load on the schema registry; benchmark against your registry’s documented rate limits before tuning aggressively.

Treat query planning, APQ resolution, and entity caching as three independently versioned state stores rather than a single monolithic “gateway cache,” and the class of anomalies described here becomes a solved, observable problem rather than an intermittent production mystery.