Killing Cache Stampedes with Request Coalescing

A single hot key expiring under load can take down a backend that was otherwise sized correctly for steady-state traffic. This is the cache stampede problem: thousands of concurrent readers hit an empty cache slot simultaneously, all fall through to the origin datastore, and the resulting synchronised load spike saturates connection pools, exhausts database CPU, and triggers cascading timeouts across dependent services. Unlike a gradual traffic ramp, a cache stampede is a step-function load event — and most autoscaling policies react far too slowly to absorb it.

This article covers the architecture required to eliminate cache stampede events in high-concurrency read-through caching layers, using request coalescing, distributed locking, and probabilistic early expiration. The patterns below assume a standard Redis or Memcached tier fronting a relational or document store, but the underlying theory applies to any read-through or read-aside cache topology.

#Why TTL-Based Caching Fails Under Concurrency

Naive read-through caching follows a predictable sequence: check cache, on miss query origin, write result back with a fixed TTL. This works acceptably at low concurrency. At high concurrency, three failure conditions compound:

• Synchronised expiry — keys written in a batch (e.g. a cache warm after deployment) expire simultaneously, producing a correlated miss wave.
• Retry amplification — client-side retries on timeout multiply the effective request rate against the origin during the exact window it is least able to cope.
• No mutual exclusion on recomputation — nothing stops N concurrent threads from independently recomputing the same expensive query result.

The result is a thundering herd against the origin precisely when it is under the most pressure to serve degraded latency. A cache stampede is not a capacity problem; it is a synchronisation problem, and it must be solved at the concurrency-control layer, not by throwing more database replicas at it.

#Architectural Breakdown of Cache Stampede Mitigation

There are three complementary architectural patterns that, combined, eliminate almost all cache stampede scenarios:

#1. Request Coalescing (In-Process Singleflight)

Within a single service instance, concurrent requests for the same cache key should be collapsed into a single upstream call. Go’s golang.org/x/sync/singleflight package implements this pattern natively — the first caller executes the function, subsequent callers block and receive the shared result. This alone reduces per-instance fan-out to origin from N to 1, but does nothing across a fleet of N instances, which is where distributed locking becomes necessary.

#2. Distributed Mutual Exclusion

Across a fleet, only one instance should be permitted to recompute a given key at a time. This is implemented via a short-lived distributed lock (typically SET key value NX PX in Redis). Losing instances either block briefly and retry the cache read, or — preferably — serve stale data while the lock holder refreshes in the background. This is the stale-while-revalidate pattern, borrowed from HTTP caching semantics (see RFC 5861).

#3. Probabilistic Early Expiration

Rather than waiting for a hard TTL expiry to trigger a stampede, each read probabilistically decides whether to recompute the value before expiry, with probability increasing as the key approaches its TTL boundary. This is the XFetch algorithm from Facebook/Meta’s caching research, and it desynchronises recomputation across readers by design — no single moment sees a spike because refreshes are spread stochastically across the pre-expiry window.

These three layers map cleanly onto the request lifecycle: singleflight handles intra-process concurrency, the distributed lock handles inter-process concurrency, and probabilistic early expiration prevents the miss from ever reaching the lock contention path in the majority of cases. When designing this into broader architectural patterns, treat the cache tier as a concurrency-control system first and a storage optimisation second.

#Implementation Logic

The recomputation path should follow this exact decision order on every cache read:

• Read the value and its stored metadata (TTL, computed-at timestamp, delta cost).
• If the value is missing entirely, attempt to acquire the distributed lock; on success, recompute and write; on failure, either block-and-retry or serve a default/degraded response.
• If the value is present but past its probabilistic early-expiry threshold, serve the cached value immediately and trigger an asynchronous background refresh guarded by the same lock.
• If the value is present and fresh, serve it directly with zero origin contact.

Critically, the lock TTL must always be shorter than the time it would take for a stalled recompute to become indistinguishable from a crashed lock holder. Set it too long and a crashed process blocks recomputation for the full duration; too short and the lock expires mid-computation, allowing a second instance to duplicate the work.

#Code and Configuration

In-process coalescing using Go’s singleflight group:

1var group singleflight.Group
2
3func GetUserProfile(ctx context.Context, id string) (*Profile, error) {
4    v, err, _ := group.Do(id, func() (interface{}, error) {
5        if cached, ok := cache.Get(id); ok {
6            return cached, nil
7        }
8        return fetchAndCacheFromOrigin(ctx, id)
9    })
10    if err != nil {
11        return nil, err
12    }
13    return v.(*Profile), nil
14}

Distributed lock acquisition via Redis, using an atomic Lua script to guarantee the lock check-and-refresh happens without a race window:

1-- acquire_and_refresh.lua
2-- KEYS[1] = cache key, KEYS[2] = lock key
3-- ARGV[1] = lock TTL ms, ARGV[2] = requesting instance id
4
5local lock_owner = redis.call('GET', KEYS[2])
6if lock_owner == false then
7    redis.call('SET', KEYS[2], ARGV[2], 'PX', ARGV[1])
8    return 'LOCK_ACQUIRED'
9elseif lock_owner == ARGV[2] then
10    return 'ALREADY_OWNER'
11else
12    return 'LOCK_HELD'
13end

Probabilistic early expiration check, applied on every read before deciding whether to trigger a background refresh (XFetch-style formula):

1import math, random, time
2
3def should_recompute(computed_at: float, ttl: float, delta: float, beta: float = 1.0) -> bool:
4    now = time.time()
5    xfetch = delta * beta * math.log(random.random())
6    return (now - (xfetch * -1)) >= (computed_at + ttl)

At the edge, NGINX’s proxy_cache_lock directive provides coalescing before requests even reach the application tier, which is often the cheapest place to stop a cache stampede:

1proxy_cache_path /var/cache/nginx levels=1:2 keys_zone=api_cache:50m max_size=2g inactive=60m;
2
3location /api/ {
4    proxy_cache api_cache;
5    proxy_cache_key "$scheme$request_method$host$request_uri";
6    proxy_cache_lock on;
7    proxy_cache_lock_timeout 2s;
8    proxy_cache_use_stale updating error timeout;
9    proxy_cache_background_update on;
10    proxy_pass http://origin_upstream;
11}

The combination of proxy_cache_use_stale updating and proxy_cache_background_update on means NGINX will serve the stale response to all waiting clients while exactly one request refreshes the cache in the background — this is request coalescing at the reverse-proxy layer, and for many services it eliminates the need for application-level singleflight entirely.

#Failure Modes and Edge Cases

A distributed lock introduces its own failure surface, and any cache stampede mitigation strategy must be evaluated against it:

• Lock holder crash mid-computation — if the process dies after acquiring the lock but before writing the result, the cache remains empty until the lock TTL expires. Keep lock TTLs tight (typically 2-5 seconds for sub-100ms queries) and always wrap the recompute in a deferred lock-release, not just a natural expiry.
• Cold cache after deployment — a full cache flush on rollout produces a stampede regardless of locking sophistication, because every key is simultaneously missing. Mitigate with cache pre-warming scripts executed before traffic cutover, or blue-green cache tiers that inherit the previous generation’s data.
• Clock skew across nodes — probabilistic early expiration relies on consistent wall-clock comparisons. In multi-region deployments, use a monotonic logical clock or origin-stamped computed-at values rather than trusting local NTP drift.
• Lock stampede — under extreme contention, hundreds of instances polling for lock release can themselves become a secondary cache stampede against Redis. Use exponential backoff with jitter on lock retry, never a fixed-interval poll.
• Negative caching gaps — if origin queries legitimately return “not found,” failing to cache that negative result means every subsequent miss re-triggers the full recompute path, effectively guaranteeing a cache stampede on any missing-key hot path (a common cause in ID-enumeration attack surfaces).

#Scaling and Security Trade-Offs

Every mitigation layer trades latency, complexity, or consistency for stampede resistance. These trade-offs should be made explicit during capacity planning:

• Singleflight vs. distributed lock — singleflight is near-zero overhead but only protects a single process; distributed locking protects the fleet but adds one round-trip (typically 0.5-2ms to Redis) to every miss path.
• Stale-while-revalidate vs. strict consistency — serving stale data during refresh dramatically reduces cache stampede blast radius but is unsuitable for financial balances, inventory counts, or any strongly consistent read path.
• Probabilistic early expiration tuning — a high beta value smooths refresh distribution but increases average origin QPS; a low beta value approaches standard TTL behaviour and re-introduces synchronised expiry risk.
• Security surface of lock keys — lock and cache keys derived directly from unsanitised user input are vulnerable to key-injection collisions; always hash or namespace keys (e.g. sha256(tenant_id + resource_id)) to prevent one tenant’s traffic pattern from inducing lock contention against another’s cache namespace in a shared multi-tenant Redis cluster.
• Operational cost — edge-layer coalescing (NGINX, Envoy) is the cheapest to operate and requires no application code changes, but offers coarser control than application-level singleflight combined with XFetch, which requires engineering investment but yields the tightest origin-load bound under adversarial traffic patterns.

None of these layers is a substitute for correct capacity planning against the origin datastore, but together they convert a step-function failure event into a smooth, bounded load curve — which is the actual engineering goal when defending against a cache stampede in production.