Fixing gRPC Load Balancing Behind L4 Proxies

A single HTTP/2 connection carrying thousands of RPCs a second looks efficient on paper. In practice, it is the root cause of the most common production incident in gRPC fleets: one pod at 95% CPU while five others idle at 4%. This is connection pinning, and it exists because most teams reach for the same Layer 4 load balancer they used for REST, without accounting for how gRPC multiplexes work over a persistent transport. Solving gRPC load balancing correctly requires abandoning the assumption that a TCP connection maps to a single request, and instead architecting around HTTP/2 stream multiplexing, client-side resolution, and control-plane-driven rebalancing.

#The Problem Statement: Why L4 Load Balancing Breaks gRPC

gRPC is built on HTTP/2. A client opens one TCP connection to a server and multiplexes concurrent RPCs as HTTP/2 streams over that single socket. From the perspective of an L4 load balancer (a network load balancer, kube-proxy iptables/IPVS rules, or a classic AWS NLB), this is a single long-lived flow. The 5-tuple hash used for connection distribution is computed once, at connection establishment, and never revisited.

The consequence: if you have three gRPC clients and ten backend pods behind an L4 service, you get at most three active backends receiving traffic, regardless of how many thousand RPCs per second flow through those three connections. Add HTTP/2 keepalive and connection reuse, and those three connections can persist for hours, well past any rolling deployment or HPA scale-out event. This is not a load balancing bug in the traditional sense; it is a mismatch between the transport model (persistent multiplexed streams) and the balancing layer (per-connection flow hashing).

Kubernetes Service objects using ClusterIP with iptables mode exhibit this precisely. So do most cloud L4 load balancers unless you explicitly enable per-request distribution, which most do not support for gRPC/HTTP2 traffic without an intermediate L7 proxy.

#Architectural Breakdown: Where Load Balancing Decisions Must Live

There are two architecturally sound fixes, and they solve different problems:

• Client-side load balancing — the gRPC client resolves a list of backend addresses (via DNS, xDS, or a custom resolver) and maintains a pool of subchannels, distributing RPCs across them using a pick-first, round_robin, or weighted policy.
• L7 proxy load balancing — an intermediary (Envoy, Linkerd’s proxy, or an Nginx build with HTTP/2 upstream support) terminates the client connection and re-multiplexes RPCs across a pool of upstream connections it manages independently, rebalancing per-stream rather than per-connection.

Both approaches move the balancing decision above Layer 4, into a component that understands HTTP/2 stream boundaries. The trade-off is where you put that intelligence: in every client binary, or in a shared proxy tier. For platform teams running heterogeneous client languages, L7 proxy load balancing is usually the pragmatic default, since it centralises the policy. For latency-sensitive service-to-service calls where an extra proxy hop is unacceptable, client-side gRPC load balancing via xDS is the correct architectural pattern — this is precisely the model documented in gRPC’s own load balancing design notes, which describes the lookaside and client-side balancing modes in detail.

Regardless of which side owns the decision, the underlying requirement is the same: the balancer must operate on a live, frequently refreshed backend set, not a DNS A-record cached for the connection’s lifetime. This is where most naive fixes — “just use round_robin in the client” — still fail, because DNS-based resolution without active health checking and short TTL enforcement simply moves the pinning problem from the L4 device to the client’s own subchannel pool.

#Implementation Logic: Building a Correct Resolution Path

The implementation has three layers that must each be configured correctly, or the fix degrades back into pinning under different conditions:

1. Service discovery: expose a headless Kubernetes Service so DNS returns every pod IP rather than a single ClusterIP VIP.
2. Client resolver and balancer policy: configure the gRPC client to use round_robin or a weighted policy instead of the default pick_first.
3. Connection lifecycle management: force periodic connection recycling with MAX_CONNECTION_AGE so that new pods entering the pool actually receive traffic, rather than long-lived connections persisting until process restart.

Step one removes the VIP that was hiding the real backend topology from the client. Step two ensures the client, once it has a full address list, actually spreads RPCs rather than pinning to the first resolved address (the gRPC default pick_first policy is optimised for single-backend simplicity, not fleet distribution). Step three prevents staleness — without connection aging, a client that resolved five backends at startup will keep hammering those five even after the deployment scales to fifty.

#Kubernetes: Headless Service for Direct Pod Resolution

1apiVersion: v1
2kind: Service
3metadata:
4  name: inventory-grpc
5  labels:
6    app: inventory
7spec:
8  clusterIP: None
9  selector:
10    app: inventory
11  ports:
12    - name: grpc
13      port: 9090
14      targetPort: 9090
15  publishNotReadyAddresses: false

clusterIP: None forces kube-dns/CoreDNS to return a full set of A/AAAA records for every ready pod matching the selector, rather than a single virtual IP. This is the mandatory prerequisite for any client-side gRPC load balancing scheme inside Kubernetes — without it, the client resolver only ever sees one address, and every balancing policy downstream becomes irrelevant.

#Client-Side gRPC Load Balancing Configuration (Go)

1conn, err := grpc.Dial(
2    "dns:///inventory-grpc.prod.svc.cluster.local:9090",
3    grpc.WithDefaultServiceConfig(`{
4        "loadBalancingPolicy": "round_robin",
5        "methodConfig": [{
6            "name": [{"service": "inventory.v1.InventoryService"}],
7            "retryPolicy": {
8                "maxAttempts": 3,
9                "initialBackoff": "0.1s",
10                "maxBackoff": "1s",
11                "backoffMultiplier": 2.0,
12                "retryableStatusCodes": ["UNAVAILABLE"]
13            }
14        }]
15    }`),
16    grpc.WithTransportCredentials(creds),
17    grpc.WithKeepaliveParams(keepalive.ClientParameters{
18        Time:                20 * time.Second,
19        Timeout:             5 * time.Second,
20        PermitWithoutStream: true,
21    }),
22)

The dns:/// scheme triggers gRPC’s DNS resolver, which re-resolves on a fixed interval and feeds the address list into the balancer. Setting loadBalancingPolicy to round_robin is the single most important line here — without it, the default pick_first policy will select one address from the resolved list and stick to it for the connection’s lifetime, reproducing the exact pinning behaviour you were trying to eliminate, just one layer up the stack.

#Server-Side Connection Aging (Go gRPC Server)

1srv := grpc.NewServer(
2    grpc.KeepaliveParams(keepalive.ServerParameters{
3        MaxConnectionAge:      15 * time.Minute,
4        MaxConnectionAgeGrace: 30 * time.Second,
5        Time:                  10 * time.Second,
6        Timeout:               5 * time.Second,
7    }),
8)

MaxConnectionAge forces the server to send a GOAWAY frame after the configured interval, prompting the client to re-resolve and re-establish subchannels. Without this, a client that started when the fleet had ten pods will keep its original subchannel set indefinitely, even as the fleet scales to a hundred pods under load. The grace period allows in-flight RPCs to drain before the connection is forcibly closed, avoiding mid-request failures during rebalancing.

#L7 Proxy Alternative: Envoy Cluster Configuration

For polyglot fleets where you cannot enforce client-side configuration consistently, terminate gRPC at Envoy and let it manage upstream distribution per-stream:

1clusters:
2  - name: inventory_grpc_cluster
3    connect_timeout: 0.5s
4    type: EDS
5    lb_policy: LEAST_REQUEST
6    http2_protocol_options:
7      max_concurrent_streams: 100
8    circuit_breakers:
9      thresholds:
10        - priority: DEFAULT
11          max_connections: 1024
12          max_pending_requests: 1024
13          max_requests: 4096
14    outlier_detection:
15      consecutive_5xx: 5
16      interval: 10s
17      base_ejection_time: 30s
18    eds_cluster_config:
19      eds_config:
20        ads_config:
21          api_type: GRPC

LEAST_REQUEST here operates per-stream, not per-connection, meaning Envoy tracks in-flight requests on each upstream and routes new streams accordingly, sidestepping the pinning problem entirely since Envoy — not the client — owns the persistent TCP connections to backends. Combined with EDS (Endpoint Discovery Service) feeding live pod IPs from the control plane, this gives you sub-second rebalancing on scale events without touching client code.

#Failure Modes and Edge Cases

Fixing gRPC load balancing introduces new failure surfaces that a naive L4 setup never had to handle:

Reconnection storms. If every client shares the same MaxConnectionAge value with no jitter, a fleet of a thousand clients will attempt to re-resolve and reconnect within the same second, producing a synchronised spike against DNS and against the control plane. Always add randomised jitter (a percentage spread on the age value) to avoid herd behaviour.

Stale subchannel state during rolling deploys. During a rolling update, pods terminate and their IPs disappear from DNS, but a client that cached a resolution result and has an established subchannel in READY state may continue sending RPCs to a terminating pod until it receives a connection reset. Pair this with Kubernetes preStop hooks that sleep for a few seconds before SIGTERM, giving in-flight RPCs and DNS propagation time to catch up.

DNS TTL versus resolver polling interval mismatch. gRPC’s built-in DNS resolver polls on a fixed interval (typically 30 seconds by default), independent of the DNS record’s actual TTL. If your CoreDNS TTL is shorter than the resolver’s poll interval, you gain nothing from a fast-changing backend set — the client simply won’t notice until its next poll cycle. Tune the poll interval explicitly rather than trusting the default.

Uneven request cost under round_robin. Pure round_robin distributes connections evenly but not necessarily request cost evenly — a batch RPC and a trivial health check are treated identically. For workloads with highly variable per-RPC cost, LEAST_REQUEST or a custom weighted policy based on backend-reported load (via ORCA load reporting) is architecturally more correct than naive round_robin.

Health check false positives during GC pauses. A backend under a long garbage collection pause may fail its gRPC health check transiently, get ejected by outlier detection, and then flood back in the moment it recovers — causing an oscillation. Tune base_ejection_time and consecutive_5xx thresholds conservatively for JVM-based services with unpredictable pause profiles.

#Scaling and Security Trade-offs

Choosing between client-side gRPC load balancing and an L7 proxy tier is not free in either direction. When evaluating this against your broader architectural patterns for service communication, weigh the following:

• Operational surface area — client-side balancing pushes configuration (policy, keepalive, retry logic) into every service binary, requiring consistent libraries across languages; an L7 proxy centralises this but adds an extra network hop and a new dependency to keep healthy.
• mTLS termination point — with client-side balancing, mTLS
  The KBY Lexicon

  mTLS (Mutual Transport Layer Security)

  mTLS is a TLS handshake extension in which both the client and server present X.509 certificates to cryptographically authenticate each other before establishing an encrypted channel, rather than only the server proving its identity as in standard TLS. It forms the cryptographic identity layer underpinning zero-trust network architectures, replacing implicit network-location trust with explicit, per-connection workload authentication.

  Read Full Context
  is negotiated per-subchannel directly with each backend pod, preserving end-to-end encryption; with an L7 proxy, you must decide whether the proxy terminates and re-originates TLS (breaking end-to-end guarantees unless you run mTLS proxy-to-proxy as well), which has real compliance implications for regulated workloads.
• Control plane blast radius — xDS-driven client balancing and EDS-driven Envoy clusters both depend on a control plane (e.g. Istio’s istiod, or a custom ADS server); an outage or misconfiguration there can silently starve every client of endpoint updates, effectively re-creating the pinning problem fleet-wide.
• Connection count at scale — pure client-side round_robin against N backends means every client holds N persistent connections, which multiplies quickly (M clients × N backends) and can exhaust backend file descriptor limits; an L7 proxy tier reduces this to a bounded connection pool per proxy instance, trading client complexity for proxy-side connection pooling discipline.
• Rebalancing latency — DNS-based client resolution typically rebalances on the order of tens of seconds; EDS-driven Envoy rebalancing typically reacts within single-digit seconds of a control plane push, which matters materially during rapid autoscaling events.

Neither approach is universally correct. Latency-critical internal RPC meshes with a single dominant language runtime tend to benefit from client-side gRPC load balancing with xDS, minimising hop count. Heterogeneous, multi-language estates with strict compliance requirements around traffic inspection generally justify the proxy tier, accepting the additional hop as the cost of centralised policy and observability.