Keep-Alive Pins Retries: 429 Retry Without a New Connection Hits the Same Pod Forever
429 says I'm full, but keep-alive makes your retries knock on the same door. Disable it so L4 load balancing can re-pick a pod with capacity.
429 says “I’m full.” But keep-alive makes your retries knock on the same door every time.
L4 Load Balancing’s Blind Spot
- K8s Service defaults to L4 load balancing: the routing decision is made once at TCP connection setup (runtime environment).
- With HTTP keep-alive, all requests reuse the same connection → same pod. A 429 retry keeps hitting the full pod.
- Fix: disable keep-alive (fresh connection per retry attempt) so kube-proxy / iptables re-picks. L4 re-pick is random → pair with retry + jitter.
Role in This OOM
The caller service’s HTTP client defaulted to connection reuse. After the screenshot service returned 429, retries all went back to the same pod, making the 429 useless. Setting max_keepalive_connections=0 opened a new TCP connection per retry → kube-proxy could pick a pod with capacity (load shedding with TryAcquire).
429 + keep-alive = knocking on the same locked door forever. 429 + new connection = a chance to find an open one.
References:
Related: see load shedding with TryAcquire and shared resource blast radius, or go back to the series overview.