Remote-Controlling Chrome in Containers: The Full Journey
Splitting Chrome from an API pod for KEDA autoscaling broke DevTools, DNS routing, and capacity planning. Here are the 12 lessons from that breakage.
“The fastest way to understand a system is to watch it break.”
The capture service was growing. The original architecture put the API server and a sidecar Chromium inside one pod. One pod, no horizontal scaling. When burst traffic hit, that single Chrome became the bottleneck.
We split API and Chrome into separate pods, Chrome behind KEDA for autoscaling. Straightforward in theory. But the moment we split, Chrome upgraded to version 128 and ignored its own --remote-debugging-address flag. DevTools locked to localhost. Cross-pod calls stopped working.
From that breakage, we traced a path through networking, protocol design, container startup, load balancing, security, and capacity testing.
This is the tour. Each section names the problem and the fix, then links to the full deep dive.
A Pod Is a Phone
In Kubernetes, a pod is a phone. Apps on the same phone talk through a group chat (127.0.0.1). Apps on different phones must dial the extension number (pod IP). After Chrome version 128, Chrome only listens to the group chat. Outside pods cannot reach it.
Deeper dive: Network basics: IP, port, localhost.
How Chrome Gets Remote-Controlled
Chrome DevTools Protocol (CDP) needs two steps: an HTTP call to get a GUID, then a WebSocket connection using that GUID. Both steps must hit the same Chrome process. This constraint bites hard when you scale.
Deeper dive: CDP two-phase connection.
Chrome Version 128: The Flag That Does Nothing
Chrome version 128 ignores --remote-debugging-address=0.0.0.0. DevTools locks to 127.0.0.1. Same pod: fine. Cross-pod: broken.
Deeper dive: Why Chrome ignores its own flag.
First Fix: A Receptionist Inside the Pod
socat acts as a receptionist. It sits at the front desk (0.0.0.0:9222) and accepts outside calls. It walks each call to Chrome’s group chat (127.0.0.1:9223) and carries the answer back. Chrome thinks socat is a local app.
Deeper dive: How socat bridges the gap.
The Morning Checklist: Container Startup
The entrypoint script is an opening checklist: start socat, make sure that it is alive, then hand the PID 1 badge to Chrome with exec. The port config has a five-layer chain. Helm’s default() can silently swallow a missing value.
Deeper dives: PID 1 handoff and The five-layer config chain.
GUID Race: The Scaling Problem
KEDA adds three pods. kube-proxy picks a random pod per connection. CDP’s two steps land on different pods. The GUID does not match. Three pods: 33% success. Six pods: 17%.
Deeper dive: The math behind the failure.
Headless Service: A Phone Directory Instead of a Switchboard
A regular Service is a switchboard that routes randomly. A headless Service is a phone directory: you get all extension numbers and pick one yourself. Both steps hit the same Chrome. The GUID always matches.
Deeper dive: DNS pinning fixes the GUID race.
Security: Locking the Unauthenticated Front Door
socat is open to the entire cluster. CDP has no password. One NetworkPolicy restricts ingress to only the render API on port 9222.
Deeper dive: One rule locks the front door.
How Much Traffic Can It Handle?
max concurrent = per-pod ceiling × Ready pod count. Three pods handle 150. Push to 200 and it breaks. “Ready” does not mean “usable”: a GPU node reports Ready before the driver finishes installing.
Deeper dives: The capacity formula and Pressure testing and initialization lag.
Why Fresh TCP Matters
Go’s HTTP client reuses connections by default. When KEDA scales Chrome pods down, a reused connection points to a dead pod. DisableKeepAlives: true forces a fresh DNS lookup on every request. Three defenses work together: headless Service, atomic round-robin, and fresh TCP.
Deeper dive: Why keep-alive breaks scaling.
“Every layer of abstraction hides complexity for you. Until it starts hiding bugs for you.”
References: