From 19 Goroutines to Exit 137: A Go Concurrency OOM Postmortem
19 goroutines, one shared browser, exit 137. A postmortem tracing Go concurrency, GMP scheduling, memory layers, and the semaphore fix that stopped the bleeding.
19 goroutines, one shared browser, exit 137. When the Go runtime cannot see a ceiling outside its own process, you have to build the wall yourself.
The Story
Our screenshot service hit exit 137. The service renders web pages via a browser sidecar. It used to launch a fresh browser per request. Later we switched to sharing one for efficiency.
At the time I was not that familiar with how goroutines work internally. I knew GOMAXPROCS=1 was set and assumed that meant “one at a time.” It did not. 19 goroutines opened 19 browser tabs simultaneously, the browser ran out of memory, exit 137. Every in-flight screenshot request failed at once.
To understand why GOMAXPROCS=1 cannot stop 19 concurrent captures, I started from the basics (process, thread, kernel, syscall) and worked my way up. That is why this series has so many term explanations. Each one is something I had to look up while chasing the root cause.
Reading Path
Read in number order. Each piece covers one concept I had to learn along the way:
- OS Vocabulary: Process to Syscall - process / thread / kernel / syscall
- Runtime Environment: Socket, Container, Stack, Heap, cgo - socket / container / sidecar / stack / heap / cgo
- GMP Scheduler and Netpoller - how goroutines run and wait on I/O
- GOMAXPROCS Caps Parallelism, Not Concurrency - why cpu:1 does not stop 19
- Memory Three-Layer Model: cgroup to Heap - cgroup → allocator → heap: three layers of truth
- OOM Diagnostic Decision Tree - a top-down decision tree starting at pprof
- Shared Resource Blast Radius - why sharing one browser = everyone dies together
- Allocator Internals: Size Class and Fragmentation - how the allocator works, where fragmentation comes from
- Load Shedding with TryAcquire - semaphore + 429 design
- Keep-Alive Pins Retries - why 429 retries hit the same pod
- Wrong Fix: GOMEMLIMIT and Git Archeology - tuning the wrong process
Why It Blew Up: Three Conditions Intersecting
- Implicit concurrency:
net/httpgives every request a goroutine, and the handler spawns another. Concurrency exists before you ever writego(GOMAXPROCS Caps Parallelism, Not Concurrency). - GOMAXPROCS is a false ceiling: it caps goroutines running Go code, not goroutines parked on I/O. Parked goroutines hold 0 CPU / 0 P / 0 M (GMP Scheduler and Netpoller).
- Shared = single blast radius: one browser dies, every request using it dies. The Go runtime cannot see an out-of-process memory ceiling. Backpressure does not flow back automatically (Shared Resource Blast Radius).
The Fix
- Add a semaphore (
TryAcquire) in the screenshot service: more than 6 concurrent captures → immediate HTTP 429, no queuing (Load Shedding with TryAcquire). - The caller service must disable keep-alive on 429 retries. Otherwise L4 load balancing pins the retry to the same pod (Keep-Alive Pins Retries).
What Went Wrong First
An earlier attempt set GOMEMLIMIT to fix the OOM, but GOMEMLIMIT controls the Go heap, not the browser sidecar’s memory. Fix-locus (where you fix) ≠ failure-locus (where it breaks). A git log walkback revealed that accumulated architecture changes made the OOM possible over time (Wrong Fix: GOMEMLIMIT and Git Archeology).
Centralization saves resources but concentrates risk. Add the wall back, or 19 requests go down with one browser.
References: