Skip to content
All writing
Engineering · 6 min read

From 19 Goroutines to Exit 137: A Go Concurrency OOM Postmortem

19 goroutines, one shared browser, exit 137. A postmortem tracing Go concurrency, GMP scheduling, memory layers, and the semaphore fix that stopped the bleeding.

19 goroutines, one shared browser, exit 137. When the Go runtime cannot see a ceiling outside its own process, you have to build the wall yourself.

The Story

Our screenshot service hit exit 137. The service renders web pages via a browser sidecar. It used to launch a fresh browser per request. Later we switched to sharing one for efficiency.

At the time I was not that familiar with how goroutines work internally. I knew GOMAXPROCS=1 was set and assumed that meant “one at a time.” It did not. 19 goroutines opened 19 browser tabs simultaneously, the browser ran out of memory, exit 137. Every in-flight screenshot request failed at once.

To understand why GOMAXPROCS=1 cannot stop 19 concurrent captures, I started from the basics (process, thread, kernel, syscall) and worked my way up. That is why this series has so many term explanations. Each one is something I had to look up while chasing the root cause.

Reading Path

Reading path of the Go OOM series Seven stages in reading order: foundations, mechanism, memory, trap, allocator, fix, and pitfalls, each with its key terms. The trap stage, one shared browser that every request fans in to, is where the OOM happened. 0 Foundations process · syscall · container · heap 1 Mechanism GMP · netpoller · GOMAXPROCS ≠ cap 1.5 Memory cgroup · cAdvisor · allocator · heap 2 Trap shared browser · fan-in · correlated OOM 2.5 Allocator size class · fragmentation · return policy 3 Fix TryAcquire → 429 · keep-alive off 4 Pitfalls GOMEMLIMIT ≠ browser mem · git archeology
Seven stages in reading order. Stage 2, the shared browser, is the trap that caused the OOM.

Read in number order. Each piece covers one concept I had to learn along the way:

Why It Blew Up: Three Conditions Intersecting

Three conditions behind exit 137 net/http spawns a goroutine per request; GOMAXPROCS caps only parallel work, so I/O-parked goroutines are uncapped; the shared browser has no cap, so 19 tabs exist at once. Together they cause a browser OOM, exit 137, and all requests die together. net/http goroutine-per-request each request auto-spawns a goroutine GOMAXPROCS only caps parallel I/O-parked goroutines are uncapped shared browser has no cap 19 tabs exist simultaneously browser OOM (exit 137) all requests die together
No single condition is enough. The browser runs out of memory only when all three hold at once.
  1. Implicit concurrency: net/http gives every request a goroutine, and the handler spawns another. Concurrency exists before you ever write go (GOMAXPROCS Caps Parallelism, Not Concurrency).
  2. GOMAXPROCS is a false ceiling: it caps goroutines running Go code, not goroutines parked on I/O. Parked goroutines hold 0 CPU / 0 P / 0 M (GMP Scheduler and Netpoller).
  3. Shared = single blast radius: one browser dies, every request using it dies. The Go runtime cannot see an out-of-process memory ceiling. Backpressure does not flow back automatically (Shared Resource Blast Radius).

The Fix

  • Add a semaphore (TryAcquire) in the screenshot service: more than 6 concurrent captures → immediate HTTP 429, no queuing (Load Shedding with TryAcquire).
  • The caller service must disable keep-alive on 429 retries. Otherwise L4 load balancing pins the retry to the same pod (Keep-Alive Pins Retries).

What Went Wrong First

An earlier attempt set GOMEMLIMIT to fix the OOM, but GOMEMLIMIT controls the Go heap, not the browser sidecar’s memory. Fix-locus (where you fix) ≠ failure-locus (where it breaks). A git log walkback revealed that accumulated architecture changes made the OOM possible over time (Wrong Fix: GOMEMLIMIT and Git Archeology).

Centralization saves resources but concentrates risk. Add the wall back, or 19 requests go down with one browser.

References:

↑↓ navigate ↵ open esc close