Engineering Aug 18, 2026 · 3 min read

From 19 Goroutines to Exit 137: A Go Concurrency OOM Postmortem

19 goroutines, one shared browser, exit 137. A postmortem tracing Go concurrency, GMP scheduling, memory layers, and the semaphore fix that stopped the bleeding.

19 goroutines, one shared browser, exit 137. When the Go runtime can’t see a ceiling outside its own process, you have to build the wall yourself.

The Story

Our screenshot service hit exit 137. The service renders web pages via a browser sidecar. It used to launch a fresh browser per request; later we switched to sharing one for efficiency.

At the time I wasn’t that familiar with how goroutines work internally. I knew GOMAXPROCS=1 was set and assumed that meant “one at a time.” It didn’t. 19 goroutines opened 19 browser tabs simultaneously, the browser ran out of memory, exit 137. Every in-flight screenshot request failed at once.

To understand why GOMAXPROCS=1 couldn’t stop 19 concurrent captures, I started from the basics (process, thread, kernel, syscall) and worked my way up. That’s why this series has so many term explanations. Each one is something I had to look up while chasing the root cause.

Reading Path

Foundations (0)    Mechanism (1)     Memory (1.5)        Trap (2)         Allocator (2.5)    Fix (3)        Pitfalls (4)
process, thread    GMP scheduler     cgroup accounting   shared browser   latency/size-cls   semaphore      wrong target
kernel, syscall    netpoller/epoll   cAdvisor metrics    → correlated OOM fragmentation      TryAcquire→429 GOMEMLIMIT ≠
socket, container  goroutine-per-req allocator concept   backpressure gap return policy      keep-alive off   browser mem
stack, heap, cgo   GOMAXPROCS ≠ cap  heap metrics        fan-in on shared                    shed, not queue git archeology

Read in number order. Each piece covers one concept I had to learn along the way:

OS Vocabulary: Process to Syscall - process / thread / kernel / syscall
Runtime Environment: Socket, Container, Stack, Heap, cgo - socket / container / sidecar / stack / heap / cgo
GMP Scheduler and Netpoller - how goroutines run and wait on I/O
GOMAXPROCS Caps Parallelism, Not Concurrency - why cpu:1 doesn’t stop 19
Memory Three-Layer Model: cgroup to Heap - cgroup → allocator → heap: three layers of truth
OOM Diagnostic Decision Tree - a top-down decision tree starting at pprof
Shared Resource Blast Radius - why sharing one browser = everyone dies together
Allocator Internals: Size Class and Fragmentation - how the allocator works, where fragmentation comes from
Load Shedding with TryAcquire - semaphore + 429 design
Keep-Alive Pins Retries - why 429 retries hit the same pod
Wrong Fix: GOMEMLIMIT and Git Archeology - tuning the wrong process

Why It Blew Up: Three Conditions Intersecting

net/http (goroutine-per-request)      each request auto-spawns a goroutine
        × GOMAXPROCS only caps parallel   I/O-parked ones are uncapped
        × shared browser has no cap       19 tabs exist simultaneously
        = browser OOM (exit 137)          all requests die together

Implicit concurrency: net/http gives every request a goroutine, and the handler spawns another. Concurrency exists before you ever write go (GOMAXPROCS Caps Parallelism, Not Concurrency).
GOMAXPROCS is a false ceiling: it caps goroutines running Go code, not goroutines parked on I/O. Parked goroutines hold 0 CPU / 0 P / 0 M (GMP Scheduler and Netpoller).
Shared = single blast radius: one browser dies, every request using it dies. The Go runtime can’t see an out-of-process memory ceiling; backpressure doesn’t flow back automatically (Shared Resource Blast Radius).

The Fix

Add a semaphore (TryAcquire) in the screenshot service: more than 6 concurrent captures → immediate HTTP 429, no queuing (Load Shedding with TryAcquire).
The caller service must disable keep-alive on 429 retries; otherwise L4 load balancing pins the retry to the same pod (Keep-Alive Pins Retries).

What Went Wrong First

An earlier attempt set GOMEMLIMIT to fix the OOM, but GOMEMLIMIT controls the Go heap, not the browser sidecar’s memory. Fix-locus (where you fix) ≠ failure-locus (where it breaks). A git log walkback revealed that accumulated architecture changes made the OOM possible over time (Wrong Fix: GOMEMLIMIT and Git Archeology).

Centralization saves resources but concentrates risk. Add the wall back, or 19 requests go down with one browser.

References:

Tags #go #concurrency #reliability