The Wrong Fix: GOMEMLIMIT on the Wrong Process, and Git Archeology

fix-locus (where you fix) ≠ failure-locus (where it breaks) → the fix does nothing.

Wrong Process

• Confirm fix-locus == failure-locus: same process, same resource the kernel is killing. A control that looks on-topic can be completely orthogonal.
• GOMEMLIMIT governs the Go heap only, not cgo, mmap, stacks, or other processes (runtime environment).
• Red flag: a commit titled “fix OOM” that tunes a different process than the one being killed.

Git Archeology: Why Didn’t This Happen Before?

• The bug is “new” but the code never had protection. The real question is “what changed the conditions.” git log answers that.
• Look for architecture swaps (isolated → shared), retry changes (multipliers), resource-limit edits (wrong target).
• chore: / refactor: commits can silently flip a failure mode. Diff the mechanism, not the ticket title.

Lessons

1. Before fixing, confirm the thing you’re fixing and the thing that’s dying are the same process.
2. “It never happened before” → git log --since=<date> --reverse -- <hot-file> to find what changed the conditions.
3. Look for prior “fix attempts”. They reveal the team’s mental model, and sometimes reveal the model was wrong.

The most dangerous fix is “looks related, actually orthogonal”. It passes code review without question and does nothing when the OOM hits.

References:

• https://go.dev/doc/gc-guide
• https://pkg.go.dev/runtime/debug
• https://git-scm.com/docs/git-bisect
• https://blog.xargs.io/2014/03/15/git-archeology-find-the-secrets-of-those-who-came-before/
• https://gun.io/news/2025/05/git-bisect-debugging-guide/

Related: see shared resource blast radius and runtime environment, or go back to the series overview.