A Golang Upgrade That Triggered a Debugging Chain Reaction
An automated dependency update bumped the Golang version and cascaded through Docker, Alpine, and libvips. Four layers of verification agreed upstream was broken, but all four used the same flawed input.
Four layers of verification agreed “upstream is broken,” but all four used the same flawed input. Four consistent conclusions aren’t four pieces of evidence. They’re one piece of evidence counted four times.
Dependabot (GitHub’s bot that auto-opens PRs to bump your dependencies) bumped go.mod (Golang’s dependency file) from 1.20 to 1.25. Looked like a language version upgrade and nothing more. But golang:1.25-alpine ships a newer Alpine, newer Alpine ships a newer libvips, and the newer cgif encoder in that libvips merges identical frames. A GIF frame-count test went red, and a chain reaction began.
1: The Version Trap
Golang × Alpine is a sliding window, not a fixed pairing. Bumping the Golang version shifts the entire dependency chain. Before hand-composing a base image, query the tag matrix to check if the combination you need is already published. Only when the matrix genuinely lacks your combo is there a reason to graft the Golang toolchain onto a different base.
2: Finding the Culprit (and How It Fooled Us)
We dropped down the layers: app → wrapper → raw CLI → version bisect (binary-searching releases to find which one broke). Every layer was red. It looked like ironclad proof: libvips 8.18 had regressed (broke something that used to work).
The problem was that every layer was fed the same identical-frame test image. A contaminated test oracle made four layers of verification tell the same lie. It wasn’t until we recognized that GIF encoders legally merge identical frames and swapped to a differing-frame fixture (test input data) that the same libvips version turned green instantly.
3: The Discipline That Would Have Caught It
A test result is a fact about its input, not about the system. When a test verdict is about to drive a structural decision (pinning a version, switching a base image, introducing a graft), re-derive from a production-shaped input first. This gate almost never fires, but it fires exactly when being wrong is most expensive.
4: Containment (Corrected)
We pinned to alpine:3.20 with a textbook exit note. But 3.20 was already EOL (End of Life — no longer receiving security patches), and apk upgrade silently returned exit 0 with zero patches applied. A correctness pin traded away a real stream of CVE fixes (publicly disclosed security vulnerabilities) for a fix that didn’t exist.
Both gates were skipped, and either one would have stopped the episode on its own: Gate 1 because the “regression” wasn’t real, Gate 2 because 3.20 was EOL.
Resolution
The upgrade was real; the GIF “regression” was not. Final ship: supported golang:1.25.11-alpine3.22, graft removed, tests switched to differing-frame fixtures.
The bug was in the fixture, and every layer of rigor applied downstream of it (bisect, pin, graft) inherited the error and made it look better-founded.
Sources: