A Golang Upgrade That Triggered a Debugging Chain Reaction
An automated dependency update bumped the Golang version and cascaded through Docker, Alpine, and libvips. Four layers of verification agreed upstream was broken, but all four used the same flawed input.
Four layers of verification agreed “upstream is broken,” but all four used the same flawed input. Four consistent conclusions are not four pieces of evidence. They are one piece of evidence counted four times.
Dependabot (GitHub’s bot that auto-opens PRs to bump your dependencies) bumped go.mod (Golang’s dependency file) from 1.20 to 1.25. Looked like a language version upgrade and nothing more. But golang:1.25-alpine ships a newer Alpine, newer Alpine ships a newer libvips, and the newer cgif encoder in that libvips merges identical frames. A GIF frame-count test went red, and a chain reaction began.
1: The Version Trap
Golang × Alpine is a sliding window, not a fixed pairing. Bumping the Golang version shifts the entire dependency chain. Before hand-composing a base image, query the tag matrix to check if the combination you need is already published. Only when the matrix genuinely lacks your combo is there a reason to graft the Golang toolchain onto a different base.
2: Finding the Culprit (and How It Fooled Us)
We dropped down the layers: app → wrapper → raw CLI → version bisect (binary-searching releases to find which one broke). Every layer was red. It looked like ironclad proof: libvips 8.18 had regressed (broke something that used to work).
The problem was that every layer was fed the same identical-frame test image. A contaminated test oracle made four layers of verification tell the same lie. We recognized that GIF encoders legally merge identical frames. We swapped to a differing-frame fixture (test input data), and the same libvips version turned green instantly.
3: The Discipline That Can Catch It
A test result is a fact about its input, not about the system. A test verdict can drive a structural decision: a version pin, a base-image switch, or a graft. Before that happens, re-derive from a production-shaped input first. This gate almost never fires, but it fires exactly when being wrong is most expensive.
4: Containment (Corrected)
We pinned to alpine:3.20 with a textbook exit note. But 3.20 was already EOL (End of Life, no longer receiving security patches), and apk upgrade silently returned exit 0 with zero patches applied. A correctness pin traded away a real stream of CVE fixes (publicly disclosed security vulnerabilities) for a fix that did not exist.
Both gates were skipped. Either one can stop the episode on its own: Gate 1 because the “regression” was not real, Gate 2 because 3.20 was EOL.
Resolution
The upgrade was real. The GIF “regression” was not. Final ship: supported golang:1.25.11-alpine3.22, graft removed, tests switched to differing-frame fixtures.
The bug was in the fixture. Every layer of rigor applied downstream of it (bisect, pin, graft) inherited the error and made it look better-founded.
Sources: