A Contaminated Test Oracle Inverts Your Diagnosis
When the test you trust to judge a change is itself defective, every result it returns is evidence for the wrong conclusion. More rigor just confirms the error.
A GIF frame-count test went red. Three layers of verification all pointed at “libvips 8.18 regressed” (broke something that used to work). Then I swapped to a differing-frame test image: same libvips version, instantly all green. The problem was never in the code; it was in the image we fed it.
When the test you trust to judge a change is itself defective, every result it returns is evidence for the wrong conclusion. Re-running it more carefully just confirms the error with more confidence.
| fixture (test input) | verdict | what you then conclude |
|---|---|---|
| identical-frame (synthetic) | RED | “libvips 8.18 regressed” → pin 3.20, graft, EOL base |
| differing-frame (production-shaped) | GREEN | “8.18 is fine” → ship 3.22, fix the fixtures |
Same libvips 8.18.4 both times: only the input changed.
The oracle never malfunctioned; it answered a question nobody meant to ask. That’s why more rigor made things worse: rigor multiplies whatever the oracle is measuring, and here it was measuring the fixture.
- The tell: a red test drives a big, structural conclusion (“the upstream lib is broken”), but the test’s input is synthetic and never resembled production.
- Rigor on a bad oracle diverges from truth. A three-row version table looks thorough, but if all rows share one flawed input, they all lie the same way.
- Break it by attacking the INPUT, not by re-running. Feed a production-shaped input and see if the verdict flips.
- The contradicting signal is usually present but buried. Here it was a log line proving frames were written (see GIF encoders legally merge identical frames). Hunt for the fact that shouldn’t exist if your diagnosis were true.
The question the oracle answered, and the question you thought you were asking, are not necessarily the same.
Related: Back to the chain reaction overview, or see isolating a regression by layer-drop, GIF encoders merging frames, or a test result is FACT only about its input.