Crop, Resize, Fill: Four Image-Processing Lessons Learned

Summary

• Scenario
  • I picked up an image-processing service for ad creatives (Go + libvips) and had to add three parameter-driven transforms: crop, resize-to-fit, and fill. The upstream passes down a crop box, a target size, and a fill color, and the service renders each asset to the placement spec. But in testing, one kind of asset, GIF, hit a serious crop problem. That kicked off a whole chain of deeper digging into the implementation and into what an “image” even is.

• Debugging train of thought

  • Once the three transforms were written, ordinary JPEG and PNG passed fine. The moment I fed in an animated GIF, resize still animated. crop and fill came out garbled, and sometimes only the first frame survived with the animation gone.
  • My first guess was mismatched frame sizes, or a coordinate I miscalculated somewhere. Logging revealed nothing. What actually blocked me was a naive model of “what an image is”. I assumed a picture is just a grid of pixels, and that crop, resize, and fill are just coordinate math.
  • Digging into libvips, the underlying C library, showed the real shape. It stores an animation as one tall stacked strip (a page-strip), not as N separate frames. And the default load reads only the first page, so it never decodes the rest. resize is safe, crop breaks, and the only difference is whether the transform looks at coordinates.

• Motivation

  • Instead of guessing every time a format problem showed up, I wanted to relearn images from “which dimensions does a file actually have.” I also wanted to turn the scattered facts into one mental model. This way, my handling of “images” in code comes from a deeper understanding.

• Goal

  • One processing path that correctly handles stills and animation together.
  • The path knows what I trade off when I pick an output format.
  • The path never silently ships a wrong image when the animation breaks.

Reject beats silently wrong. The painful part this time was not “crop breaks.” It was that the animation loaded as a single frame with no error at all. The program kept running, output a still, and nobody noticed the animation was gone. There is an engineering rule I hold onto. A path that can produce a “looks fine but is wrong” result must fail loudly rather than pass quietly. So afterward I propagated the Export and Resize errors upward, and made “fill without resize” report itself instead of getting swallowed.

The counter-intuitive turn: resize is safe, crop breaks

Same batch of transforms, so why is resize fine while crop and fill break? It comes down to one question: does this transform care where a pixel is?

• resize does the same thing to every pixel regardless of which frame it sits in. So scaling the whole strip scales every frame together and the animation stays alive.
• crop and fill act on a specific coordinate or region. On the strip, that coordinate points into a pile of stacked frames, not “the same spot in each frame.” So one cut slices across a frame boundary and produces garbage.

The roadmap I took

1. Break the “image = pixel grid” illusion first, and see the independent dimensions a file carries. → An Image Is More Than a Grid of Pixels
2. Get the format trade-offs straight: capabilities, where size comes from, how to pick under ad serving. → Image Format Cheat Sheet, Lossy vs Lossless Drives File Size, Ad Serving Flips the Format Priority
3. Understand how libvips stores animation, which explains why crop breaks. → libvips Stores an Animation as One Tall Stacked Strip
4. Fix the load step so stills and animation take the same path. → One Load Path for Stills and Animation
5. Use “uniform vs positional” to tell which transforms run on the whole strip and which go frame by frame. → Uniform vs Positional Transforms on a Stitched Strip

Lesson 1: An image is more than a grid of pixels

The “an image is one picture” model breaks the moment you touch real files. An image file carries several independent dimensions at once: frame count, transparency, color storage, lossy vs lossless, raster vs vector, bit depth, and hidden metadata. Each one has a format that violates the naive assumption: JPEG has no alpha, GIF is a 256-color palette, and a GIF’s per-frame delay lives in metadata. Asking which dimensions a file uses before processing saves a lot of debugging later.

Before processing, ask which dimensions this file actually uses.

→ Deeper dive: An Image Is More Than a Grid of Pixels (plus the format homework: cheat sheet, lossy vs lossless, ad-serving priority)

Lesson 2: An animation is one tall stacked strip

libvips does not treat an animation as N frames. It stacks every frame vertically into one tall image (officially a page-strip, and I picture a toilet roll). 5 frames of 100px make one 500px image. The structure lives in metadata: page-height (one frame’s height), n-pages (frame count), delay (ms per frame), with n-pages × page-height = total height. So Height() returns the strip height, and Pages() > 1 is how you know you are holding a strip rather than a still. I originally guessed the frames were different sizes. The opposite is true: every frame is identical in size, which is exactly why libvips can stack them neatly.

Pages() > 1 is what tells you it is animated, not the file extension.

→ Deeper dive: libvips Stores an Animation as One Tall Stacked Strip

Lesson 3: The default reads only the first page, so the animation just vanishes

The loader’s n defaults to 1, so feeding in a 5-frame GIF decodes only page 0. The loader never decodes the other 4 frames: no exception, no warning, just a quietly dropped animation and a still output. The fix is to pass n=-1 at load (NumPages.Set(-1) in govips) to read every frame at once. That also turns loading into one unconditional path: a still returns Pages()==1, an animation returns Pages()==5, and passing -1 on a still does not error. Single vs multi is decided after the load with Pages().

A default is not neutral. It will quietly drop the frames you did not read.

→ Deeper dive: One Load Path for Stills and Animation

Lesson 4: Uniform transforms run on the whole strip, positional ones go frame by frame

Back to that broken crop. Whether a transform can run on the whole strip depends only on whether it looks at coordinates. resize treats every pixel identically (uniform), so scaling the whole strip is fine. crop and fill target a coordinate or region (positional), so on the strip they cross frame boundaries. Those need to:

• load with [n=-1]
• split the strip by page-height
• apply the operation identically to each frame
• vips_arrayjoin them back
• then restore page-height, n-pages, and delay

Scales or recolors everything the same means uniform and safe on the whole strip. Targets a coordinate means positional, so go frame by frame.

→ Deeper dive: Uniform vs Positional Transforms on a Stitched Strip

Closing

The four lessons are really one: an animation is not a pile of frames. It is one image stacked out of them. Both “loading it” and “editing it” have to know that. resize happened to be harmless on the whole strip, which is what fooled me into thinking the problem was just bad coordinate math.

Questions I ask myself now before touching image processing:

• Does this path handle stills and animation the same way, and does the load read every frame in?
• Does the transform I’m about to run look at coordinates? If so, am I still running it on the whole strip?
• When the animation or the alpha channel drops, does the program fail loudly, or quietly ship an image that looks fine?