$ blog / engineering

Engineering

Lessons from what I run into at work, learned from and written down.

Series

Series · 12 parts Aug 2026

From 19 Goroutines to Exit 137

19 goroutines, one shared browser, exit 137. A postmortem tracing Go concurrency, GMP scheduling, memory layers, and the semaphore fix that stopped the bleeding.

00From 19 Goroutines to Exit 137: A Go Concurrency OOM Postmortem(overview)01OS Vocabulary: Process, Thread, Kernel, Syscall 02Runtime Environment: Socket, Container, Stack, Heap, cgo 03GMP Scheduler and Netpoller: I/O Waiting Costs Zero CPU 04GOMAXPROCS Caps Parallelism, Not Concurrency 05Memory Three-Layer Model: cgroup to Allocator to Heap 06OOM Diagnostic Decision Tree: Drill Down from pprof 07Shared Resource Blast Radius: When One Dies, Everyone Dies 08Allocator Internals: Size Class, Fragmentation, Scavenger 09Load Shedding with TryAcquire: Fast Rejection Beats Slow Queuing 10Keep-Alive Pins Retries: 429 Retry Without a New Connection Hits the Same Pod Forever 11The Wrong Fix: GOMEMLIMIT on the Wrong Process, and Git Archeology

Series · 10 parts Aug 2026

A Golang Upgrade's Chain Reaction

An automated dependency update bumped the Golang version and cascaded through Docker, Alpine, and libvips, teaching nine lessons about version coupling, contaminated test oracles, and silent security gaps.

00A Golang Upgrade That Triggered a Debugging Chain Reaction(overview)01Query the Tag Matrix Before Composing a Base Image 02Graft the Golang Toolchain Onto the Base Image You Need 03Isolate a Regression by Dropping Down the Layers 04A Contaminated Test Oracle Inverts Your Diagnosis 05GIF Encoders Legally Merge Identical Frames 06A Test Result Is FACT Only About Its Input 07Pin to the Last-Good Version With an Exit Plan 08An EOL Base Image Silently No-Ops Security Upgrades 09Codify the Lessons Before They Fade

Series · 10 parts Jul 2026

Reaching a Remote DB From Your Laptop

One mental model for reaching a remote database from your laptop: the five coordinates, the tunnel, docker packaging, and one-shot vs REPL queries.

00Reaching a Remote DB From Your Laptop: One Mental Model(overview)01A Database Connection Is Just Five Coordinates 02The Connection URI Is Canonical; Flags Are It Decomposed 03Two Tunnels to a Remote DB: Cloud SQL Auth Proxy vs kubectl port-forward 04docker-compose image vs build: Provenance, Not Existence 05Build-Time COPY Bakes Files In; Volumes Mount Them Live 06One-Shot Query vs Interactive REPL: DB CLIs Fork on One Flag 07The MySQL CLI: Remote Connection and Flags 08The MySQL CLI: Four Ways to Feed Inline SQL 09sqlit: a Keyboard-Only DBeaver, and the Whole Model on One Page

Series · 7 parts Jul 2026

PKI & TLS Behind a Migration Ticket

A one-line migration ticket hid two opposite-facing TLS setups. Untangling the PKI behind it — the certificates, keys, and CAs — from inbound one-way TLS to outbound mTLS to Kafka.

00Two SSLs Behind One Migration Ticket(overview)01TLS Basics, and Why There Are Three Certificate Files 02One-Way TLS vs Mutual TLS (mTLS)03Kafka Ports: 9092 (Plain) vs 9093 (Encrypted)04The Kafka Client's SSL Config, and the Absence-Gating Trick 05Ingress TLS Termination: Encrypting at the Front Door 06Delivering Certs to a Pod: A Two-Stage Deploy Render

Series · 8 parts Jul 2026

Crop, Resize, Fill: Image Processing

A GIF crop bug that rendered garbage taught how images and animation really work — format trade-offs, libvips page-strips, and one load path for stills and animation.

00Crop, Resize, Fill: Four Image-Processing Lessons Learned(overview)01An Image Is More Than a Grid of Pixels 02Image Format Cheat Sheet 03Lossy vs Lossless Drives File Size 04Ad Serving Flips the Format Priority 05libvips Stores an Animation as One Tall Stacked Strip 06One Load Path for Stills and Animation 07Uniform vs Positional Transforms on a Stitched Strip

Series · 10 parts Jul 2026

One Operation, Many Engines

One operation, three engines: query ClickHouse, Spark, and MongoDB by thinking in intent instead of memorizing three sets of syntax.

00One Operation, Many Engines: A Cross-Engine Query Mental Model(overview)01ClickHouse Data Types: The Set You'll Actually Hit 02Spark SQL Data Types: Primitives, Nested, and struct vs map 03The Portable SQL Core: The 80% That Runs Anywhere 04count(*) vs count(col): It Comes Down to NULL 05LIKE vs ILIKE: Substring Search (and Case) Across SQL Engines 06The ClickHouse Array Functions You'll Actually Reach For: has, arrayMap, multiIf 07Spark Array Functions & collect_list vs collect_set 08From SQL to MongoDB: Operators and Vocabulary, Mapped 09Cross-Engine Query Exercises: Test Yourself (With Answers)

Series · 5 parts Jul 2026

Thread-Safe Login, the Hard Way

A production race condition taught four lessons about locks, atomicity, and safe publication.

00Making Login Thread-Safe: Four Lessons a Race Condition Taught Me(overview)01A Mutex Gives You ACID's 'I', and Nothing Else 02Finish First, Then Publish: Never Ship a Half-Built Object 03Python's `with lock:` Always Releases, Even on Exception 04Double-Checked Locking: Two Identical `if`s, Two Different Jobs