Memory Three-Layer Model: cgroup to Allocator to Heap

RSS larger than heap inuse is normal. An allocator sits between kernel and heap, holding memory it hasn't returned. Three layers, three metrics.

RSS > heap inuse is normal. There’s an allocator in between, and it doesn’t give memory back immediately.

Three Layers, Three Sets of Metrics

cgroup Layer

memory.current = all physical memory charged to this container: RSS + kernel allocations (file cache, socket buffers). memory.max = hard ceiling; exceed it = OOM Kill (exit 137).
Don’t just watch RSS: kernel-charged memory (many open sockets, file cache) can push past memory.max too.

Allocator Layer

new(obj) doesn’t syscall. The allocator pre-fetches pages from the kernel and slices them into size-class slots. When GC frees an object, the slot returns to the allocator, not to the kernel, so pprof inuse_space drops while RSS stays.

Heap Layer (cAdvisor + Go runtime)

container_memory_usage_bytes ≈ cgroup memory.current. container_memory_working_set_bytes = what K8s eviction uses.
heap_alloc = live objects in precise bytes. heap_inuse − heap_alloc = fragmentation (live objects pinning full spans). heap_idle − heap_released = pages the allocator holds but hasn’t returned to the kernel.

Role in This OOM

The browser’s memory isn’t in any of these three layers. It’s a separate process with its own cgroup account. A small Go heap doesn’t mean the pod is safe; you also need to watch the browser sidecar’s container_memory_usage_bytes (OOM Diagnostic Decision Tree).

Memory has three layers of truth: application (pprof) → process (RSS + allocator) → kernel (cgroup). Whichever layer is large tells you where to look.

References:

Related: see OOM Diagnostic Decision Tree for a top-down diagnostic starting at pprof, Allocator Internals for how fragmentation happens, or go back to the series overview.

Tags #go #concurrency #reliability