Memory Three-Layer Model: cgroup to Allocator to Heap
RSS larger than heap inuse is normal. An allocator sits between kernel and heap, holding memory it hasn't returned. Three layers, three metrics.
RSS > heap inuse is normal. There is an allocator in between, and it does not give memory back immediately.
Three Layers, Three Sets of Metrics
cgroup Layer
memory.current= all physical memory charged to this container: RSS + kernel allocations (file cache, socket buffers).memory.max= hard ceiling. Exceed it = OOM Kill (exit 137).- Do not just watch RSS: kernel-charged memory (many open sockets, file cache) can push past
memory.maxtoo.
Allocator Layer
new(obj)does not syscall. The allocator pre-fetches pages from the kernel and slices them into size-class slots. When GC frees an object, the slot returns to the allocator, not to the kernel, sopprof inuse_spacedrops while RSS stays.
Heap Layer (cAdvisor + Go runtime)
container_memory_usage_bytes≈ cgroupmemory.current.container_memory_working_set_bytes= what K8s eviction uses.heap_alloc= live objects in precise bytes.heap_inuse − heap_alloc= fragmentation (live objects pinning full spans).heap_idle − heap_released= pages the allocator holds but has not returned to the kernel.
Role in This OOM
The browser’s memory is not in any of these three layers. It is a separate process with its own cgroup account. A small Go heap does not mean the pod is safe. You also need to watch the browser sidecar’s container_memory_usage_bytes (OOM Diagnostic Decision Tree).
Memory has three layers of truth: application (pprof) → process (RSS + allocator) → kernel (cgroup). Whichever layer is large tells you where to look.
References:
- https://man7.org/linux/man-pages/man7/cgroups.7.html
- https://docs.kernel.org/admin-guide/cgroup-v2.html
- https://github.com/google/cadvisor/blob/master/docs/storage/prometheus.md
- https://kubernetes.io/docs/concepts/scheduling-eviction/node-pressure-eviction/
- https://pkg.go.dev/runtime#MemStats
- https://go.dev/doc/gc-guide
Related: see OOM Diagnostic Decision Tree for a top-down diagnostic starting at pprof, Allocator Internals for how fragmentation happens, or go back to the series overview.