Fernando Simões encountered unexpected 40ms garbage collection pauses in a Go production service, prompting an investigation that revealed swap usage as the root cause. The issue arose in a cgroup containing two processes: a Go application performing io.ReadAll and proto.Unmarshal operations to create blob and graph structures, and a mostly idle HTTP server. During garbage collection, the Go runtime must read its metadata structures located outside the heap during stop-the-world pauses. Simões found that under memory pressure, the kernel's LRU-based page eviction moved these metadata pages to swap. When the garbage collector attempted to access them, each page triggered a major page fault requiring the kernel to read page table entries, invoke do_swap_page, allocate a new frame, charge it to the cgroup, read the page contents, submit a bio request, wait for disk I/O completion, and restore the page to memory. To diagnose the problem, Simões used a BPF script to count page faults during stop-the-world events. The worst case showed 39,902 microseconds of the 40ms pause consumed by page faults: 228 faults accounting for 39,013 microseconds spent in fault handling. The addr2line output traced these faults to GC bookkeeping functions including runtime.(*spanSet).reset, runtime.finishsweep_m, and runtime.gcStart.func2. The garbage collector performs stop-the-world pauses at sweep termination and mark termination points. In a 30-minute test, 312 such pauses occurred, with the 40ms delay representing 800 times the median pause of 51 microseconds observed when metadata remained in memory. Since every processor (P) stops during these pauses, any goroutine waiting for I/O could miss completion signals, causing downstream latency. Additionally, Simões noted that building a 511KiB message increased from 3-5ms to 105ms on NVMe and 903ms on Hetzner's network volume, though he could not confirm whether this slowdown stemmed from the same swap mechanism. An update tested with Go 1.26's Green Tea garbage collector showed negligible impact on the swap-related pause issue. The experiments, including plots, mock allocator, BPF and Python scripts, are available at https://github.com/frnsimoes/go-gc-swap-cost. The core finding remains: swap and Go's garbage collector are incompatible for low-latency workloads, as OS-level memory pressure can induce unbounded stop-the-world pauses through metadata paging.
HN Digest / · HN #49959654
A 40ms Go Garbage Collector Pause Traced to Swap-Paged Metadata
A Go developer discovered that swap usage can cause 40ms stop-the-world garbage collector pauses when GC metadata gets paged out, revealing a hidden latency risk in production systems.

Screenshot of the source page (frn.sh)