Cloudflare’s Pingora Backend Router routes cacheable requests to servers using consistent hashing, a technique that maps servers and tasks onto a number line so that adding or removing nodes causes minimal reshuffling. Engineer Ivan noticed that the pingora‑ketama library, which implements this hashing for PBR, was consuming far more memory than expected, prompting an investigation into the underlying data structures.
This image shows Ivan Babrou, the engineer who first noticed the memory inefficiency in Pingora’s consistent hashing implementation, prompting the optimization effort.
The team traced the excess to the way each hash‑point was stored in memory. The original struct held a 32‑bit hash and a 32‑bit server index, requiring eight bytes per point due to Rust’s alignment rules, even though the index never needed more than 16 bits given Cloudflare’s server counts. Recognizing this waste, Zaidoon replaced the struct with a raw byte array and accessor methods, effectively packing the hash and index into six bytes and cutting the per‑point memory footprint by 25 %. To further improve load distribution without exploding memory use, the engineers increased the number of hashes assigned to each server. Consistent hashing’s error margin falls as the square root of the hash count, so moving from the default 160 hashes per server to a weighted count (for example, 100 000 hashes for a heavily weighted server) dramatically reduces the coefficient of variation, ensuring that no server is overloaded while others sit idle.
This image depicts Zaidoon Abd Al Hadi, whose restructuring of the hash-point struct from eight to six bytes was a key step in reclaiming over 100TB of RAM.
By combining the tighter storage layout with a smarter hash‑per‑server strategy, Cloudflare shaved enough memory from its global consistent‑hashing tables to reclaim more than 100 TB of RAM. This saving stacks atop the 100 TB previously freed by the DNS team, illustrating how low‑level algorithmic tweaks compound at Cloudflare’s scale to yield massive resource efficiencies. The improvements were implemented entirely in Rust, showcasing how systems‑level language choices can enable both performance gains and substantial memory economies in production services at scale.