IPinfo, a provider of IP address data, has developed a method to shrink MMDB files, which are widely used for fast lookups of IP metadata like geolocation and ASN information. The MMDB format consists of two primary components: a search tree, which is a binary trie over the bits of an IP address, and a data section where the actual records reside. While the data section already supports record sharing, the search tree typically does not. Standard writers often emit a fresh node for every position in the tree, leading to significant bloat when multiple IP prefixes resolve to the same data record.
This image illustrates the primary components of the MMDB format mentioned in the first paragraph.
To address this, IPinfo implemented a technique known as hash-consing. This method involves traversing the existing tree from the leaves upward and using a hash table to identify structurally identical nodes. If a pair of child nodes has already been processed, the tool reuses the existing canonical node rather than creating a duplicate. This transforms the search tree into a directed acyclic graph (DAG), where multiple parents can point to a single shared subtree. Because the MMDB format already supports this type of sharing, the optimization does not require any changes to the file format or the reader libraries; the resulting files remain fully compatible with existing tools.
This visual explains the hash-consing mechanism and the resulting DAG structure described in the second paragraph.
In production tests, this optimization resulted in a 31% reduction in the total size of IPinfo's 89-database catalogue, with some specific databases seeing reductions as high as 80%. Because MMDB files are memory-mapped, these savings translate directly into a smaller memory footprint and fewer page faults during lookups. While the primary saving comes from the search tree, the data section remains untouched. IPinfo has open-sourced the optimization as `mmdbshrink`, allowing other developers to reduce the size of their own MMDB files while maintaining identical lookup results and performance.
One remaining technical consideration involves the `record_size` field within the MMDB header. As the number of nodes in the search tree decreases significantly due to hash-consing, there may be an opportunity to reduce the number of bits allocated for record sizes, potentially yielding an additional 12-25% in space savings. However, such a change would require a modification to the file's internal metadata structure, whereas the current `mmdbshrink` tool achieves its results without changing a single byte of the underlying data format.