Ternary Large Language Models (LLMs) represent each weight using one of three symbols: -1, 0, or +1. Theoretically, this corresponds to an information-theoretic limit of approximately 1.58 bits per weight (log2 3). In practical deployment, most current systems use "five-trit packing," where five ternary weights are packed into a single byte. Because computer memory is organized in powers of two, this method effectively rounds up the cost to 1.625 bits per weight, treating all three possible values as if they were equally likely.
However, researchers analyzing 29 different ternary LLM models discovered that these weights are not uniformly distributed; instead, zeros can account for up to 51.5% of all weights. To capitalize on this imbalance, the authors introduced BITCOS, a distribution-adaptive layout. Rather than assigning a fixed number of bits to every weight, BITCOS uses a dense presence bitmap to identify which weights are non-zero and a compacted sign vector to store the remaining values. This approach allows the model's storage cost to scale based on its specific zero density (z), calculated as `2 - z` bits per weight element. On some tested models, this reduced the effective bit-width to as low as 1.485 bits per weight.
The transition from standard packing to BITCOS provides measurable performance gains across various hardware platforms. The study found that at real-world zero densities, the proposed layout achieved up to a 1.28x gain in matrix-vector multiplication compared to production state-of-the-art kernels. Furthermore, end-to-end inference tests showed decode throughput improvements of up to 1.18x on CPUs and 1.27x on Intel Xe2 GPUs. While BITCOS is highly efficient, it requires specialized unpacking sequences—optimized for architectures like AVX-512 and Intel Xe2 GPUs—to maintain these speed advantages during computation. A key unresolved question remains how much efficiency can be gained if future custom silicon (ASICs) is designed to support the BITCOS format natively, potentially removing the need for software-based unpacking entirely.