Context Language Models (CLMs) introduce a novel approach where language models inherently manage their own context by treating it as a file that can be freely updated. According to the primary source, this implementation 'treats the context as a file and allowing the model to make unrestricted updates to this file,' which enables the system to learn what information is most important to retain. This contrasts with traditional methods where context management is controlled externally through harnesses or prompting techniques.
The paper demonstrates that CLMs zero-shot with existing models significantly outperform state-of-the-art context management strategies across multiple benchmarks. Specifically, the research shows '11.4% higher accuracy with 21.5% fewer FLOPs on BrowseComp-Plus, 5% higher scores with 59% fewer FLOPs on 12-hour EdgeBench, and 65% greater improvement with the same compute on a 24-hour multi-repository agent-swarm task.' These results indicate that CLMs achieve better performance while consuming substantially less computational resources.
Beyond efficiency gains, CLMs enable both in-context and parametric learning of context-management strategies by shifting control from external systems to the model itself. The research further shows that CLMs can be steered with natural-language instructions evolved through a standard skill-optimization loop, improving held-out accuracy by up to 35.9 points on a context-management task while reducing compute. Additionally, the co-designed Suffix Cache Reuse for CLM serving further reduces server-side compute by 35% relative to standard SGLang at matched performance.