Quantized Attention at Frontier Scale
Abstract
Attention is the bottleneck that makes frontier models expensive at long context. We study quantizing attention weights and key-value caches at frontier scale and find that aggressive quantization is nearly lossless on long-context tasks, while delivering savings that grow with context length. The result matters for one reason: it makes a million-token context window affordable enough to be free.
Every token in a long conversation multiplies the work attention has to do. At a million tokens of context, the cost of attention dominates everything else in the system. If attention is the bottleneck, then attention is where efficiency work should concentrate.
This paper is about quantizing the attention machinery itself — the weights that compute it and the key-value cache that feeds it — rather than the dense layers that everyone quantizes first. The empirical question was simple: how far can we push it before long-context reasoning breaks?
Approach
We trained and fine-tuned variants of Corus with progressively more aggressive quantization applied to attention weights and the key-value cache, from 8-bit down to 3-bit, and evaluated on a suite of long-context tasks: needle-in-haystack retrieval, long-document synthesis, and cross-document reasoning.
Crucially, we evaluated at multiple context lengths, because the interesting behavior was whether quantization errors compound or stay flat as context grows.
Results
The results were better than we expected and consistent across lengths. At 4-bit attention weights with a quantized key-value cache, long-context task scores were within 1 percentage point of the full-precision baseline. The savings were large and grew with context length: at 256k tokens, compute fell by 22%; at 1M tokens, by 39%.
Retrieval was the most surprising domain. Needle-in-haystack accuracy — the task we expected to break first — stayed above 97% even at 4-bit quantization, because the quantization error behaved like noise that the attention mechanism absorbed rather than amplified.
Quantization did eventually bite: at 3-bit, cross-document reasoning degraded measurably (about 6 points), and the failures clustered in exactly the compositional tasks our reasoning team cares about.
Discussion
The result is a trade that makes product sense: keep attention at 4-bit, keep the quality, take the 39% savings at the context lengths that matter. For Ichnus, that is the difference between offering a million-token context as a headline and offering it as the default for everyone.
We are cautious about extrapolation. The tests were ours, the task suite is finite, and quantization noise is notoriously domain-dependent. But the headline — frontier-class attention tolerates aggressive quantization — is worth a lot of careful work, because it is the lever that makes long context cheap enough to be free.
References
- Dettmers, T., et al. (2022). QLoRA: efficient fine-tuning of quantized LLMs. NeurIPS 2022.
- Guo, R., et al. (2024). KV cache quantization for long-context serving. MLSys 2024.
- Nakamura, E., & Ferrand, L. (2025). Attention-first quantization at frontier scale. Curos Internal Report 2025-10.
- Sheng, Y., et al. (2023). High-throughput generation for long sequences. NeurIPS 2023.
- Tao, L., et al. (2024). Quantization-aware training for attention. ICML 2024.
- Zheng, L., et al. (2024). Where attention breaks under compression. Workshop on Efficiency, ICLR 2024.