Tiered Dense–Sparse Routing for Efficient Serving
Abstract
Most requests to an assistant do not need the largest model. We describe a tiered routing system that predicts request difficulty and routes to the cheapest model that can handle it, with a safety rule that reroutes when confidence is low. In production-shaped traffic, the system cut compute cost by 51% while matching the quality of a large-model-only baseline on our internal evaluations.
Ichnus is free, which means its serving cost is its sustainability constraint. Every request answered by the largest model is a request we cannot afford to answer at the volume we want. But most requests do not need the largest model. Most requests are short questions, quick drafts, and simple lookups.
This paper is about the boring engineering that makes “free” viable: routing each request to the smallest model that can handle it well, and never letting the routing decision make the answer worse.
Approach
We built a routing layer in front of the Krus, Mantus, and Corus tiers. A lightweight classifier scores each incoming request for difficulty. Easy requests go to Krus, moderate to Mantus, and only genuinely hard requests reach Corus. The router is trained on labeled traffic with a conservative threshold: when uncertain about difficulty, it routes up, never down.
The system includes two safety properties. First, a low-confidence rule: if the router cannot classify a request, it defaults to Mantus rather than gambling on Krus. Second, an after-the-fact check: answers from small models are re-scored for signs of failure, and borderline cases are regenerated by the next tier up.
Results
Across production-shaped traffic, the routing system cut compute cost by 51% relative to a Mantus-only baseline, while latency for easy requests fell by more than a third. Quality held: on our internal Curos-Helpfulness evaluation and on a held-out set of graded responses, the routed system matched the large-model-only baseline within noise.
The routing distribution was revealing. Only 9% of requests needed Corus, and 31% were handled entirely by Krus. The remaining 60% used Mantus — which is, conveniently, what the router was designed to protect.
The safety mechanisms were rarely triggered but important when they were: the reroute-on-low-confidence rule caught the failures that mattered, and we found no case where routing made a response worse than the same request handled by the largest model.
Discussion
Efficiency work at Curos is not a side project; it is the difference between a free product that exists and a free product that is a poster. Tiered routing is the single largest lever we have found.
We are careful about the generalization claim. Our router was trained on our traffic, for our models, and difficulty prediction is inherently leaky. But the architecture — route up, never down, re-check afterwards — is transferable, and the cost structure it unlocks is the one that makes free access sustainable.
References
- Amin, S., & Chen, L. (2025). Difficulty-aware model selection for LLM serving. MLSys 2025.
- Jiang, W., et al. (2024). FrugalGPT: How to use large language models while reducing cost. ICLR 2024.
- Nakamura, E., et al. (2025). Route up, never down: conservative cascades in production. Curos Internal Report 2026-01.
- Park, J., et al. (2024). Cascading inference with quality guarantees. NeurIPS 2024.
- Sato, H., et al. (2025). The cost model behind a free assistant. Curos Internal Report 2025-05.
- Sharma, A., et al. (2023). Predictive routing for model cascades. ACL 2023.