Compositional Reasoning under Distraction
Abstract
Chain-of-thought reasoning is meant to hold a model's place as it works through a problem. We find that placing unrelated material between steps disrupts compositional reasoning far more than expected — accuracy on multi-step tasks falls by nearly a third — and that models trained or prompted to keep intermediate results explicitly in view are markedly more robust. The finding has practical consequences for how Ichnus handles long, mixed documents.
Ask a model to solve a four-step reasoning problem and it usually can. Interleave three paragraphs of unrelated text between the steps and the same problem starts to fall apart. This is not a new observation — long-context degradation is well documented — but we wanted to measure exactly where composition breaks, because the answer determines what to do about it.
The practical stakes are immediate. Ichnus users paste in long, noisy documents and ask questions that require stitching together pieces that are far apart. If composition dies under distraction, the product needs to know, and the models need to be trained around it.
Approach
We built a controlled task: compositional problems (multi-step arithmetic, transitive reasoning, and document-spanning synthesis) with irrelevant context inserted at controlled points — before the problem, between steps, and after. We tested Krus, Mantus, and Corus, and compared two mitigation strategies: an explicit prompt instructing the model to restate intermediate results, and a training variant that saw mixed-interleaved data.
Results
Distraction between steps was far worse than distraction at the edges. Inserting unrelated text between the steps of a four-step problem cut accuracy by an average of 29% across models, while the same text placed before or after the problem cost only 4–6%. The effect was largest on transitive reasoning, where an intermediate conclusion has to be carried forward.
Both mitigations worked, unequally. Prompting the model to restate its intermediate results recovered 21 of the 29 lost points, almost without cost. The training variant did better — recovering 26 points and generalizing to unseen interleaving patterns — but at the price of a small throughput cost that made it a deployment trade-off rather than a free win.
Discussion
Composition is not a single skill; it is a capacity that depends on the distance material has to travel. The mechanism behind the fragility — attention diluting as unrelated content interposes — suggests models are holding intermediate state in something closer to working memory than to persistent storage.
For Ichnus, the practical fix is straightforward: when a user’s question spans a long document, the assistant can pull the relevant passages together before answering, rather than reasoning over the whole stream. That is a product decision, and it is the direct consequence of a research result — which is exactly the loop Curos wants between its research and its product.
References
- Dai, L., et al. (2024). Lost in the middle: how long contexts mislead reasoning. NeurIPS 2024.
- Lindgren, M., et al. (2025). Restatement as a reasoning anchor. Curos Internal Report 2026-02.
- Luo, K., et al. (2023). Recursive summarization for long-context QA. EMNLP 2023.
- Ostrowski, M., & Feld, J. (2025). Interleaving as a stress test for composition. ACL 2025.
- Petrova, S., et al. (2025). Retrieval-then-reason as a product pattern. Curos Internal Report 2025-10.
- Wang, X., et al. (2022). Self-consistency improves chain of thought reasoning. ICLR 2023.