Chain-of-Confidence: Probabilistic Reasoning Traces
Abstract
Chain-of-thought reasoning improves accuracy, but it is silent about how sure the model is at each step. We ask the model to annotate its reasoning with lightweight per-step confidence signals and find that the resulting traces are dramatically better calibrated at the final answer — without the model being trained on any calibration labels. The result suggests confidence is latent in reasoning, and that surfacing it is largely an extraction problem.
A chain of thought is a transcript of a model’s reasoning. Like most transcripts, it is flat — every step is written with the same grammatical certainty, even though the model’s internal confidence in each step differs wildly. The final answer inherits that flatness: a model that reasoned through four shaky steps and one solid one will state its conclusion as if all five were certain.
What if the confidence were in the transcript? We tested a disarmingly simple idea: ask the model, as it reasons, to flag how confident it is in each step. No new training, no calibration labels — just a request to annotate its own reasoning.
Approach
We prompted Mantus and Corus with a modified reasoning protocol: after each intermediate step, the model emits a lightweight confidence tag — one of a small fixed vocabulary from “well-supported” to “guessing.” We evaluated on math and factual synthesis tasks where we could score both accuracy and calibration, and compared against standard chain-of-thought baselines of matched output length.
Results
The annotated traces were substantially better calibrated. On the final answer, expected calibration error dropped by nearly half relative to the baseline, with essentially no loss in accuracy (within 1 percentage point). The gains were concentrated in the middle of the confidence range — exactly the answers where a user most needs to know how much to trust the model.
The tags were also meaningful on their own. When a model tagged a step “guessing,” its eventual answer was wrong 73% of the time — a signal that would be useful to surface directly to a user, or to trigger a retrieval step before answering.
Discussion
Confidence, on this evidence, is not a property the model has to be trained to produce — it is latent in the reasoning itself, and extracting it is largely a prompt-and-parsing problem. That is good news: it makes honest confidence cheap.
We are careful about what we are not claiming. Self-reported confidence is not ground truth; it can be systematically miscalibrated in ways this study is too small to bound. But the signal-to-noise ratio of the tags — and their cost, which is near zero — makes this an attractive default for Ichnus, and a candidate for the honesty features we ship.
References
- Kadavá, A., et al. (2024). Self-annotation and calibration in reasoning traces. EMNLP 2024.
- Lindgren, M., & Sato, H. (2025). Extraction over estimation: confidence in transcripts. Curos Internal Report 2025-09.
- Paulus, R., et al. (2023). Confidence elicitation in language models. NeurIPS 2023.
- Ram, S., et al. (2024). Should chain-of-thought carry its own metadata? Workshop on Reasoning, ICLR 2024.
- Sato, H., et al. (2025). Serving cost of annotated reasoning. Curos Internal Report 2025-08.
- Zhang, Y., et al. (2023). Understanding self-consistency across reasoning variants. ICLR 2024.