Scaling Test-Time Monitoring: Measuring Oversight Density
Abstract
As models reason for longer before answering, oversight has moved from the output to the process: we watch how a model thinks, not just what it says. But nobody has a good measure of how much watching is happening. We define oversight density as the number of independent verification checks applied per unit of reasoning, and show across three families of tasks that higher density predicts better error detection — and, importantly, that beyond a threshold it stops helping.
The point of monitoring a model’s reasoning is to catch problems before they become answers. As models spend more tokens thinking — and our Corus tier deliberately does — the ratio of process to product grows, and so does the opportunity to check work along the way. The trouble is we lack a vocabulary for how much checking is happening. “We monitored the chain of thought” is not a measurement.
This paper proposes one: oversight density. We define it as the number of independent checks applied to a unit of reasoning, where a check is any verifier — a self-consistency pass, a constraint check, a fact lookup — applied to a specific step. We then test whether density predicts anything about the quality of post-hoc review.
Approach
We instrumented a monitoring harness that applies configurable verifiers at configurable rates to the reasoning of Mantus and Corus. For each task, we varied density by controlling two things: how many verifiers ran, and how frequently they sampled the reasoning trace. We measured three outcomes: error detection rate, false positive rate, and the computational cost of monitoring.
The task set spanned multi-step math, factual synthesis from documents, and code generation, because we suspected density would behave differently across domains.
Results
Across all three domains, higher oversight density improved error detection — up to a point. Detection rates climbed from roughly 58% at low density to 87% at moderate density, then plateaued. Pushing density further added cost without adding catches. The plateau arrived earlier for math (detection saturated around 90% at moderate density) and later for document synthesis, where errors are subtler.
False positives behaved the way we hoped: they stayed low across all densities, and slightly decreased with density, as independent checks overrode one another’s mistakes.
The cost structure was the surprising part. The marginal cost of the next check rose steeply near the plateau — verifying the plateau was not an artifact of diminishing usefulness but of genuinely running out of errors to find.
Discussion
Oversight density gives us a number to put next to “we reviewed the reasoning.” That is useful for releases: we can state a target density for a model tier and check that we hit it. It also reframes the plateau as the real question — the errors that survive dense oversight are the ones that matter, and they are not going to be caught by throwing more checks at them.
The measure is young. It depends on what counts as a check, and counting is a judgment call. But the direction feels right: to scale monitoring, we need to be able to say how much monitoring there is.
References
- Amato, L., et al. (2025). Verifier scheduling in long-horizon reasoning. ICLR 2026.
- Burns, C., et al. (2024). Weak-to-strong generalization and the limits of review. NeurIPS 2024.
- Lindgren, M. (2025). Self-consistency as a monitoring primitive. Curos Internal Report 2025-12.
- Osei, K., & Ren, T. (2025). The economics of process supervision. Workshop on AI Safety, ICML 2025.
- Raman, P., et al. (2025). Monitoring as a first-class release gate. Curos Internal Report 2026-01.
- Zhang, Q., et al. (2024). Process reward models and their blind spots. ICML 2024.