A look inside Ichnus's safety evals
How the evaluation suite works, what it catches, what it does not, and why a release can be delayed by a committee.
When someone asks “is Ichnus safe?” the honest answer has two parts. First: we test every model before release against a published suite of evaluations, and a committee with the power to delay a release reviews the results. Second: no. Nothing with “safe” in the sentence is a guarantee. Here is what the testing actually looks like.
The four layers
Every release passes through four kinds of evaluation, in order:
Capability benchmarks measure what a model can do — knowledge, reasoning, math, code. These are the numbers on the model pages. They tell us where the model stands.
Safety evals test for behaviors we have committed not to ship: following instructions that would cause harm, providing reliable information on dangerous capabilities, behaving deceptively. These evals are built to fail — their job is to find where the model breaks, not to confirm it is good.
Honesty evals measure whether the model overstates confidence, invents sources, or refuses to admit uncertainty. For an assistant people actually rely on, honesty is a safety property.
Red-teaming is where humans try to break the model on purpose — internally, and since this year, with the community through our public red-teaming windows.
The gate
Passing the suite is necessary but not sufficient. The Safety & Evaluations Committee — which includes people outside the evaluations team — reviews the results and signs off, or does not. They have delayed two releases this year. That is the system working; a release gate that never delays anything is a decoration.
What we publish
Alongside every release we publish the evaluation report: the suite, the numbers, the red-team findings, and explicitly what we did not test and why. The report is the announcement. If you only read one thing about a release, read the report, not the press release.
The honest part
Evaluations cannot prove a model is safe. They bound our confidence, and they make our judgments checkable. New behaviors appear after deployment — which is why we publish incident postmortems and why the community can report problems.
Safety, at Curos, is a practice with published receipts. That is the best we know how to do.
All news →