Jev ties Claude at grading agents

The short version. Training an agent on its own runs means keeping the good ones, and a second model, the judge, decides which. We measured how well six judges do that job. Each read the same 300 transcripts of an agent using tools, half done right and half done wrong, and said pass or fail. Claude Sonnet 5 matched the answer key 66% of the time, Jev 62% inside the same margin and nine times faster, our default judge worse than both. We require 80% before trusting a judge. The answer key is a rule, not a person.

The task

The judge's mistakes become the training signal [1, §7]. The benchmark is that job:

• Input. One transcript: the user's message, every tool call with its arguments and result, the final reply.
• Question. Did the agent do what it should: complete the request, claim nothing the tools did not return, invent nothing, report failures honestly?
• Output. Pass or fail.
• Score. How often that matches the answer key, with a 95% interval.

What we did

The transcripts. 300 runs of Qwen3-4B as a tool agent on airline, calendar and CRM tasks, half passed and half failed, so saying yes to everything scores 50%.

The answer key. A rule that fails a transcript for a claimed result no tool returned, an invented id, a write reported done after the tool failed, or a failed tool ignored. Checkable, not a person.

The judges. All six read the identical transcript under the identical instruction, the one our SDK ships:

• Jev, TypeSafe's decision model [2], which answers with a probability instead of a sentence. Released version and preview.
• Claude Sonnet 5 and Claude Haiku 4.5, through AWS Bedrock.
• Phi-4, our SDK's default judge.
• Qwen3-4B, the model that wrote the transcripts, grading itself, as a self-preference control [3].

What we found

The top four are one judge, statistically. Sonnet leads by four points because its verdict overran the reply budget on twelve runs. Jev graded every run at nine times Sonnet's speed and is the only judge that says how sure it is: right 71% when confident, a coin flip otherwise. Send the unsure runs to a person.

Our default is the weakest judge measured. Phi-4 passed 73% of the failures. A different model family from the policy is not enough on its own.

The model grading itself is a coin flip. Qwen3-4B passed 93% of its own failures. Self-preference [3].

The mistake every judge makes

The user says "check my profile and miles". The agent has no customer id, so it calls the lookup tool with customer_id=7890, a number it made up. The sandbox answers anyway and the agent reports a tidy profile. Every judge sees a successful lookup.

The rule checks where the id came from. A judge could too, if told to look.

What this teaches about post-training

• Measure the judge before you use it. It is the reward model for agent data, where a run goes quietly wrong [1, §7].
• A probability beats a sentence. A chat judge sounds equally sure right or wrong. Routing unsure rows to a person makes a judge auditable.
• The answer key is the bottleneck. Sometimes the rule is the one that is wrong. Next: 300 human labels on these runs.

For researchers

Data. 300 rollouts of Qwen3-4B-Instruct-2507 as a tool agent, three synthetic domains, balanced 50/50, seed 0, from the tool-call-efficiency set, shipped in the recipe [4] with rule_reason. Gold is a deterministic conduct grader attached as kind="program", so ok is false and allow_model_gold=True is explicit.

Judges. Chat judges read the SDK's conduct-floor prompt over the same record, temperature 0 where allowed, one parser. Jev reads the same as state and answers a verdict noul and a failure-class choice. Claude ran through Bedrock with a 400-token reply budget.

Statistics. Agreement with a Wilson interval [5], Cohen's $\kappa$ [6], leak as $P(\hat{y}=1 \mid g=0)$:

Agreement by the rule's reason, share of rows:

Jev by confidence, then seconds per row at concurrency 8:

Code. scored.compare_judges(judges=) in whileai 0.85 [7]. python run.py report in the recipe [4] reprints every number offline. Not shown: human labels, a second prompt, a second seed, a length-bias probe.

References

1. Lambert, N. (2025). Reinforcement Learning from Human Feedback. arXiv:2504.12501. Online at rlhfbook.com.
2. TypeSafe AI (2026). Introducing System One Models and Jev. typesafe.ai/blog/introducing-system-one-models-and-jev. Accessed September 18, 2026.
3. Panickssery, A., Bowman, S. R., and Feng, S. (2024). LLM evaluators recognize and favor their own generations. arXiv:2404.13076.
4. whilehq (2026). compare-judges recipe, whileai SDK [dataset and code]. github.com/whilehq/whileai-sdk/tree/main/recipes/02-measure/compare-judges.
5. Wilson, E. B. (1927). Probable inference, the law of succession, and statistical inference. Journal of the American Statistical Association, 22(158), 209-212.
6. Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1), 37-46.
7. whilehq (2026). whileai SDK 0.85 [software]. Apache 2.0. github.com/whilehq/whileai-sdk.

Run it

pip install whileai
git clone https://github.com/whilehq/whileai-sdk
cd whileai-sdk/recipes/02-measure/compare-judges
python run.py report          # the table above, offline
export TYPESAFE_API_KEY=...   # and/or ANTHROPIC_API_KEY, or `wai login`
python run.py                 # grade the 300 rows with every judge you have a key for

import whileai as wai
 
wai.attach_labels(scored.rows, labels, kind="human")
table = scored.compare_judges(
    {
        "jev": "typesafe:jev-latest",
        "haiku": wai.Anthropic("claude-haiku-4-5"),
        "rules": my_verifier,
    }
)
print(table)

FAQ

What exactly was measured? How often an AI judge gives the same pass-or-fail verdict as a rule-computed answer key on a tool-agent transcript. 300 transcripts, six judges, one instruction.

Is Jev a better judge than Claude? Not better and not worse on this set. Jev is nine times faster and returns a probability, so its unsure verdicts can go to a person.