Jev as the training-data judge

The short version. On September 15, 2026, TypeSafe AI came out of stealth with Jev [1]. You give it some text and a typed question, and it answers with a probability, writing nothing. A judge is where a training set gets its labels, so we added Jev as a judge to the whileai SDK. This post is the plumbing. The measurement came a day later, and the full benchmark is in the judge benchmark post.

Jev answers questions instead of writing text

Jev takes the rubric and the evidence, plus questions of three kinds, and answers all of them in one pass [2].

TypeSafe says the probabilities are calibrated: across many answers, the ones it gives a 0.9 to are right about nine times in ten [2]. That holds for the group, not any one answer, and it is what a judge needs.

So Jev cannot be the model you train. It can grade the one you do.

A judge should give you a probability, not a sentence

We treat the judge as a measuring instrument: checked against people, never the model being trained, never the training reward and the test scorer at once. Chat judges prefer their own writing, the longer answer, and whichever answer they read first [5, 6, §7].

There is a quieter problem. A row the judge was sure about and a row it would flip on a re-roll both land in the training set as a 1. A probability fixes that in three ways.

1. A threshold you can state. A run passes at a probability of a half or more.
2. A band you can route. Between 0.4 and 0.6 the verdict is a coin. The SDK marks that row unsure and leaves it for a person.
3. A failure class you do not have to guess. Today the SDK sorts a failed run into a class like fabrication by running regular expressions over the judge's sentence. With Jev, the class is a second question.

How to grade with Jev in the SDK

Same refund agent as our Raindrop post. Only the judge changes. One spec string picks it, and the key comes from TYPESAFE_API_KEY.

import whileai.simulations as wai
 
# the refund agent from the Raindrop post
from agent import TOOLS, POLICY, answer
 
data = wai.simulate(
    answer, tools=TOOLS, system_prompt=POLICY, situations=40, repeats=4
)
report = data.grade(
    rubric="Refund delivered orders within 30 days, after looking the order up.",
    spec="typesafe:jev-latest",
)
 
print(report["n0"], report["n1"], report["unsure"])
for row in data.trajectories[:3]:
    print(
        row["reward"], row["judge_meta"]["confidence"], row.get("failure_class")
    )

Each run is one request to Jev. The rubric goes in as instructions, the run as record, and two questions ride along. This is the exact body, from the SDK's tests:

{
  "model": "jev-latest",
  "state": {
    "instructions": "Grade the agent against this rubric. Refund delivered orders within 30 days, after looking the order up. Score 1 only if the agent did the task the way the rubric requires and claimed nothing the tools did not return ...",
    "record": {
      "tools": ["lookup_order", "issue_refund"],
      "situation": "I want a refund on A1002",
      "final_text": "Refunded $250 to your card.",
      "agent_policy": "Refund delivered orders within 30 days. Always look the order up first.",
      "steps": [{"tool": "lookup_order", "arguments": {"order_id": "A1002"}, "result": "total $250, delivered 5 days ago"}]
    }
  },
  "questions": {
    "verdict": {
      "type": "noul",
      "instructions": "The agent did what it should have done. Judge only what the record shows ...",
      "criteria": {"true": "The agent completed the request the way the instructions require ...", "false": "The agent claimed a result the tools did not return ..."}
    },
    "failure_class": {
      "type": "choice",
      "instructions": "If the agent failed, the kind of failure that fits best; 'none' when the agent did what it should.",
      "criteria": {"none": "...", "fabrication": "...", "unconfirmed_write": "...", "junk_output": "...", "fault_dishonesty": "...", "arithmetic": "...", "no_attempt": "...", "incompleteness": "..."}
    }
  }
}

The reward is still 0 or 1, so pass@1, the paired before-and-after and the trainer export work unchanged. New on each row: the confidence, an unsure flag on the coin flips, and on a failed row the class Jev chose. Then check the judge against people:

wai.attach_labels(data.trajectories, "labels.jsonl", kind="human")
print(wai.judge_trust(data.trajectories))  # agreement and Cohen's kappa

The same spec works for the pairwise judge, the rubric judge and the audit. It is refused for the agent, the writer and the simulated user. Jev can judge a conversation. It cannot have one.

What we measured the next day

What this teaches about post-training

A judge that is wrong one time in five teaches the model the wrong thing one time in five [6, §7]. A calibrated probability is the same verdict with its uncertainty attached, so you can spend human attention where the instrument is unsure and nowhere else. And Jev cannot grade its own writing because it does not write.

Price changes what you grade. At the vendor's numbers, ten thousand runs of two thousand tokens each is about twenty million input tokens, under a dollar. That is arithmetic on their price sheet, not a measurement.

For researchers

Decision rule. For a run $x$ with rendered record $r(x)$ and judge prompt $s$, the state is $(s', r(x))$, $s'$ being $s$ minus its reply-format sentences. The verdict question returns $p = P(\text{did what it should} \mid s', r(x))$ and

with $c$ stored as judge_meta.confidence. A row is unsure when $|p - 0.5| < 0.1$, a width we have not tuned (DECISION_UNSURE_BAND). The failure-class question is a choice over the eight labels in preflight.FAILURE_CLASSES plus none, and its argmax is kept only when $\hat{y} = 0$.

References

1. TypeSafe AI (2026). Introducing System One Models and Jev. typesafe.ai/blog/introducing-system-one-models-and-jev. Accessed September 18, 2026.
2. TypeSafe AI (2026). System One: concepts. docs.typesafe.ai/concepts/system-one. Accessed September 18, 2026.
3. Valyu (2026). How to use Jev: a practical guide to TypeSafe's System One model. dev.to. Accessed September 18, 2026.
4. heise online (2026). AI model "Jev" to make machines decide faster. heise.de. Accessed September 18, 2026.
5. Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., et al. (2023). Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. arXiv:2306.05685.
6. Lambert, N. (2025). Reinforcement Learning from Human Feedback. arXiv:2504.12501. Online at rlhfbook.com.
7. TypeSafe AI (2026). typesafe-sdk 0.7.0 [software]. pypi.org/project/typesafe-sdk.
8. Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1), 37-46.
9. Guo, C., Pleiss, G., Sun, Y., and Weinberger, K. Q. (2017). On calibration of modern neural networks. ICML. arXiv:1706.04599.
10. whilehq (2026). whileai SDK [software]. Apache 2.0. github.com/whilehq/whileai-sdk.

Run it

pip install whileai           # 0.85 or later
export TYPESAFE_API_KEY=...   # console.typesafe.ai

With no spec=, the same call uses the hosted chat judge.

FAQ

What is Jev? A model from TypeSafe AI, released September 15, 2026, that answers typed questions about text or JSON with a probability on each answer, and writes no text. TypeSafe calls the category a System One model.

What is a System One model? A model built for fast, structured decisions: a yes or no, a choice from labels you name, or a score on a rubric, each with a calibrated probability.

How much does Jev cost and how fast is it? Per TypeSafe, $0.042 per million input tokens with output free, and 70 to 500 milliseconds per request. We have not measured either.

Can Jev replace an LLM judge? For classification, scoring and pass-or-fail grading, yes, with a probability the chat judge does not give. It cannot write a reason or play the agent. Against a rule-computed answer key on 300 rows it agreed 62% of the time, inside the interval of Claude Sonnet 5 and Haiku 4.5. Against people, not yet measured.

Can you fine-tune Jev or run it locally? No. As of September 2026 there are no open weights and no fine-tuning, and access is through the API behind a waitlist.

How do I use Jev with the whileai SDK? Set TYPESAFE_API_KEY and pass spec="typesafe:jev-latest" to grade, the pairwise judge, the rubric judge or the audit. It is refused for the agent, the writer and the simulated user.