Blog
Results and methods.
What we shipped, how it was measured, and the numbers behind it.
Research / September 22, 2026
Swarm the prompts GRPO throws away
An RL training set is only the prompts the model sometimes gets right. The hardest ones never make it in, because every rollout fails. We gave 223 such prompts the same extra budget four ways. A swarm of attempts that share programs rescued one in ten. So did plain resampling. Spending the samples mattered. How we structured them did not.
Benchmark / September 21, 2026
MiMo or Claude for support agents?
Xiaomi released MiMo-V2.6 today. We ran the new open-weights model through the frozen refund-agent test we use for Claude, and tried the reward trick from its report on a 4B model. Flash and Pro tie Claude Sonnet 5 and Haiku 4.5 on the agent, at a fraction of the price. The reward trick did nothing on a one-line SQL task.
Guide / September 21, 2026
Harness vs model
An agent is a model plus a harness: the prompt, the tools, the context files, the turn cap. On a closed model the harness is the only part you can change, and it often moves the score more than switching models. whileai now runs, versions and compares harnesses, and tells you which lever moved the score.
Research / September 20, 2026
Distill without a reward
We trained one small model three ways on the same task: with a reward (GRPO), with a frozen stronger model scoring every token (OPD), and with the model scoring itself after seeing the answer (OPSD). The frozen teacher matched the reward in twenty steps. Self-distillation moved a fifth as far, and the log says why.
Benchmark / September 19, 2026
Jev ties Claude at grading agents
Six AI judges read the same 300 agent transcripts and said pass or fail. Claude Sonnet 5 and TypeSafe's Jev tied, and Jev was nine times faster. None reached the 80% agreement we require before trusting a judge with training data.
Recipe / September 19, 2026
Build a reasoning model with wai
A 4B base model learned to reason from a rule reward in 46 minutes on one GPU, going from 51% to 72% on held-out math problems. Paying for the right answer instead of the right format was worth 9 points. One training seed per arm, so the gap is real at this size but not yet banked.
Guide / September 19, 2026
Agent evals on frontier models
Keep the model you run and your own key. The library writes situations your agent has never met, scores them with a 95% interval, and fails the pull request on a regression. A refund agent on Claude Haiku 4.5 passed 95% of the first test and 70% of the fourth, which found a real gap.
Guide / September 18, 2026
Jev as the training-data judge
TypeSafe's new model Jev answers yes-or-no questions with a probability instead of a sentence. That is the shape a judge should have, so the whileai SDK now takes it as one. On 300 real runs it agreed with the answer key 62% of the time, level with Claude and faster.
Research / September 17, 2026
RL for text to SQL
Two small models learn to write SQL with reinforcement learning and no judge model. The reward is a program that runs the query and checks the rows. Nemotron-Nano-8B went from 26% to 35% in 44 minutes. Qwen3-4B needed a better-fed trainer, then 53% to 74%. A newer 9B base reached 82% in one round.
Compare / September 17, 2026
Open-source alternative to Raindrop Simulations
Raindrop announced Simulations on September 17, 2026, and it is behind a waitlist. The whileai SDK runs the same test today on the agent you already have, with a confidence interval on the result, and then trains the agent on the runs it failed.
Guide / September 17, 2026
Evals that get harder
An agent that passes every test cannot show you a gain. On a refund agent, the default test had 2 prompts out of 40 the agent could fail. Aiming the test at what it failed took that to 16, and found a real gap.
Company / September 16, 2026
Agents improve while they work
Most agents fail the same way every day, and a prompt edit is a guess. While. catches the failure in live traffic, trains the model on it, and proves the gain on a held-out test before anything ships.
Research / September 9, 2026
Simulated data for support agents
We trained a support agent on 1,057 simulated conversations that cost about 15 dollars. On a public benchmark it beat the same model trained on expert-written conversations, 30% to 17.5%. The margin over the experts is promising, not settled.