Deep Agents SFT from traces

The short version. We fine-tuned a LangChain Deep Agents code reviewer on its own LangSmith traces with LangChain's smithtune, changing only which traces it learned from. LangChain's team pointed out that our first try skipped smithtune's main step: reading traces and writing the rubric together. So we reran it their way and published the rules before training. The traces While. chose still won, 52% of reviews right on a public benchmark against 36%, on every training run. Doing it smithtune's way helped too: the two choices now share most of their traces.

Five steps from traces to a fair comparison

1. Run the stock agent and keep its runs. A stock Deep Agents agent [1] on Qwen3.8-27B, an open model, reads a proposed patch for a real GitHub issue and answers approve or reject. Each run is recorded step by step (a trace) in LangSmith [2]. Hidden tests already decided whether each patch works, so the right answer is known.
2. Choose the training traces two ways. Without While., we follow the smithtune skill [3]: read 20 varied traces, write the rubric from what they show, check it on a 20-trace trial, then let smithtune's panel of model judges (its council) keep or drop each trace. With While., a trace is kept only if it is correct and used at most eight model calls.
3. Hold everything else equal. Both sides keep 174 traces and are fine-tuned with smithtune's own renderer and split for 2 passes over the data, three times each with different seeds.
4. Score on tests nobody here built. SWE-bench Verified [4] patch review, and 84 repos the traces never touched. One server, same sampling.
5. Publish the rules first. Every rule was merged before training [8].

The short traces won on every run

Reading the traces mattered. The council kept 184 traces instead of 258, and its choice shares 142 traces with the While. pick. The gap comes from the rest: the council still keeps long correct reviews, and a model trained on them keeps reading files until it runs out of steps without an answer. The eight-call budget in the While. pick teaches it to finish.

Spending 400 more runs did not help either side

A second experiment gave each side 400 more agent runs on unused tasks. smithtune cannot aim, so its side drew them at random. While. aimed its draw at the kinds of patches the stock agent got wrong. Each side then picked its own way. The While. side won again, but neither side beat its own result from the first experiment, so the extra runs did not pay off here.

One mistake we fixed on the way

We first trained with smithtune's default stopping rule, which keeps the pass with the lowest validation loss. The first and second passes scored almost the same on that check, so the rule picked close to at random, and first-pass models mostly loop. Scores then swung with luck more than with traces. We published that result as registered, changed the rule to exactly two passes before scoring the second experiment, and reran both [8]. The registered result pointed the same way.

What this teaches about post-training

Fine-tuning copies behavior, all of it. A correct review that took twenty calls is a lesson in taking twenty calls. Filtering outputs by a score and training on the survivors is rejection sampling [5]; a budget on calls turns efficiency into data.

The LangChain team was right that reading traces matters. It moved the council most of the way to the same answer. A rule that encodes what the reader noticed finishes the job, and runs again on the next batch of traces without anyone reading them.

The pipeline stayed LangChain's. Deep Agents ran the agent, LangSmith kept the traces and smithtune trained the model. While. decided what to train on and whether it worked.

For researchers

The metric is accuracy, $y_{i,s} \in \{0, 1\}$ for review $i$ under seed $s$, and the comparison is the paired mean difference over reviews, seeds pooled, with a bootstrap interval over reviews:

Seed spread inside one side is up to 18 points, which a bootstrap over reviews does not see. The seed-pair count in the chart above is the check that does not lean on it. Every number is in the recipe's results.json [7].

References

1. LangChain (2025). Deep Agents [software]. github.com/langchain-ai/deepagents. Accessed September 28, 2026.
2. LangChain (2026). LangSmith documentation. docs.langchain.com/langsmith. Accessed September 28, 2026.
3. LangChain (2026). smithtune v0.1.0 [software], including the smithtune skill. github.com/langchain-ai/smithtune. Accessed October 1, 2026.
4. Jimenez, C. E., Yang, J., Wettig, A., Yao, S., Pei, K., Press, O. and Narasimhan, K. (2024). SWE-bench: Can language models resolve real-world GitHub issues? ICLR 2024. arXiv:2310.06770. SWE-bench Verified subset: OpenAI (2024), Introducing SWE-bench Verified, openai.com/index/introducing-swe-bench-verified.
5. Lambert, N. (2025). Reinforcement Learning from Human Feedback, chapter Rejection Sampling. arXiv:2504.12501. Online at rlhfbook.com.
6. Nebius (2024). SWE-agent-trajectories and SWE-bench-extra [datasets]. huggingface.co/datasets/nebius/SWE-agent-trajectories.
7. whilehq (2026). deepagents-review-four-arms recipe, whileai SDK [code and data]. github.com/whilehq/whileai-sdk/tree/main/recipes/community/deepagents-review-four-arms.
8. whilehq (2026). Pre-registration of the fair rerun and amendment 1, deepagents-review-four-arms recipe [code and data]. github.com/whilehq/whileai-sdk/blob/main/recipes/community/deepagents-review-four-arms/PREREGISTRATION.md.

Run it

pip install whileai
git clone https://github.com/whilehq/whileai-sdk
cd whileai-sdk/recipes/community/deepagents-review-four-arms
python run.py --dry-run   # the comparison and the selection rule on fixture rows, no keys
python make_results.py    # every number above, from the committed rows

Every command for the rerun, from the smithtune pull to scoring all twelve models, is in the recipe README.