Deep Agents SFT from traces

The short version. We wanted to know if the choice of training examples matters as much as the trainer. A LangChain Deep Agents agent reviewed code patches and its runs went to LangSmith. We fine-tuned it twice with LangChain's smithtune, changing only which runs it learned from. Trained on correct, short runs, it got 57% of reviews right on a public benchmark, against 46% when trained on every correct run. It also took fewer steps. On unseen repos the two tied. Each arm has one training run so far, and a second is next.

Six steps from traces to a tuned reviewer

1. Run the stock agent and keep its runs. A stock Deep Agents agent [1] on Qwen3.8-27B, an open model, reads a proposed patch for a real GitHub issue with read-only file tools and answers approve or reject. Each run is recorded step by step (a trace) in LangSmith [2] with a score, correct or not. Hidden tests already decided whether each patch fixed its issue, so the right answer is known. We collected 400 training reviews.
2. Choose the training traces two ways. Without While., a panel of model judges (smithtune's judge council) [3] reads each trace against a plain rubric. It kept 258 of 260 correct traces. With While., a trace is kept only if it is correct and used at most eight model calls, spread across the different ways the agent used its tools. That kept 173.
3. Fine-tune both the same way. The model learns to copy the kept examples (supervised fine-tuning, SFT), with smithtune's own renderer, split and schedule.
4. Serve both from one server. Same sampling, thinking off, so any gap comes from the weights.
5. Score on a benchmark nobody here built. SWE-bench Verified [4] patch review has 250 reviews on 125 issues, and half the patches fixed their issue. None of its repos appear in the training traces.
6. Measure the noise first. A single base run is not a baseline. Re-running the unchanged base once moved it 4.5 points, with an interval that excluded zero. So every comparison here carries how far the score moves when nothing changes (the noise floor), taken from three base runs.

Lean traces taught a better reviewer

The base model mostly ran out of steps before deciding. Tuning the Deep Agents settings (its harness), with no training, lifted it by about nine points. The model trained on While.'s traces was right more often, with a gain between +5 and +17 points, the range the true gain very likely sits in. That clears the noise floor. It also used about five fewer model calls per review.

On the second test, repos the model never trained on (held-out repos), accuracy was not proven either way. The lean model still used about three fewer calls per review. Its training set was a third smaller and took 65% fewer training tokens.

What this teaches about post-training

Which traces you train on matters as much as the trainer. Fine-tuning copies behavior, all of it. Keeping every correct trace also teaches the agent the long, wandering ones. A correct review that took twenty calls is a lesson in taking twenty calls.

Filtering sampled outputs by a score and training on the survivors is rejection sampling [5]. Adding a budget on model calls turns efficiency into data. Choosing correct and lean traces taught a model that is more accurate on unfamiliar large repos such as django and sympy, and cheaper to run, from a third less data.

The pipeline stayed LangChain's. Deep Agents ran the agent, LangSmith kept the traces and smithtune trained the model. While. decided what to train on and whether it worked.

Three runs come next. A second training seed per arm. A random set the size of the lean set drawn from the council's set, to separate the choice of traces from their number. And the base with thinking on, which scored between 70 and 75% on the held-out repos, on SWE-bench Verified. The tuned models are non-thinking because smithtune leaves reasoning out of training rows by default.

For researchers

The metric is accuracy, $y_i \in \{0, 1\}$ per review, and the comparison is the paired mean difference with a bootstrap interval over reviews:

Every number is in the recipe's results.json [7].

References

1. LangChain (2025). Deep Agents [software]. github.com/langchain-ai/deepagents. Accessed September 28, 2026.
2. LangChain (2026). LangSmith documentation. docs.langchain.com/langsmith. Accessed September 28, 2026.
3. LangChain (2026). smithtune v0.1.0 [software]. github.com/langchain-ai/smithtune. Accessed September 28, 2026.
4. Jimenez, C. E., Yang, J., Wettig, A., Yao, S., Pei, K., Press, O. and Narasimhan, K. (2024). SWE-bench: Can language models resolve real-world GitHub issues? ICLR 2024. arXiv:2310.06770. SWE-bench Verified subset: OpenAI (2024), Introducing SWE-bench Verified, openai.com/index/introducing-swe-bench-verified.
5. Lambert, N. (2025). Reinforcement Learning from Human Feedback, chapter Rejection Sampling. arXiv:2504.12501. Online at rlhfbook.com.
6. Nebius (2024). SWE-agent-trajectories and SWE-bench-extra [datasets]. huggingface.co/datasets/nebius/SWE-agent-trajectories.
7. whilehq (2026). deepagents-review-four-arms recipe, whileai SDK [code and data]. github.com/whilehq/whileai-sdk/tree/main/recipes/community/deepagents-review-four-arms.

Run it

pip install whileai
git clone https://github.com/whilehq/whileai-sdk
cd whileai-sdk/recipes/community/deepagents-review-four-arms
python run.py --dry-run   # the comparison and the selection rule on fixture rows, no keys
python run.py traces      # stock agent over the training reviews, traces to LangSmith
python pick.py            # the lean traces, written back as LangSmith feedback
python make_results.py    # paired comparisons and the noise floor