Simulated data for support agents

The short version. To fine-tune a support agent, most teams pay experts to write example conversations. We asked whether a simulator could write them instead, given only the agent's tools and policy. It wrote 1,057 conversations for a public support benchmark. We trained the same model an expert-data vendor had trained, with their recipe, and scored both on the benchmark's own test. Ours won, 30% to 17.5%. The gain over the untrained model is beyond doubt. The margin over the experts rests on 40 tasks, so we call it promising. Every training row is on Hugging Face.

The experiment

tau2-bench is a public benchmark from Sierra. A model plays a support agent with real tools and a policy document, and a second model plays the customer. A task passes only when the world at the end matches the answer key.

In April 2026 the expert-data company AfterQuery published a run on it: 1,057 expert-written conversations, an 8-billion-parameter open model, and the recipe. Same base model, same recipe, same scorer. Only the conversations changed. Ours came from a simulator that had seen only the agent's tools and policy.

How the simulator writes a conversation

It maps the situations first. Ask a model for a thousand customer requests and you get a thousand versions of the same one. So the simulator reads the policy as data and builds six axes.

Six axes describe hundreds of thousands of situations. The simulator picks a set where every pair of values appears together at least once (a pairwise covering design, from software testing), because most real failures come from two things interacting. For telecom that came to 240 situations, and 150 were written out.

It gives each customer a way of typing. Prose, lowercase, clipped, or typos.

It plays the conversation out. One hosted small model plays both sides. The agent has the tools and the policy. The customer has the situation card and nothing about the policy. A fake back-end answers every tool call the same way for a given seed.

It keeps or drops. Rule checks remove conversations that are wrong by construction: an action claimed but never taken, an account number the customer never gave, success declared after a failed tool call. Then a grader model reads every survivor against a rubric and keeps or drops it with a reason. For telecom, 369 in, 310 out.

What the model learned

Twelve of the 40 test tasks end correctly by handing the customer to a person. The trained model passed nine. A typical win: the lookups fail, and the model transfers the customer instead of guessing. Fifteen of the 310 training conversations contain that pattern, and nobody wrote them. The simulator drew them from the policy's own escalation rules. The passes split evenly across the benchmark's hard and easy customer tags.

Three caveats. The model learned when to hand off better than when to keep going, so restraint is graded in the next set. Retail and airline, trained the same way, did not match the expert-data numbers. The expert figure is their best checkpoint per domain, ours the final one.

What this teaches about post-training

Post-training is where an agent's behavior is made. Forty-eight minutes on one GPU changed nothing about what the model knows and everything about what it does.

The expensive part was never the writing. It was choosing which situations to write. An expert knows that a suspended line plus a failed lookup plus an impatient customer is a case that matters. A covering design found that case without being told. The next run is the full 114-task split with repeated trials.

Grading is where a dataset gets its quality. The 310 training rows survived a rule check and a rubric.

A number is only as good as its test. The test is the benchmark's own held-out split, and every training row was checked for overlap against all 2,449 benchmark tasks [5, §16].

For researchers

Single trial on the official test split, protocol pre-registered and matched to AfterQuery's run line by line [2]. Pass counts $2$, $7$ and $12$ of $40$. Two-sided Fisher exact test [6] on the $2 \times 2$ table: simulated vs untrained $p = 0.0064$, simulated vs expert $p = 0.29$.

References

1. Barres, V., Dong, H., Ray, S., Si, X. and Narasimhan, K. (2025). τ²-Bench: Evaluating conversational agents in a dual-control environment. arXiv:2506.07982.
2. AfterQuery (2026). How AfterQuery expert data drives model performance on τ²-bench. afterquery.com. Accessed September 9, 2026.
3. Kuhn, D. R., Wallace, D. R. and Gallo, A. M. (2004). Software fault interactions and implications for software testing. IEEE Transactions on Software Engineering, 30(6), 418-421.
4. Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L. and Chen, W. (2022). LoRA: Low-rank adaptation of large language models. International Conference on Learning Representations. arXiv:2106.09685.
5. Lambert, N. (2025). Reinforcement Learning from Human Feedback. arXiv:2504.12501. Online at rlhfbook.com.
6. Fisher, R. A. (1935). The logic of inductive inference. Journal of the Royal Statistical Society, 98(1), 39-82.
7. While. (2026). tau2-simulated [dataset]. Hugging Face. huggingface.co/datasets/while-ai/tau2-simulated.
8. whilehq (2026). whileai SDK [software]. Apache 2.0. github.com/whilehq/whileai-sdk.

Run it

The simulator is the open-source whileai SDK. This call produced the set, and it takes any agent's tools and system prompt:

pip install whileai

import whileai.simulations as wai
 
data = wai.simulate(
    agent="openai:gpt-4.1-mini", tools=TOOLS, system_prompt=POLICY
)
data.grade(
    rubric=RUBRIC
)  # the hosted judge writes reward and reason onto every row
print(data.pass_at)

The SDK README goes from that call to a gated training set.