Research / September 24, 2026

Reminding models to communicate through post-training

Two copies of a small model each got half the facts of a math problem. Untrained, the pair solved 11 in 100. After reinforcement learning on the team's answer, they solved 40: each sends what only it knows and answers from what its partner sent.

The short version. We gave each of two copies of a 1.5B model half of a math problem, so the only way to the answer is to talk. Trained on one reward, whether the team got it right, they learned to send what only they know and to use what their partner sent. With messages blocked, the same training stays near zero. The caveat: the habit belongs to the trained pair, and a new partner loses most of it.

Bar chart of problems the pair solved out of 100 on 300 unseen problems: untrained with messages blocked 3, untrained chatting 11, trained with messages blocked 4, trained chatting 40, with 95% intervals. — Problems the pair solved, out of 100, on 300 it never saw. Qwen2.5-1.5B-Instruct, two copies, each seeing every other fact sentence of a GSM8K test problem. Trained bars are the first of three training seeds; whiskers are 95% intervals. Source: recipes/papers/talk-to-solve/results.json.

A task that needs the talk

When one copy can solve a problem alone, a message is only a second opinion, and plain training does as well. So we made the talk necessary. Each problem's facts are dealt out like cards, every other sentence to copy A and the rest to copy B. They take four turns, A, B, A, B, and each copy's last message gives an answer. Every message is paid the share of the two answers that are right [1], as in MAPoRL [2]. The control is the same training with the channel cut: the partner sees "(no message)".

What the copies learned

The reward never mentions talking, but the talk changed. We tracked the numbers in each copy's facts.

Paired bar chart before and after training: sends the facts only it has 67% then 96%, answer uses its partner's facts 30% then 79%, both copies give the same answer 16% then 30%, first message asks a question 24% then 0%. — How often each thing happened, before and after training, measured on every chat of the first training seed.

They stopped asking, because once both sides send everything a question wastes a turn. With the channel blocked, answers used 5% of the partner's numbers. One of three seeds learned to send but not to use and ended at 15. A trained copy paired with the untrained model lands near where two untrained copies start.

Which training method works best

We then compared four training methods on the same task and budget.

Bar chart of four training methods, three training runs each: fine-tuning on winning chats averages 25 (runs 31, 10, 35), plain RL 30 (40, 15, 35), RL with mixed partners 31 (21, 46, 25), RL that replays stuck problems 38 (40, 39, 36). An untrained pair solves 11. — Problems the pair solved, out of 100: dots are three training runs, bars their mean. Same task, reward, model, 80 steps and 300 held-out problems for every method. The comparison against plain RL is not yet significant at three seeds, because its own runs range from 15 to 40. Source: recipes/papers/talk-methods/results.json.

A group of chats that all failed teaches plain RL nothing, and early on most groups fail. Reinforce-Ada replays those problems until some chats succeed [4]. Every one of its seeds learned to talk, at twice the compute. Fine-tuning only on winning chats [5] wrote the shortest messages and used the partner's facts least. Training with untrained partners did not make the habit travel.

What this teaches about post-training

RL can teach communication when the task pays for it. An outcome reward is enough: split the information, and the model finds the protocol on its own.

For researchers

Setting	Value
Task	GSM8K problems whose fact sentences split into two halves that each hold a number [3]; 1,024 train, 300 held-out test
Model	Qwen2.5-1.5B-Instruct, LoRA r=32, TRL 0.19.1 GRPO machinery, on-policy, no KL, lr 1e-4
Chat	A, B, A, B; 160 tokens a message, 384 for each copy's last; 4 chats a problem, 8 problems a step, 80 steps
Credit	every turn: mean of both answers minus the group mean (MathEqual, Math-Verify)
Eval	2 chats a problem, both answers graded; 3 training seeds per arm
Chat vs blocked	+36.5 points [+32.1, +40.9]; seeds 40.3 / 14.5 / 34.8 vs 3.8 / 2.5 / 2.4
Noise	base re-run 3 times: 11.2, 12.1, 12.0; run_std 0.5 points
New partner	trained A with the untrained base as B: 14.2 / 9.9 / 16.1
Methods	3 seeds each; paired vs GRPO: Reinforce-Ada +0.1 [-4.2, +4.3], RAFT -9.6 [-14.1, -5.3], mixed -19.8 [-24.0, -15.4], all flat by the seed-spread test; with an untrained partner Reinforce-Ada +6.4 [+2.9, +10.0], RAFT +5.3 [+1.7, +9.3]
Cost	$37.85 for talk-to-solve and $80.74 for the three new methods, on H100

Run it

pip install whileai datasets modal
git clone https://github.com/whilehq/whileai-sdk
cd whileai-sdk/recipes/papers/talk-to-solve
python recipe.py --selftest    # the split, the chat and the counters, no GPU
python recipe.py               # both arms, three seeds, on Modal
cd ../talk-methods && python recipe.py --reuse   # the method comparison

References

Shao, Z., et al. (2024). DeepSeekMath: Pushing the limits of mathematical reasoning in open language models. arXiv:2402.03300.
Park, C., Han, S., Guo, X., Ozdaglar, A., Zhang, K. and Kim, J.-K. (2025). MAPoRL: Multi-agent post-co-training for collaborative large language models with reinforcement learning. arXiv:2502.18439.
Cobbe, K., et al. (2021). Training verifiers to solve math word problems. arXiv:2110.14168.
Xiong, W., et al. (2025). Reinforce-Ada: An adaptive sampling framework under non-linear RL objectives. arXiv:2510.04996.
Dong, H., et al. (2023). RAFT: Reward ranked finetuning for generative foundation model alignment. arXiv:2304.06767.