Research / September 23, 2026
The swarm learned the test
Our particle swarm over rollouts never beat plain resampling. We went looking for why. Every fitness we could hand it was a cliff or noise. And on a task built to be a hill, the swarm climbed it by satisfying the tests it was shown while failing the ones it was not.
The short version. Last week a swarm of rollouts that share their best programs rescued as many hard prompts as sampling more. We asked why. The score the swarm climbs says nothing about being close to a pass: partial credit on tests is zero information until the last few tests, and a model judge is chance. So we built a task where partial credit is real by construction and ran it again. The swarm climbed the shown tests; the hidden tests did not follow. Caveat: one small model family, and star was a point or two ahead, never proven.
Qwen3-4B on the code_contests tasks it fails every time: 1,776 saved programs regraded on 40 generated tests as the visible credit, with the private tests plus 20 more as the hidden target. The 27B and text-to-SQL curves have the same shape; on SQL every partial bucket is zero. Source: recipes/01-simulate/swarm-rescue/calibrate.py and sql_calibrate.py.
| Item | Value |
|---|---|
| No visible tests passed | 0.0% |
| Up to a quarter | 0.0% |
| A quarter to a half | 0.7% |
| Half to three quarters | 1.2% |
| Three quarters to 90% | 34.1% |
| 90% to all but one | 29.3% |
| Every visible test | 54.5% |
Is there a hill
A swarm moves toward whatever scores best. That only helps if the score rises as programs get closer to correct. We checked, with no new model calls: take every saved program, grade it on many more tests than the swarm saw, and ask how often a program at each level of partial credit passes tests it never saw.
The chart above is the answer for code. A program passing half the tests is as far from a pass as one passing none. The step is at the top. On text-to-SQL, scored by how many cells of the right answer a query returns, it is worse: no partial bucket ever passes, and the tasks whose wrong queries score higher are the ones the model solves less often.
A model judge is the other candidate for a graded score. We showed the 27B each program and the problem, blind to every test, and asked for the probability of a pass. Its number ranks real passes above real fails at 0.52, which is a coin.
A hill by construction
So we built one. A task is five functions from MBPP [1]. Each function shows one example assert; two more are hidden. The swarm's score is the number of functions whose shown assert holds, so it counts work left, and a neighbour that has your missing function working carries real information. We wrote the metric, the predictions and the decision rule into the recipe before the run.
Bundles of five MBPP functions where Qwen3-4B failed all 8 base samples: 174 tasks on seed 0, 154 on seed 1. The swarms double the share of programs that satisfy the shown asserts. The note is how many of those then pass every hidden assert. Source: results-mbpp.json and results-mbpp-s1.json in the recipe.
| Item | Value |
|---|---|
| Resample, seed 0 | 8.4% |
| Solo, seed 0 | 8.5% |
| Ring, seed 0 | 13.2% |
| Star, seed 0 | 17.7% |
| Resample, seed 1 | 5.9% |
| Solo, seed 1 | 7.5% |
| Ring, seed 1 | 11.8% |
| Star, seed 1 | 14.9% |
The swarm climbed. Star's round-2 programs passed all five shown asserts twice as often as resampling's on both seeds. Almost none of them passed the hidden asserts. The primary metric, pass rate per sample paired by task, moved by less than a point against a pre-registered bar of five. The velocity update satisfied the example it was shown, not the function it was asked for.
What went wrong, in three lines
The particles anchored. A refined program shared 80 to 93% of its text with the particle's own first attempt, against 68% for a fresh draw, and most rescues in every arm came from the fresh draws of round 0. Confident commitment to a wrong idea, then surface fixes.
Copies of one model have nothing to trade. Particle swarms work because particles sample different regions. Eight instances of the same 4B share one set of blind spots, and sharing programs pools them.
A step in text is not small. Swarms assume that moving toward the best neighbour is a short move on a smooth landscape. A rewrite conditioned on another program is a large, unstructured jump, with neither the locality of a swarm nor the diversity of sampling.
What this teaches about post-training
The check that would have ended this on day one costs nothing but grading time: does your fitness predict the pass. It is the same lesson as reward signal in RL, where variance in the score is not signal and only correlation with the target is [2, 3]. And the hill result is over-optimization in miniature [4]: given a proxy, an optimizer finds the proxy. Both checks now ship as scripts in the recipe. What still moved the training set was the plain thing: more samples on hard prompts, however you structure them.
For researchers
| Setting | Value |
|---|---|
| Calibration, code | code_contests all-fail tasks; fitness = share of 40 generated tests passed, every test run; target = private tests + 20 more generated; 4B 1,776 samples, 27B 816 |
| Calibration, SQL | 600 tasks x 8 samples on the 4B, cell-level F1 vs the gold result; exact match target; task-level correlation of wrong-attempt fitness with pass rate -0.19 |
| Judge | Qwen3.8-27B, blind to tests, "probability of passing every hidden test"; 1,608 samples (108 passes); AUC 0.523 |
| Positive control | 5 MBPP functions a task, 192 bundles from 964 problems; visible = the shown assert per function, hidden = the other two; 8 base samples; arms on the all-fail bundles at 24 samples each |
| Primary metric | per-sample pass rate paired by task, seed 0: solo +0.86 [+0.24, +1.63], ring +0.53 [-0.08, +1.28], star +0.43 [-0.12, +1.10]; seed 1: +0.18, +0.04, +0.42, all inside their intervals |
| Rescue rate | seed 0: 8.0 / 8.6 / 8.0 / 8.6%, band 5.3; seed 1: 3.9 / 2.6 / 2.6 / 5.2%, band 6.0 |
| Did not replicate | seed 0's fall in hidden asserts passed among shown-test-passing programs across rounds reversed on seed 1 |
| Grader | every visible test runs and the count is the fitness; a Windows process-creation fault voided one run and now stops the recipe instead of grading zeros |
Run it
pip install whileai pyarrow
git clone https://github.com/whilehq/whileai-sdk
cd whileai-sdk/recipes/01-simulate/swarm-rescue
python calibrate.py --out out # is the fitness a hill
python run.py --tasks mbpp-bundle --noise-runs 2 # the positive controlReferences
- Austin, J., Odena, A., Nye, M., Bosma, M., Michalewski, H., Dohan, D., Jiang, E., Cai, C., Terry, M., Le, Q. and Sutton, C. (2021). Program synthesis with large language models. arXiv:2108.07732.
- Kennedy, J. and Eberhart, R. (1995). Particle swarm optimization. Proceedings of IEEE International Conference on Neural Networks, 1942 to 1948.
- Yu, Q., et al. (2025). DAPO: An open-source LLM reinforcement learning system at scale. arXiv:2503.14476.
- Gao, L., Schulman, J. and Hilton, J. (2023). Scaling laws for reward model overoptimization. International Conference on Machine Learning. arXiv:2210.10760.
- Li, Y., et al. (2022). Competition-level code generation with AlphaCode. Science 378(6624), 1092 to 1097. arXiv:2203.07814.