RL for text to SQL

The short version. We wanted small models that turn a plain-English question into a SQL query for a shop's database. Instead of a judge model, the reward is a program: it runs the query and checks the rows. Nemotron-Nano-8B improved in a single 44-minute round, and the gain held up on three repeat scores. Qwen3-4B barely moved in four rounds. Feeding the trainer differently in a fifth round took it from 53% to 74%. A newer base, Qwen3.5-9B, reached 82% in one round, once we found two bugs in our own tooling that had hidden the gain. The recipe is public.

The recipe in six steps

Seed. A Postgres database with data in it. Ours has eight tables and 300 orders.

Author. A teacher model writes each question with its SQL. We keep a task only if the SQL returns one to fifty rows on two runs. That gave 741 tasks. Before anything else, set aside the test set: 140 tasks picked by a hash of the id, never trained on.

Verify. The verifier runs the model's query read-only and compares the rows to the reference. A match scores 1, wrong rows 0.1, a query that does not run 0. There is no judge, so a longer answer cannot score higher.

Benchmark. Sample each test task four times. Report the first-try pass rate (pass@1) with its 95% interval, beside the rate for any of four tries (pass@4). The gap between them is what RL can work with. Then read the failures.

Train. GRPO with a LoRA adapter and eight samples per question. 600 steps took 44 minutes on one H100.

Score. Sample the test set four times and take the paired difference against the base. A gain is proven only when the interval excludes zero, and we score both sides three times.

What happened

Nemotron moved in one round. Most of the gain was queries that now run.

Qwen3-4B started at 58% and got the meaning wrong, not the syntax. Four GRPO rounds and one SFT pass reached 61%, a gain we cannot separate from noise. In the first two rounds the training reward rose while the test score did not. Without the held-out set, that run would have shipped.

Round five: the same reward, fed differently

A larger run on another schema used the same reward, tasks and model. Three things about feeding the trainer differed.

We also grew the test, because a 6-point band can never prove a 3-point gain. On the bigger test the round-four gain turned out to be real.

What the model learned is reliability, not new ability.

Next. The curve is flat from step 50. Re-measure the difficulty band on the round-five model, then a bigger base or harder questions.

Qwen3.5 on the same test

Does a newer base start higher? Same prompt, reply budget and 459 questions.

One round of the round-five recipe puts the 9B above every Qwen3-4B checkpoint, and a second scoring pass on the same adapter landed within a point of the first. Three quarters of the questions it now gets right all four times. Most of the gain comes early: 79% at step 25, 81% at step 50, 82% at step 75, and round two's first checkpoint, 83%, is within noise of that. Round three re-picked the training questions from the step-75 model itself, so the trainer saw questions it still gets wrong sometimes, and its first checkpoint is 84%: two points over step 75, an interval that just clears zero. Step 50, the end of the round, is 84% again, with the easy and the hard questions both up. That is where this model stops on this test: 84% on the first try, 88% in four tries, and the recipe has no lever left that the interval can see.

We nearly missed it. For two days every checkpoint of this model scored the same as its base, and we wrote the run up as flat. The training log had said so all along: the trainer's own reward on the questions it trained on was 0.95 while the served adapter scored 0.65 on them. Both bugs are fixed.

For researchers

Each held-out task $i$ gets $n = 4$ samples at temperature 0.7 and $c_i$ pass. pass@$k$ is the estimator of Chen et al. [3], pass$^k$ the share passing every sample, intervals are percentile bootstraps over tasks [4], $B = 2{,}000$:

The difference is paired on tasks, $\hat p_i = c_i / n$ each side:

The verifier is a program, so the reward is verifiable in the sense of [1, §7]. For a candidate query $q$ and the reference $q^\star$,

Floats compare to two decimals, column order is free, row order counts only when $q^\star$ has ORDER BY. GRPO [2] on a LoRA adapter [5], TRL [7], vLLM generation [6]. Advantage $A_j = (r_j - \operatorname{mean}(r)) / \operatorname{std}(r)$ over the $G$ samples of a prompt, KL penalty against the base with coefficient $\beta$.

References

1. Lambert, N. (2025). Reinforcement Learning from Human Feedback. arXiv:2504.12501. Online at rlhfbook.com.
2. Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., Bi, X., Zhang, H., Zhang, M., Li, Y. K., Wu, Y. and Guo, D. (2024). DeepSeekMath: Pushing the limits of mathematical reasoning in open language models. arXiv:2402.03300.
3. Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. de O., Kaplan, J., et al. (2021). Evaluating large language models trained on code. arXiv:2107.03374.
4. Efron, B. (1979). Bootstrap methods: Another look at the jackknife. The Annals of Statistics, 7(1), 1-26.
5. Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L. and Chen, W. (2022). LoRA: Low-rank adaptation of large language models. International Conference on Learning Representations. arXiv:2106.09685.
6. Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J. E., Zhang, H. and Stoica, I. (2023). Efficient memory management for large language model serving with PagedAttention. Proceedings of the 29th ACM Symposium on Operating Systems Principles. arXiv:2309.06180.
7. von Werra, L., Belkada, Y., Tunstall, L., Beeching, E., Thrush, T., Lambert, N., Huang, S., Rasul, K. and Gallouédec, Q. (2020). TRL: Transformer Reinforcement Learning [software]. github.com/huggingface/trl.
8. whilehq (2026). whileai SDK [software]. Apache 2.0. github.com/whilehq/whileai-sdk.

Run it

pip install "whileai>=0.51" "psycopg[binary]" openai anthropic
git clone https://github.com/whilehq/whileai-sdk
cd whileai-sdk/recipes/04-train/text-to-sql
python rollout.py --model qwen3-4b --split holdout --k 4
python build.py

The README has the database setup, the Modal command for training, and delta.py for the paired score.