Tested 2026-10-20
Writing SQL queries from plain-English questions: Auto Router won, 0 of 5 models failed a must-have
Turn plain-English questions into SQL for a known database schema — for a two-person analytics consultancy.
What this project actually needed
- takes written text as input — must have
- produces written text — must have
- writes SQL that answers the question using only the given tables — must have: correct, runnable SQL on every task
- is good at coding — decides the order
- cost at your volume — decides the order
The winner
Auto Router passed every must-have and ranked first by how this job said to choose.
Dark horse: Mistral: Mistral Nemo passed every must-have and came 5th of 5 at 43× less than Auto Router.
OpenRouter's automatic picker sent this job to openai/gpt-6-luna and won. It picks one message at a time and never sees this job's must-haves.
Every model on every requirement
| Model | is good at coding | cost at your volume | produces written text | takes written text as input | writes SQL that answers the question using only the given tables |
|---|---|---|---|---|---|
| Auto Router | scored 1.00 of 1 across 3 tasks (graded) | $0.0043 a day (measured on your tasks) | not applicable — the picker chooses a model per request | not applicable — the picker chooses a model per request | passed 3 of 3 tasks |
| Meta: Muse Wonder | scored 1.00 of 1 across 3 tasks (graded) | $0.5287 a day (measured on your tasks) | yes (estimate — not observed) | yes (estimate — not observed) | passed 3 of 3 tasks |
| OpenAI: o4 Mini High | scored 1.00 of 1 across 3 tasks (graded) | $0.1460 a day (measured on your tasks) | yes (estimate — not observed) | yes (estimate — not observed) | passed 3 of 3 tasks |
| Mistral: Mistral Nemo | scored 0.87 of 1 across 3 tasks (graded) | $0.0001 a day (measured on your tasks) | yes (estimate — not observed) | yes (estimate — not observed) | passed 3 of 3 tasks |
| Anthropic: Claude Sonnet 5.5 | scored 1.00 of 1 across 3 tasks (graded) | $0.0596 a day (measured on your tasks) | yes (estimate — not observed) | yes (estimate — not observed) | passed 3 of 3 tasks |
5 models tested on sql from question; cost of this test $0.0729.
Your app does something like this? XF Model Watch keeps it on the best model. XF Model Watch re-tests your own job as new models arrive and emails you only when one passes every check. Watch this job