Tested 2026-10-24
Writing SQL queries from plain-English questions: Auto Router won, 2 of 6 models failed a must-have
Turn plain-English questions into SQL for a known database schema — for a two-person analytics consultancy.
What this project actually needed
- takes written text as input — must have
- produces written text — must have
- writes SQL that answers the question using only the given tables — must have: correct, runnable SQL on every task
- is good at coding — decides the order
- cost at your volume — decides the order
The winner
Auto Router passed every must-have and ranked first by how this job said to choose.
OpenRouter's automatic picker sent this job to z-ai/glm-5.3-flash and won. It picks one message at a time and never sees this job's must-haves.
Who failed, in their own words
OpenAI: o4 Mini High
- gives an answer:
no answer came back after three tries (HTTP 404: b'{"error":{"message":"Provider returned error","c)
Mistral: Mistral Nemo
- writes SQL that answers the question using only the given tables:
ORDER BY month;
Every model on every requirement
| Model | is good at coding | cost at your volume | produces written text | takes written text as input | writes SQL that answers the question using only the given tables |
|---|---|---|---|---|---|
| Auto Router | scored 1.00 of 1 across 6 tasks (graded) | $0.0079 a day (measured on your tasks) | not applicable — the picker chooses a model per request | not applicable — the picker chooses a model per request | passed 6 of 6 tasks |
| Meta: Muse Wonder | scored 1.00 of 1 across 3 tasks (graded) | $0.4933 a day (measured on your tasks) | yes (estimate — not observed) | yes (estimate — not observed) | passed 3 of 3 tasks |
| OpenAI: o4 Mini High | not judged — no graded answers | not measured | yes (estimate — not observed) | yes (estimate — not observed) | not judged — no graded answers |
| Mistral: Mistral Nemo | scored 0.78 of 1 across 3 tasks (graded) | $0.0001 a day (measured on your tasks) | yes (estimate — not observed) | yes (estimate — not observed) | passed 2 of 3 tasks |
| Anthropic: Claude Opus 5.5 | scored 1.00 of 1 across 3 tasks (graded) | $0.1819 a day (measured on your tasks) | yes (estimate — not observed) | yes (estimate — not observed) | passed 3 of 3 tasks |
6 models tested on sql from question; cost of this test $0.0787.
Your app does something like this? XF Model Watch keeps it on the best model. XF Model Watch re-tests your own job as new models arrive and emails you only when one passes every check. Watch this job