Tested 2026-10-28
Fixing bugs in small Python functions: Auto Router won, 3 of 7 models failed a must-have
Find and fix the bug in a short Python function and return only the corrected function — for a solo developer maintaining a small web app.
What this project actually needed
- takes written text as input — must have
- produces written text — must have
- returns a corrected function that gives the expected result on every example — must have: a working fix on every task
- is good at coding — decides the order
- cost at your volume — decides the order
The winner
Auto Router passed every must-have and ranked first by how this job said to choose.
OpenRouter's automatic picker sent this job to deepseek/deepseek-v4.1-flash-1017 and won. It picks one message at a time and never sees this job's must-haves.
Who failed, in their own words
OpenAI: o4 Mini High
- gives an answer:
no answer came back after three tries (HTTP 404: b'{"error":{"message":"openai/o4-mini-high is no l)
Mistral: Mistral Nemo
- returns a corrected function that gives the expected result on every example:
Here's the corrected function. The issue with the original function is that it's not ignoring case when checking if a word has been seen before.
- facts match the task:
The function you provided is already correct and should return the expected results.
PrismML: Ternary Bonsai 2 9B
- returns a corrected function that gives the expected result on every example:
repeated('The cat and the dog and THE bird') == ['the', 'and', 'the']
Every model on every requirement
| Model | is good at coding | cost at your volume | produces written text | takes written text as input | returns a corrected function that gives the expected result on every example |
|---|---|---|---|---|---|
| Auto Router | scored 1.00 of 1 across 6 tasks (graded) | $0.0141 a day (measured on your tasks) | not applicable — the picker chooses a model per request | not applicable — the picker chooses a model per request | passed 6 of 6 tasks |
| Meta: Muse Wonder | scored 0.97 of 1 across 3 tasks (graded) | $1.0828 a day (measured on your tasks) | yes (estimate — not observed) | yes (estimate — not observed) | passed 3 of 3 tasks |
| OpenAI: o4 Mini High | not judged — no graded answers | not measured | yes (estimate — not observed) | yes (estimate — not observed) | not judged — no graded answers |
| Mistral: Mistral Nemo | scored 0.30 of 1 across 3 tasks (graded) | $0.0006 a day (measured on your tasks) | yes (estimate — not observed) | yes (estimate — not observed) | passed 1 of 3 tasks |
| Anthropic: Claude Opus 5.5 | scored 1.00 of 1 across 3 tasks (graded) | $0.3389 a day (measured on your tasks) | yes (estimate — not observed) | yes (estimate — not observed) | passed 3 of 3 tasks |
| PrismML: Ternary Bonsai 2 9B | scored 0.77 of 1 across 3 tasks (graded) | $0.0109 a day (measured on your tasks) | yes (estimate — not observed) | yes (estimate — not observed) | passed 2 of 3 tasks |
7 models tested on python bug fix; cost of this test $0.0692.
Your app does something like this? XF Model Watch keeps it on the best model. XF Model Watch re-tests your own job as new models arrive and emails you only when one passes every check. Watch this job