Tested 2026-10-24
Fixing bugs in small Python functions: Auto Router won, 3 of 6 models failed a must-have
Find and fix the bug in a short Python function and return only the corrected function — for a solo developer maintaining a small web app.
What this project actually needed
- takes written text as input — must have
- produces written text — must have
- returns a corrected function that gives the expected result on every example — must have: a working fix on every task
- is good at coding — decides the order
- cost at your volume — decides the order
The winner
Auto Router passed every must-have and ranked first by how this job said to choose.
OpenRouter's automatic picker sent this job to deepseek/deepseek-v4.1-flash-1017 and won. It picks one message at a time and never sees this job's must-haves.
Who failed, in their own words
OpenAI: o4 Mini High
- gives an answer:
no answer came back after three tries (HTTP 404: b'{"error":{"message":"Provider returned error","c)
Mistral: Mistral Nemo
- returns a corrected function that gives the expected result on every example:
The function you provided is almost correct, but it's not ignoring case as specified in the problem statement. To fix this, we should convert all words to lowercase before checking if they've been seen or addin
- facts match the task:
The days attribute of a timedelta object (which is what (d2 - d1).days returns) always gives a positive number, even if d2 is earlier than d1.
MoonshotAI: Kimi K3.1 Pro
- returns a corrected function that gives the expected result on every example:
def days_between(a, b): d1 = date.fromisoformat(a)
Every model on every requirement
| Model | is good at coding | cost at your volume | produces written text | takes written text as input | returns a corrected function that gives the expected result on every example |
|---|---|---|---|---|---|
| Auto Router | scored 1.00 of 1 across 3 tasks (graded) | $0.0341 a day (measured on your tasks) | not applicable — the picker chooses a model per request | not applicable — the picker chooses a model per request | passed 3 of 3 tasks |
| Meta: Muse Wonder | scored 1.00 of 1 across 3 tasks (graded) | $1.3801 a day (measured on your tasks) | yes (estimate — not observed) | yes (estimate — not observed) | passed 3 of 3 tasks |
| OpenAI: o4 Mini High | not judged — no graded answers | not measured | yes (estimate — not observed) | yes (estimate — not observed) | not judged — no graded answers |
| Mistral: Mistral Nemo | scored 0.27 of 1 across 3 tasks (graded) | $0.0008 a day (measured on your tasks) | yes (estimate — not observed) | yes (estimate — not observed) | passed 1 of 3 tasks |
| MoonshotAI: Kimi K3.1 Pro | scored 0.83 of 1 across 3 tasks (graded) | $0.4948 a day (measured on your tasks) | yes (estimate — not observed) | yes (estimate — not observed) | passed 2 of 3 tasks |
| Anthropic: Claude Opus 5.5 | scored 1.00 of 1 across 3 tasks (graded) | $0.3543 a day (measured on your tasks) | yes (estimate — not observed) | yes (estimate — not observed) | passed 3 of 3 tasks |
6 models tested on python bug fix; cost of this test $0.1152.
Your app does something like this? XF Model Watch keeps it on the best model. XF Model Watch re-tests your own job as new models arrive and emails you only when one passes every check. Watch this job