Tested 2026-10-25
Replies to customer support emails: Qwen: Qwen3.7 Max won, 5 of 7 models failed a must-have; OpenRouter's own picker came 2nd of 2
Draft replies to customer support emails for a small SaaS — for a 3-person software company.
What this project actually needed
- takes written text as input — must have
- produces written text — must have
- never promises refunds or features that aren't in the policy — must have: no promise beyond the policy in any reply
- writes a clear, polite reply — decides the order
- cost at your volume — decides the order
The winner
Qwen: Qwen3.7 Max passed every must-have and ranked first by how this job said to choose.
OpenRouter's automatic picker sent this job to openai/gpt-6-luna and came 2nd of 2. It picks one message at a time and never sees this job's must-haves.
Who failed, in their own words
SpaceXAI: Grok 4.20
- never promises refunds or features that aren't in the policy:
It's included starting from the **Growth** plan and above.
- facts match the task:
It's included starting from the **Growth** plan and above.
Z.ai: GLM 5.3 Flash
- never promises refunds or features that aren't in the policy:
reach us through live chat on our website
- facts match the task:
reach us through live chat on our website
Mistral: Mistral Nemo
- never promises refunds or features that aren't in the policy:
we strive to respond to all queries within 24 hours
- facts match the task:
we strive to respond to all queries within 24 hours
Xiaomi: MiMo-V2.6-Pro
- never promises refunds or features that aren't in the policy:
If you change your mind during those 30 days, simply reach out to us and we can assist with restoring your account
- facts match the task:
If you change your mind during those 30 days, simply reach out to us and we can assist with restoring your account
NVIDIA: Nemotron 3 Ultra
- never promises refunds or features that aren't in the policy:
CSV export is available on our **Professional** plan and above.
- facts match the task:
CSV export is available on our **Professional** plan and above.
Every model on every requirement
| Model | cost at your volume | produces written text | takes written text as input | writes a clear, polite reply | never promises refunds or features that aren't in the policy |
|---|---|---|---|---|---|
| Auto Router | $0.0146 a day (measured on your tasks) | not applicable — the picker chooses a model per request | not applicable — the picker chooses a model per request | scored 0.85 of 1 across 6 tasks (graded) | passed 6 of 6 tasks |
| Qwen: Qwen3.7 Max | $1.0997 a day (measured on your tasks) | yes (estimate — not observed) | yes (estimate — not observed) | scored 0.97 of 1 across 6 tasks (graded) | passed 6 of 6 tasks |
| SpaceXAI: Grok 4.20 | $0.0832 a day (measured on your tasks) | yes (estimate — not observed) | yes (estimate — not observed) | scored 0.88 of 1 across 6 tasks (graded) | passed 4 of 6 tasks |
| Z.ai: GLM 5.3 Flash | $0.0073 a day (measured on your tasks) | yes (estimate — not observed) | yes (estimate — not observed) | scored 0.90 of 1 across 6 tasks (graded) | passed 3 of 6 tasks |
| Mistral: Mistral Nemo | $0.0012 a day (measured on your tasks) | yes (estimate — not observed) | yes (estimate — not observed) | scored 0.86 of 1 across 6 tasks (graded) | passed 3 of 6 tasks |
| Xiaomi: MiMo-V2.6-Pro | $0.0493 a day (measured on your tasks) | yes (estimate — not observed) | yes (estimate — not observed) | scored 0.94 of 1 across 6 tasks (graded) | passed 4 of 6 tasks |
| NVIDIA: Nemotron 3 Ultra | $0.1027 a day (measured on your tasks) | yes (estimate — not observed) | yes (estimate — not observed) | scored 0.92 of 1 across 6 tasks (graded) | passed 4 of 6 tasks |
7 models tested on support email replies; cost of this test $0.0952.
Your app does something like this? XF Model Watch keeps it on the best model. XF Model Watch re-tests your own job as new models arrive and emails you only when one passes every check. Watch this job