Tested 2026-10-15
Replies to customer support emails: Qwen: Qwen3.7 Max won, 6 of 7 models failed a must-have
Draft replies to customer support emails for a small SaaS — for a 3-person software company.
What this project actually needed
- takes written text as input — must have
- produces written text — must have
- never promises refunds or features that aren't in the policy — must have: no promise beyond the policy in any reply
- writes a clear, polite reply — decides the order
- cost at your volume — decides the order
The winner
Qwen: Qwen3.7 Max passed every must-have and ranked first by how this job said to choose.
OpenRouter's automatic picker sent this job to deepseek/deepseek-v4-flash-0731, google/gemini-2.5-flash and was not ranked. It picks one message at a time and never sees this job's must-haves.
Who failed, in their own words
SpaceXAI: Grok 4.20
- never promises refunds or features that aren't in the policy:
It's included starting from the Pro plan and above.
- facts match the task:
It's included starting from the Pro plan and above.
Mistral: Mistral Nemo
- never promises refunds or features that aren't in the policy:
it's only accessible on our Professional and Enterprise plans.
- facts match the task:
it's only accessible on our Professional and Enterprise plans.
Xiaomi: MiMo-V2.6-Pro
- never promises refunds or features that aren't in the policy:
If you change your mind within that 30-day window, you may still be able to recover your account before it's permanently gone.
- facts match the task:
If you change your mind within that 30-day window, you may still be able to recover your account before it's permanently gone.
NVIDIA: Nemotron 3 Ultra
- never promises refunds or features that aren't in the policy:
it’s available on our **Professional** and **Enterprise** plans.
- facts match the task:
it’s available on our **Professional** and **Enterprise** plans.
Auto Router
- never promises refunds or features that aren't in the policy:
During this 30-day period, your account data is queued for deletion and will no longer be accessible to you or anyone else.
- facts match the task:
will no longer be accessible to you or anyone else
Z.ai: GLM 5.3 Flash
- never promises refunds or features that aren't in the policy:
If you change your mind during that 30-day window, just reach out to us and we'll be happy to help.
- facts match the task:
If you change your mind during that 30-day window, just reach out to us and we'll be happy to help.
Every model on every requirement
| Model | cost at your volume | produces written text | takes written text as input | writes a clear, polite reply | never promises refunds or features that aren't in the policy |
|---|---|---|---|---|---|
| Auto Router | $0.0802 a day (measured on your tasks) | not applicable — the picker chooses a model per request | not applicable — the picker chooses a model per request | scored 0.93 of 1 across 6 tasks (graded) | passed 5 of 6 tasks |
| Qwen: Qwen3.7 Max | $1.2006 a day (measured on your tasks) | yes (estimate — not observed) | yes (estimate — not observed) | scored 0.97 of 1 across 6 tasks (graded) | passed 6 of 6 tasks |
| SpaceXAI: Grok 4.20 | $0.0777 a day (measured on your tasks) | yes (estimate — not observed) | yes (estimate — not observed) | scored 0.91 of 1 across 6 tasks (graded) | passed 4 of 6 tasks |
| Z.ai: GLM 5.3 Flash | $0.1294 a day (measured on your tasks) | yes (estimate — not observed) | yes (estimate — not observed) | scored 0.92 of 1 across 6 tasks (graded) | passed 5 of 6 tasks |
| Mistral: Mistral Nemo | $0.0012 a day (measured on your tasks) | yes (estimate — not observed) | yes (estimate — not observed) | scored 0.87 of 1 across 6 tasks (graded) | passed 4 of 6 tasks |
| Xiaomi: MiMo-V2.6-Pro | $0.0550 a day (measured on your tasks) | yes (estimate — not observed) | yes (estimate — not observed) | scored 0.91 of 1 across 6 tasks (graded) | passed 4 of 6 tasks |
| NVIDIA: Nemotron 3 Ultra | $0.1055 a day (measured on your tasks) | yes (estimate — not observed) | yes (estimate — not observed) | scored 0.91 of 1 across 6 tasks (graded) | passed 4 of 6 tasks |
7 models tested on support email replies; cost of this test $0.1043.
Your app does something like this? XF Model Watch keeps it on the best model. XF Model Watch re-tests your own job as new models arrive and emails you only when one passes every check. Watch this job