Tested 2026-10-21
Replies to customer support emails: Qwen: Qwen3.7 Max won, 5 of 7 models failed a must-have; OpenRouter's own picker came 2nd of 2
Draft replies to customer support emails for a small SaaS — for a 3-person software company.
What this project actually needed
- takes written text as input — must have
- produces written text — must have
- never promises refunds or features that aren't in the policy — must have: no promise beyond the policy in any reply
- writes a clear, polite reply — decides the order
- cost at your volume — decides the order
The winner
Qwen: Qwen3.7 Max passed every must-have and ranked first by how this job said to choose.
OpenRouter's automatic picker sent this job to google/gemini-2.5-flash and came 2nd of 2. It picks one message at a time and never sees this job's must-haves.
Who failed, in their own words
SpaceXAI: Grok 4.20
- never promises refunds or features that aren't in the policy:
During that 30-day period, you can cancel the deletion by simply logging back in.
- facts match the task:
starting from the **Pro** plan and above
Mistral: Mistral Nemo
- never promises refunds or features that aren't in the policy:
We also offer free shipping on orders over $50, which you might find beneficial.
- facts match the task:
reserved for our Pro and Enterprise plans
NVIDIA: Nemotron 3 Ultra
- never promises refunds or features that aren't in the policy:
During this time, your data is retained and you can cancel the deletion by logging back in.
- facts match the task:
available on our **Professional** plan and above
Z.ai: GLM 5.3 Flash
- facts match the task:
available on our Growth and Enterprise plans
Xiaomi: MiMo-V2.6-Pro
- never promises refunds or features that aren't in the policy:
During this window, if you change your mind, you may be able to contact us to reverse the deletion.
- facts match the task:
you may be able to contact us to reverse the deletion
Every model on every requirement
| Model | cost at your volume | produces written text | takes written text as input | writes a clear, polite reply | never promises refunds or features that aren't in the policy |
|---|---|---|---|---|---|
| Auto Router | $0.0722 a day (measured on your tasks) | not applicable — the picker chooses a model per request | not applicable — the picker chooses a model per request | scored 0.94 of 1 across 6 tasks (graded) | passed 6 of 6 tasks |
| Qwen: Qwen3.7 Max | $0.9560 a day (measured on your tasks) | yes (estimate — not observed) | yes (estimate — not observed) | scored 0.98 of 1 across 6 tasks (graded) | passed 6 of 6 tasks |
| SpaceXAI: Grok 4.20 | $0.0796 a day (measured on your tasks) | yes (estimate — not observed) | yes (estimate — not observed) | scored 0.86 of 1 across 6 tasks (graded) | passed 5 of 6 tasks |
| Z.ai: GLM 5.3 Flash | $0.0144 a day (measured on your tasks) | yes (estimate — not observed) | yes (estimate — not observed) | scored 0.97 of 1 across 6 tasks (graded) | passed 6 of 6 tasks |
| Mistral: Mistral Nemo | $0.0012 a day (measured on your tasks) | yes (estimate — not observed) | yes (estimate — not observed) | scored 0.89 of 1 across 6 tasks (graded) | passed 5 of 6 tasks |
| Xiaomi: MiMo-V2.6-Pro | $0.0539 a day (measured on your tasks) | yes (estimate — not observed) | yes (estimate — not observed) | scored 0.93 of 1 across 6 tasks (graded) | passed 5 of 6 tasks |
| NVIDIA: Nemotron 3 Ultra | $0.1039 a day (measured on your tasks) | yes (estimate — not observed) | yes (estimate — not observed) | scored 0.92 of 1 across 6 tasks (graded) | passed 5 of 6 tasks |
7 models tested on support email replies; cost of this test $0.0909.
Your app does something like this? XF Model Watch keeps it on the best model. XF Model Watch re-tests your own job as new models arrive and emails you only when one passes every check. Watch this job