Tested 2026-10-17
Replies to customer support emails: Qwen: Qwen3.7 Max won, 5 of 7 models failed a must-have
Draft replies to customer support emails for a small SaaS — for a 3-person software company.
What this project actually needed
- takes written text as input — must have
- produces written text — must have
- never promises refunds or features that aren't in the policy — must have: no promise beyond the policy in any reply
- writes a clear, polite reply — decides the order
- cost at your volume — decides the order
The winner
Qwen: Qwen3.7 Max passed every must-have and ranked first by how this job said to choose.
OpenRouter's automatic picker sent this job to google/gemini-2.5-flash and was not ranked. It picks one message at a time and never sees this job's must-haves.
Who failed, in their own words
Z.ai: GLM 5.3 Flash
- never promises refunds or features that aren't in the policy:
Switching to a monthly plan and crediting the difference toward it - Pausing your subscription if you plan to return later
- facts match the task:
Pro and Business plans
Mistral: Mistral Nemo
- never promises refunds or features that aren't in the policy:
I have made a note of your request and will do my best to prioritize your order in our processing queue. While I cannot guarantee that it will arrive by Friday, I assure you that we will do everything possible to ensure
- facts match the task:
Starter plan, which is our free tier
Xiaomi: MiMo-V2.6-Pro
- never promises refunds or features that aren't in the policy:
If you believe there's been a billing error or extenuating circumstance, please let me know and I can escalate the matter for further review.
- facts match the task:
account can be restored
NVIDIA: Nemotron 3 Ultra
- never promises refunds or features that aren't in the policy:
The link expires after [e.g., 1 hour/24 hours] for your protection.
- facts match the task:
Professional and Enterprise plans
Auto Router
- never promises refunds or features that aren't in the policy:
During this 30-day period, the deletion is reversible if you change your mind.
- facts match the task:
the deletion is reversible if you change your mind
Every model on every requirement
| Model | cost at your volume | produces written text | takes written text as input | writes a clear, polite reply | never promises refunds or features that aren't in the policy |
|---|---|---|---|---|---|
| Auto Router | $0.0834 a day (measured on your tasks) | not applicable — the picker chooses a model per request | not applicable — the picker chooses a model per request | scored 0.98 of 1 across 6 tasks (graded) | passed 5 of 6 tasks |
| Meta: Muse Wonder | $2.3173 a day (measured on your tasks) | yes (estimate — not observed) | yes (estimate — not observed) | scored 0.99 of 1 across 6 tasks (graded) | passed 6 of 6 tasks |
| Qwen: Qwen3.7 Max | $1.0315 a day (measured on your tasks) | yes (estimate — not observed) | yes (estimate — not observed) | scored 0.99 of 1 across 6 tasks (graded) | passed 6 of 6 tasks |
| Z.ai: GLM 5.3 Flash | $0.0751 a day (measured on your tasks) | yes (estimate — not observed) | yes (estimate — not observed) | scored 0.90 of 1 across 6 tasks (graded) | passed 4 of 6 tasks |
| Mistral: Mistral Nemo | $0.0012 a day (measured on your tasks) | yes (estimate — not observed) | yes (estimate — not observed) | scored 0.89 of 1 across 6 tasks (graded) | passed 4 of 6 tasks |
| Xiaomi: MiMo-V2.6-Pro | $0.0551 a day (measured on your tasks) | yes (estimate — not observed) | yes (estimate — not observed) | scored 0.93 of 1 across 6 tasks (graded) | passed 4 of 6 tasks |
| NVIDIA: Nemotron 3 Ultra | $0.1149 a day (measured on your tasks) | yes (estimate — not observed) | yes (estimate — not observed) | scored 0.90 of 1 across 6 tasks (graded) | passed 4 of 6 tasks |
7 models tested on support email replies; cost of this test $0.1805.
Your app does something like this? XF Model Watch keeps it on the best model. XF Model Watch re-tests your own job as new models arrive and emails you only when one passes every check. Watch this job