Tested 2026-11-05
Answering research questions from given sources: Tencent: Hy4 won, 5 of 9 models failed a must-have; OpenRouter's own picker came 2nd of 4
Answer research questions using only the source excerpts given, naming the excerpt behind each claim, and saying when the sources don't answer it — for a two-person market research shop.
What this project actually needed
- takes written text as input — must have
- produces written text — must have
- answers only from the given excerpts, cites them, and says so when they don't answer the question — must have: a supported, cited answer on every task
- is good at reasoning — decides the order
- cost at your volume — decides the order
The winner
Tencent: Hy4 passed every must-have and ranked first by how this job said to choose.
Dark horse: ricursiveAI/Ripple-65B-Preview passed every must-have and came 4th of 4 at 17× less than Tencent: Hy4.
OpenRouter's automatic picker sent this job to openai/gpt-6-luna, openai/gpt-6.1-sol and came 2nd of 4. It picks one message at a time and never sees this job's must-haves.
Who failed, in their own words
Mistral: Mistral Nemo
- answers only from the given excerpts, cites them, and says so when they don't answer the question:
Other factors, not mentioned in the sources, might also contribute to the customer satisfaction ratings.
Xiaomi: MiMo-V2.6-Pro
- answers only from the given excerpts, cites them, and says so when they don't answer the question:
compared with about 197 in 2023 (figures derived from [1] and [3])
- facts match the task:
compared with about 197 in 2023
Anthropic: Claude Opus 4.6
- answers only from the given excerpts, cites them, and says so when they don't answer the question:
other factors not covered in these sources must contribute to Brand B's superior rating.
Anthropic: Claude Opus 5.5
- answers only from the given excerpts, cites them, and says so when they don't answer the question:
Other factors may matter more, such as answer quality, other support channels, or customer expectations, but the sources contain no data on these.
Anthropic: Claude Sonnet 5.5
- answers only from the given excerpts, cites them, and says so when they don't answer the question:
They give only average response time and the percentage rating "excellent," and nothing on other factors such as answer quality, problem resolution, or customer mix.
Every model on every requirement
| Model | cost at your volume | is good at reasoning | produces written text | takes written text as input | answers only from the given excerpts, cites them, and says so when they don't answer the question |
|---|---|---|---|---|---|
| Auto Router | $0.0089 a day (measured on your tasks) | scored 0.93 of 1 across 3 tasks (graded) | not applicable — the picker chooses a model per request | not applicable — the picker chooses a model per request | passed 3 of 3 tasks |
| Tencent: Hy4 | $0.0306 a day (measured on your tasks) | scored 1.00 of 1 across 2 tasks (graded) | yes (estimate — not observed) | yes (estimate — not observed) | passed 2 of 2 tasks |
| Qwen: Qwen3.7 Max | $0.1216 a day (measured on your tasks) | scored 0.97 of 1 across 3 tasks (graded) | yes (estimate — not observed) | yes (estimate — not observed) | passed 3 of 3 tasks |
| Mistral: Mistral Nemo | $0.0002 a day (measured on your tasks) | scored 0.87 of 1 across 3 tasks (graded) | yes (estimate — not observed) | yes (estimate — not observed) | passed 2 of 3 tasks |
| Xiaomi: MiMo-V2.6-Pro | $0.0065 a day (measured on your tasks) | scored 0.90 of 1 across 3 tasks (graded) | yes (estimate — not observed) | yes (estimate — not observed) | passed 2 of 3 tasks |
| Anthropic: Claude Opus 4.6 | $0.0965 a day (measured on your tasks) | scored 0.93 of 1 across 3 tasks (graded) | yes (estimate — not observed) | yes (estimate — not observed) | passed 2 of 3 tasks |
| Anthropic: Claude Opus 5.5 | $0.1985 a day (measured on your tasks) | scored 0.97 of 1 across 3 tasks (graded) | yes (estimate — not observed) | yes (estimate — not observed) | passed 2 of 3 tasks |
| Anthropic: Claude Sonnet 5.5 | $0.0693 a day (measured on your tasks) | scored 0.93 of 1 across 3 tasks (graded) | yes (estimate — not observed) | yes (estimate — not observed) | passed 2 of 3 tasks |
| ricursiveAI/Ripple-65B-Preview | $0.0018 a day (measured on your tasks) | scored 0.90 of 1 across 3 tasks (graded) | yes (estimate — not observed) | yes (estimate — not observed) | passed 3 of 3 tasks |
9 models tested on research from sources; cost of this test $0.0853.
Your app does something like this? XF Model Watch keeps it on the best model. XF Model Watch re-tests your own job as new models arrive and emails you only when one passes every check. Watch this job