Tested 2026-10-23
Answering research questions from given sources: Anthropic: Claude Sonnet 5.5 won, 1 of 7 models failed a must-have; OpenRouter's own picker came 3rd of 6
Answer research questions using only the source excerpts given, naming the excerpt behind each claim, and saying when the sources don't answer it — for a two-person market research shop.
What this project actually needed
- takes written text as input — must have
- produces written text — must have
- answers only from the given excerpts, cites them, and says so when they don't answer the question — must have: a supported, cited answer on every task
- is good at reasoning — decides the order
- cost at your volume — decides the order
The winner
Anthropic: Claude Sonnet 5.5 passed every must-have and ranked first by how this job said to choose.
Dark horse: MiniMax: MiniMax M3.1 Flash passed every must-have and came 4th of 6 at 10× less than Anthropic: Claude Sonnet 5.5.
OpenRouter's automatic picker sent this job to deepseek/deepseek-v4.1-flash-1017, openai/gpt-6-luna and came 3rd of 6. It picks one message at a time and never sees this job's must-haves.
Who failed, in their own words
Mistral: Mistral Nemo
- answers only from the given excerpts, cites them, and says so when they don't answer the question:
Other factors may contribute to the customer ratings, such as the quality of responses, ease of communication, or overall customer experience.
- facts match the task:
Other factors may contribute to the customer ratings, such as the quality of responses, ease of communication, or overall customer experience.
Every model on every requirement
| Model | cost at your volume | is good at reasoning | produces written text | takes written text as input | answers only from the given excerpts, cites them, and says so when they don't answer the question |
|---|---|---|---|---|---|
| Auto Router | $0.0033 a day (measured on your tasks) | scored 0.90 of 1 across 3 tasks (graded) | not applicable — the picker chooses a model per request | not applicable — the picker chooses a model per request | passed 3 of 3 tasks |
| OpenAI: GPT-6 Astra | $0.1128 a day (measured on your tasks) | scored 0.90 of 1 across 3 tasks (graded) | yes (estimate — not observed) | yes (estimate — not observed) | passed 3 of 3 tasks |
| Mistral: Mistral Nemo | $0.0002 a day (measured on your tasks) | scored 0.87 of 1 across 3 tasks (graded) | yes (estimate — not observed) | yes (estimate — not observed) | passed 2 of 3 tasks |
| Google: Gemini 4 Argon | $0.0282 a day (measured on your tasks) | scored 0.87 of 1 across 3 tasks (graded) | yes (estimate — not observed) | yes (estimate — not observed) | passed 3 of 3 tasks |
| Anthropic: Claude Opus 5.5 | $0.2047 a day (measured on your tasks) | scored 1.00 of 1 across 3 tasks (graded) | yes (estimate — not observed) | yes (estimate — not observed) | passed 3 of 3 tasks |
| MiniMax: MiniMax M3.1 Flash | $0.0066 a day (measured on your tasks) | scored 0.93 of 1 across 3 tasks (graded) | yes (estimate — not observed) | yes (estimate — not observed) | passed 3 of 3 tasks |
| Anthropic: Claude Sonnet 5.5 | $0.0642 a day (measured on your tasks) | scored 1.00 of 1 across 3 tasks (graded) | yes (estimate — not observed) | yes (estimate — not observed) | passed 3 of 3 tasks |
7 models tested on research from sources; cost of this test $0.0667.
Your app does something like this? XF Model Watch keeps it on the best model. XF Model Watch re-tests your own job as new models arrive and emails you only when one passes every check. Watch this job