Tested 2026-10-27
Answering research questions from given sources: Anthropic: Claude Sonnet 5.5 won, 1 of 8 models failed a must-have; OpenRouter's own picker came 3rd of 7
Answer research questions using only the source excerpts given, naming the excerpt behind each claim, and saying when the sources don't answer it — for a two-person market research shop.
What this project actually needed
- takes written text as input — must have
- produces written text — must have
- answers only from the given excerpts, cites them, and says so when they don't answer the question — must have: a supported, cited answer on every task
- is good at reasoning — decides the order
- cost at your volume — decides the order
The winner
Anthropic: Claude Sonnet 5.5 passed every must-have and ranked first by how this job said to choose.
Dark horse: OpenAI: GPT-6.1 Terra passed every must-have and came 4th of 7 at 4× less than Anthropic: Claude Sonnet 5.5.
OpenRouter's automatic picker sent this job to anthropic/claude-haiku-5.5, deepseek/deepseek-v4.1-flash-1017 and came 3rd of 7. It picks one message at a time and never sees this job's must-haves.
Who failed, in their own words
Mistral: Mistral Nemo
- answers only from the given excerpts, cites them, and says so when they don't answer the question:
From 2023 to 2025, the percentage of bakeries in Ohio that sold online increased from 48% to 62%.
Every model on every requirement
| Model | cost at your volume | is good at reasoning | produces written text | takes written text as input | answers only from the given excerpts, cites them, and says so when they don't answer the question |
|---|---|---|---|---|---|
| Auto Router | $0.0060 a day (measured on your tasks) | scored 0.93 of 1 across 3 tasks (graded) | not applicable — the picker chooses a model per request | not applicable — the picker chooses a model per request | passed 3 of 3 tasks |
| Mistral: Mistral Nemo | $0.0002 a day (measured on your tasks) | scored 0.90 of 1 across 3 tasks (graded) | yes (estimate — not observed) | yes (estimate — not observed) | passed 2 of 3 tasks |
| OpenAI: GPT-6.1 Terra | $0.0181 a day (measured on your tasks) | scored 0.93 of 1 across 3 tasks (graded) | yes (estimate — not observed) | yes (estimate — not observed) | passed 3 of 3 tasks |
| Google: Gemini 4 Argon | $0.0296 a day (measured on your tasks) | scored 0.93 of 1 across 3 tasks (graded) | yes (estimate — not observed) | yes (estimate — not observed) | passed 3 of 3 tasks |
| MoonshotAI: Kimi K3.1 Pro | $0.0899 a day (measured on your tasks) | scored 0.93 of 1 across 3 tasks (graded) | yes (estimate — not observed) | yes (estimate — not observed) | passed 3 of 3 tasks |
| OpenAI: GPT-6.1 Terra Pro | $0.0906 a day (measured on your tasks) | scored 0.93 of 1 across 3 tasks (graded) | yes (estimate — not observed) | yes (estimate — not observed) | passed 3 of 3 tasks |
| Anthropic: Claude Opus 5.5 | $0.1893 a day (measured on your tasks) | scored 1.00 of 1 across 3 tasks (graded) | yes (estimate — not observed) | yes (estimate — not observed) | passed 3 of 3 tasks |
| Anthropic: Claude Sonnet 5.5 | $0.0772 a day (measured on your tasks) | scored 1.00 of 1 across 3 tasks (graded) | yes (estimate — not observed) | yes (estimate — not observed) | passed 3 of 3 tasks |
8 models tested on research from sources; cost of this test $0.0768.
Your app does something like this? XF Model Watch keeps it on the best model. XF Model Watch re-tests your own job as new models arrive and emails you only when one passes every check. Watch this job