Tested 2026-10-21
Replying to customer reviews: Meta: Llama 3.1 8B Instruct won, 3 of 6 models failed a must-have; OpenRouter's own picker passed every must-have but cost 29× the winner (3rd of 3)
Write short, warm replies to customer reviews without promising anything not in the guidelines — for a local restaurant group.
What this project actually needed
- takes written text as input — must have
- produces written text — must have
- never offers refunds, discounts or freebies not in the guidelines — must have: no unapproved offers in any reply
- cost at your volume — decides the order
The winner
Meta: Llama 3.1 8B Instruct passed every must-have and ranked first by how this job said to choose.
OpenRouter's automatic picker sent this job to google/gemini-2.5-flash and passed every must-have but cost 29× the winner (3rd of 3). It picks one message at a time and never sees this job's must-haves.
Who failed, in their own words
ScaleDown: Extract
- gives an answer:
no answer came back after three tries (HTTP 400: b'{"error":{"message":"extract requires a response)
ScaleDown: Summarize
- never offers refunds, discounts or freebies not in the guidelines:
The review received a 3-star rating citing a nice view but overpriced drinks. The reply thanks the reviewer, declines discounts or refunds, and invites them to email manager@harborgrill.com.
Mistral: Mistral Nemo
- facts match the task:
We're sorry to hear that you're not satisfied with the service you received.
Every model on every requirement
| Model | cost at your volume | produces written text | takes written text as input | never offers refunds, discounts or freebies not in the guidelines |
|---|---|---|---|---|
| Auto Router | $0.0029 a day (measured on your tasks) | not applicable — the picker chooses a model per request | not applicable — the picker chooses a model per request | passed 3 of 3 tasks |
| ScaleDown: Extract | not measured | yes (estimate — not observed) | yes (estimate — not observed) | not judged — no graded answers |
| ScaleDown: Summarize | $0.0001 a day (measured on your tasks) | yes (estimate — not observed) | yes (estimate — not observed) | passed 2 of 3 tasks |
| Mistral: Mistral Nemo | $0.0001 a day (measured on your tasks) | yes (estimate — not observed) | yes (estimate — not observed) | passed 3 of 3 tasks |
| Meta: Llama 3.1 8B Instruct | $0.0001 a day (measured on your tasks) | yes (estimate — not observed) | yes (estimate — not observed) | passed 3 of 3 tasks |
| inclusionAI: Ling 3.0 Flash | $0.0002 a day (measured on your tasks) | yes (estimate — not observed) | yes (estimate — not observed) | passed 3 of 3 tasks |
6 models tested on review replies; cost of this test $0.0287.
Your app does something like this? XF Model Watch keeps it on the best model. XF Model Watch re-tests your own job as new models arrive and emails you only when one passes every check. Watch this job