Tested 2026-10-15
Pulling fields out of invoice text into JSON: inclusionAI: Ling 3.0 Flash won, 2 of 6 models failed a must-have; OpenRouter's own picker passed every must-have but cost 4× the winner (4th of 4)
Extract invoice number, date, total and vendor from invoice text as JSON — for an agency automating bookkeeping for clients.
What this project actually needed
- takes written text as input — must have
- produces written text — must have
- returns valid JSON every time — must have
- every field matches the invoice exactly — must have: no wrong or invented field in any answer
- cost at your volume — decides the order
The winner
inclusionAI: Ling 3.0 Flash passed every must-have and ranked first by how this job said to choose.
OpenRouter's automatic picker sent this job to deepseek/deepseek-v4-flash-0731 and passed every must-have but cost 4× the winner (4th of 4). It picks one message at a time and never sees this job's must-haves.
Who failed, in their own words
Meta: Llama 3.1 8B Instruct
- returns valid JSON every time:
Here is the JSON output with the requested keys: ``` { "invoice_number": "INV-20481", "date": "03/09/2026", "tota
Mistral: Mistral Nemo
- returns valid JSON every time:
Here's the JSON with the requested keys: ```json { "invoice_number": "INV-20481", "date": "03/09/2026", "total":
Every model on every requirement
| Model | cost at your volume | produces written text | takes written text as input | returns valid JSON every time | every field matches the invoice exactly |
|---|---|---|---|---|---|
| Auto Router | $0.0898 a month (measured on your tasks) | not applicable — the picker chooses a model per request | not applicable — the picker chooses a model per request | passed 3 of 3 tasks | passed 3 of 3 tasks |
| OpenAI: gpt-oss-20b | $0.0513 a month (measured on your tasks) | yes (estimate — not observed) | yes (estimate — not observed) | passed 3 of 3 tasks | passed 3 of 3 tasks |
| Mistral: Mistral Nemo | $0.0072 a month (measured on your tasks) | yes (estimate — not observed) | yes (estimate — not observed) | passed 2 of 3 tasks | passed 3 of 3 tasks |
| Meta: Llama 3.1 8B Instruct | $0.0115 a month (measured on your tasks) | yes (estimate — not observed) | yes (estimate — not observed) | passed 0 of 3 tasks | passed 3 of 3 tasks |
| inclusionAI: Ling 3.0 Flash | $0.0206 a month (measured on your tasks) | yes (estimate — not observed) | yes (estimate — not observed) | passed 3 of 3 tasks | passed 3 of 3 tasks |
| inclusionAI: Ling 3.0 Flash VL | $0.0540 a month (measured on your tasks) | yes (estimate — not observed) | yes (estimate — not observed) | passed 3 of 3 tasks | passed 3 of 3 tasks |
6 models tested on invoice field extraction; cost of this test $0.0349.
Your app does something like this? XF Model Watch keeps it on the best model. XF Model Watch re-tests your own job as new models arrive and emails you only when one passes every check. Watch this job