Tested 2026-10-26
Planning multi-step tool use: Z.ai: GLM 4.7 Flash won, 2 of 7 models failed a must-have; OpenRouter's own picker passed every must-have but cost 2× the winner (3rd of 5)
Plan the exact tool calls for a multi-step task as JSON, using only the tools given, in a working order — for a small agency automating client onboarding.
What this project actually needed
- takes written text as input — must have
- produces written text — must have
- can call tools — must have
- can return JSON — must have
- returns a JSON list of tool calls that uses only the given tools, in an order that works — must have: a valid, complete plan on every task
- is good at tool use — decides the order
- cost at your volume — decides the order
The winner
Z.ai: GLM 4.7 Flash passed every must-have and ranked first by how this job said to choose.
OpenRouter's automatic picker sent this job to z-ai/glm-5.3-flash and passed every must-have but cost 2× the winner (3rd of 5). It picks one message at a time and never sees this job's must-haves.
Who failed, in their own words
Z.ai: GLM 5V Turbo
- returns a JSON list of tool calls that uses only the given tools, in an order that works:
```json [ { "tool": "list_invoices", "args": { "client_id": 42, "status": "unpaid" } }, {
Mistral: Mistral Nemo
- facts match the task:
[insert_link]
Every model on every requirement
| Model | can call tools | can return JSON | cost at your volume | is good at tool use | produces written text | takes written text as input | returns a JSON list of tool calls that uses only the given tools, in an order that works |
|---|---|---|---|---|---|---|---|
| Auto Router | not applicable — the picker chooses a model per request | not applicable — the picker chooses a model per request | $0.2902 a day (measured on your tasks) | scored 1.00 of 1 across 2 tasks (graded) | not applicable — the picker chooses a model per request | not applicable — the picker chooses a model per request | passed 2 of 2 tasks |
| Z.ai: GLM 5.2 | yes (estimate — not observed) | yes (estimate — not observed) | $0.4369 a day (measured on your tasks) | scored 0.97 of 1 across 3 tasks (graded) | yes (estimate — not observed) | yes (estimate — not observed) | passed 3 of 3 tasks |
| Z.ai: GLM 5 Turbo | yes (estimate — not observed) | yes (estimate — not observed) | $0.8573 a day (measured on your tasks) | scored 0.90 of 1 across 3 tasks (graded) | yes (estimate — not observed) | yes (estimate — not observed) | passed 3 of 3 tasks |
| Z.ai: GLM 5V Turbo | yes (estimate — not observed) | yes (estimate — not observed) | $0.9596 a day (measured on your tasks) | scored 0.63 of 1 across 3 tasks (graded) | yes (estimate — not observed) | yes (estimate — not observed) | passed 2 of 3 tasks |
| Z.ai: GLM 4.7 Flash | yes (estimate — not observed) | yes (estimate — not observed) | $0.1179 a day (measured on your tasks) | scored 1.00 of 1 across 2 tasks (graded) | yes (estimate — not observed) | yes (estimate — not observed) | passed 2 of 2 tasks |
| Mistral: Mistral Nemo | yes (estimate — not observed) | yes (estimate — not observed) | $0.0013 a day (measured on your tasks) | scored 0.73 of 1 across 3 tasks (graded) | yes (estimate — not observed) | yes (estimate — not observed) | passed 3 of 3 tasks |
| StepFun: Step 3.7 Flash | yes (estimate — not observed) | yes (estimate — not observed) | $0.2868 a day (measured on your tasks) | scored 1.00 of 1 across 2 tasks (graded) | yes (estimate — not observed) | yes (estimate — not observed) | passed 2 of 2 tasks |
7 models tested on agent tool plan; cost of this test $0.0691.
Your app does something like this? XF Model Watch keeps it on the best model. XF Model Watch re-tests your own job as new models arrive and emails you only when one passes every check. Watch this job