Quick answer
To compare AI models on token cost, run the same real task on each model at the same settings, record the tokens used and the price per million tokens, and work out the cost per usable result, including any retries and fixes. A model with a cheaper price list can still cost more if it uses twice as many tokens or needs three attempts. WorldofAI’s early tests suggest the Qwen 4 preview used roughly half the tokens of DeepSeek V4.1 Flash on one web development task, with better output. That is the kind of difference a simple comparison sheet will show you for your own work.
Key takeaways
- Price per token is only half the story. Tokens used per task matters just as much.
- Cost per usable result is the number to compare.
- Include retries. A cheap model that needs three goes is not cheap.
- Match settings, including reasoning effort, or the comparison is unfair.
- Use real tasks, not viral demos.
- Recheck at launch. Preview results and final pricing can change.
📘 Main article: This guide supports Qwen 4 Preview: First Look at Qwen 4 Max, 27B and Early Coding Demos. It shows you how to test claims like “Qwen 4 is more token efficient” on your own work.
Every new model launch comes with claims about speed and price. With the Qwen 4 preview, the headline from early testers is token efficiency. In WorldofAI’s Qwen 4 first look, the presenter shows a web development test where the preview used about half as many tokens as DeepSeek V4.1 Flash and produced a better result.
This guide answers one question: how do you compare AI models on what they really cost per task, not just on how good the output looks? For the wider Qwen 4 news, see our Qwen 4 preview first look.
Why does token cost matter as much as quality?
Because your bill is the price per token multiplied by the number of tokens a model uses, and models differ hugely in how many they use. Tokens are the small chunks of text that AI providers count for input and output.
Two things push token use up:
- Reasoning. Models that “think” before answering produce hidden reasoning tokens, which are usually charged as output. Our guide on choosing an AI reasoning effort level explains how this dial works.
- Verbosity. Some models write longer code, add more comments or repeat themselves.
So a model that costs less per million tokens can still cost more per job. Quality matters too, because a weaker answer you must redo doubles the cost.
What should you measure when you compare models?
| Measure | Why it matters |
|---|---|
| Input tokens | What you send, including files and instructions. |
| Output tokens (including reasoning) | Usually the bigger cost, especially at high effort levels. |
| Price per million tokens | From the provider’s pricing page, for input and output separately. |
| Attempts needed | How many runs before you had something usable. |
| Fix-up time | Minutes you spent correcting the result. |
| Wait time | How long each run took. Slow models cost you time. |
💡 Tip: Most API dashboards and many coding tools, such as Qwen Code or Claude Code, show token counts per request. Check the usage page after each test rather than guessing.
How do you run a fair token cost comparison?
- Choose two or three real tasks you do often, such as a landing page, a data clean-up script and a product description.
- Write one fixed prompt for each task and save it, so every model gets exactly the same instructions.
- Match the settings. Use the same reasoning effort level, the same tools and the same files for every model.
- Run each task on each model, and record input tokens, output tokens, time taken and number of attempts.
- Look up the price per million input and output tokens for each model on the provider’s pricing page.
- Work out the cost per run: (input tokens × input price + output tokens × output price) ÷ 1,000,000.
- Multiply by attempts to get the cost per usable result.
- Score the quality from 1 to 5, and note the minutes spent fixing it.
- Decide per task type, not overall. One model may win on code and another on writing.
Want the full Qwen 4 picture? Read Qwen 4 Preview: First Look at Qwen 4 Max, 27B and Early Coding Demos, and see how to choose an AI reasoning effort level before you test.
What does a simple comparison sheet look like?
Keep it to one row per model per task. Here is the layout, with columns you fill in from your own tests:
| Column | What to write |
|---|---|
| Task | For example, “landing page for a new offer” |
| Model and effort level | For example, “Model A, high” |
| Input / output tokens | From the usage page |
| Cost per run | Using the formula above |
| Attempts | How many runs to get a usable result |
| Cost per usable result | Cost per run × attempts |
| Quality score and fix time | 1 to 5, plus minutes |
We have deliberately not filled in figures. Pricing changes often, and Qwen 4 pricing has not been officially published. A leaked partner image suggested a launch promotion with free input and output tokens for Qwen 4 Max, but it was deleted and remains unconfirmed.
🔎 Unconfirmed / rumour: The “half the tokens” comparison with DeepSeek V4.1 Flash comes from one early test shown by WorldofAI. It may not hold across other tasks, settings or the final release. The preview API also routed requests across several Qwen 4 sizes, so results may mix models.
What mistakes should you avoid?
- Comparing price lists only. The cheapest rate per token can still be the most expensive per job.
- Forgetting reasoning tokens. They are often billed as output and can be most of the cost.
- Using different prompts or settings for each model.
- Counting one lucky run. Repeat each test at least twice.
- Ignoring your time. Ten minutes of fixing can outweigh a few pence of tokens.
- Treating free promotions as permanent. Plan for the normal price. Our look at whether AI agents are worth the cost covers budgeting beyond the trial.
⚠️ Warning: If you test with real business files, check each provider’s data terms first. For unreleased or stealth models, use dummy data only, as explained in how to use a free stealth AI model without risking your data.
Frequently asked questions
What is token efficiency?
Token efficiency is how many tokens a model needs to complete a task well. A more efficient model uses fewer tokens for the same quality, which lowers cost and often speeds things up.
Is Qwen 4 more token efficient than DeepSeek?
In one early WorldofAI test, the Qwen 4 preview used about half the tokens of DeepSeek V4.1 Flash with better output. It is a single pre-release test, so check it on your own tasks after launch.
How do I find out how many tokens a request used?
Check the usage or billing page in your API dashboard, or the token counter in tools such as Qwen Code or Claude Code.
Do reasoning tokens count towards the bill?
Usually, yes. Most providers charge hidden reasoning tokens as output tokens, so higher effort levels cost more.
How often should I rerun the comparison?
Whenever a model you use is updated or its pricing changes, and at least every few months.
The bottom line
Token cost is where real AI savings hide. Compare models on cost per usable result, using the same real tasks and settings, and include retries and fix-up time. Claims like Qwen 4 using half the tokens of DeepSeek are worth testing, not taking on trust. Your first step: pick two weekly tasks, save the prompts, and build the comparison sheet before the next big model launch.
Related posts
- Qwen 4 Preview: First Look at Qwen 4 Max, 27B and Early Coding Demos
- Low, High or Max: How to Choose an AI Reasoning Effort Level
- DeepSeek V4.1 Flash 1017 Tested: How a Cheap Open Model Stacks Up

[…] Measure cost per finished job. When cheaper models appear, you will only notice the saving if you track it. See how to compare AI models on token cost. […]
[…] a wrapper around a well-known model, compare its price with using that model directly. Our guide on comparing AI models on token cost shows how to compare like for […]