Quick answer
To test whether a cheaper AI model is good enough, give it the same five to ten real examples you would give your current model, judge every answer against a short written standard, and add up the full cost, including the time you spend checking and fixing. If the cheaper model passes your standard at a lower total cost, switch that task. If it fails, you have evidence that the premium model is worth paying for. The whole test takes about an hour and needs nothing more than a spreadsheet.
Key takeaways
- Use your own work, not public benchmarks. Real examples, including awkward ones, tell you far more.
- Decide what “good” means first, before you see any answers, so you do not move the goalposts.
- Count review time. A cheap answer that needs 20 minutes of fixing is not cheap.
- Re-run the same test whenever a new model or price change arrives.
This guide supports our main article, Do You Need the Most Expensive AI Model? Why Cheaper Is Often Enough, which explains why many businesses are moving everyday AI work to cheaper models. Here we show you exactly how to run the kind of simple test, or “eval”, that makes that decision safe.
📘 Main article: New to the idea of task saturation and why cheaper models are often enough? Read Do You Need the Most Expensive AI Model? Why Cheaper Is Often Enough first, then come back here to test your own tasks.
In this guide
- What do you need before you start?
- Step 1: Pick one task worth testing
- Step 2: Collect five to ten real examples
- Step 3: Write down what “good enough” means
- Step 4: Run both models on the same examples
- Step 5: Score the results and count the full cost
- Step 6: Decide, then re-test later
- What does a finished test look like?
- What mistakes should you avoid?
- Frequently asked questions
- Related posts
What do you need before you start?
- Access to two models: the one you use now and a cheaper one (for example a “mini”, “flash” or mid-tier model from the same company).
- A spreadsheet or a simple table in a document.
- Five to ten real examples of the task, with any private customer details removed.
- About an hour.
💡 Tip: Many AI apps let you switch models in a drop-down menu. Testing within the same supplier is the easiest start: no new account, same settings, just a different model.
Step 1: Pick one task worth testing
Choose a task you do often and that is easy to check. Good candidates:
- replying to common customer emails,
- summarising meeting notes or calls,
- writing product or service descriptions,
- turning rough notes into a social post.
Avoid starting with rare, high-stakes work such as legal or financial documents. Those are exactly the jobs where a premium model is more likely to be worth it.
Step 2: Collect five to ten real examples
Pull examples from your actual work. Aim for a mix:
- Three or four typical cases: the everyday version of the task.
- Two or three awkward cases: long, vague, unusual or emotionally tricky ones.
- One “trap” case: something where a careless answer would cause a problem, such as a refund request outside your policy.
Nate B Jones stresses this point in the video: tests should reflect real assignments, including the difficult edge cases, not just easy wins.
⚠️ Warning: Remove names, emails, order numbers and anything personal before pasting examples into an AI tool, especially one you have not used before.
Step 3: Write down what “good enough” means
Before running anything, write three to five pass/fail checks. Keep them plain:
- Is every fact correct (prices, dates, policy)?
- Does it follow our tone (friendly, British spelling, no hype)?
- Does it answer the actual question?
- Would I send it after no more than a light edit?
Writing this first matters. If you decide what “good” looks like after seeing a polished answer, you will tend to favour whichever model you already like.
Step 4: Run both models on the same examples
Use the same prompt, the same instructions and the same examples for both models. Copy each answer into your spreadsheet, one row per example and one column per model.
If you can, hide which answer came from which model before you judge them. A colleague can shuffle the columns, or you can label them A and B.
Step 5: Score the results and count the full cost
Now mark each answer pass or fail against your checks, and record how many minutes you spent fixing it. Then work out the full cost per task:
| Cost item | How to estimate it |
|---|---|
| Model cost | Token price or a share of your monthly subscription |
| Retries | How many attempts before you got a usable answer |
| Review and fixing time | Minutes spent × your hourly value |
| Risk | Any answer that could have caused a real problem counts as a fail |
You do not need exact figures. A rough estimate is enough to see which model wins. For the token side, our guide to comparing AI models on token cost explains the sums.
Step 6: Decide, then re-test later
Use a simple rule:
- Cheaper model passes everything and needs no more fixing: switch this task.
- Cheaper model passes most cases but fails the awkward ones: use it by default and send hard cases to the premium model.
- Cheaper model fails typical cases or needs lots of fixing: stay with the premium model for now.
Save your examples and checks. When a new model or price change arrives, re-run the same test in minutes. That is what turns a one-off experiment into a habit.
What does a finished test look like?
Here is an illustrative result for a small shop testing customer email replies (example figures only, not real data):
| Example | Premium model | Cheaper model | Fixing time (cheaper) |
|---|---|---|---|
| Where is my order? | Pass | Pass | 0 min |
| Change delivery address | Pass | Pass | 1 min |
| Product size question | Pass | Pass | 0 min |
| Angry, long complaint | Pass | Fail (tone too flat) | 6 min |
| Refund outside policy (trap) | Pass | Pass | 2 min |
| Wholesale enquiry | Pass | Pass | 1 min |
Verdict: the cheaper model handles the routine replies well. The complaint is the hard case, so that type goes to the premium model or gets a human reply.
Your next step: pick one weekly task and collect six real examples today. Then read the main article, Do You Need the Most Expensive AI Model? Why Cheaper Is Often Enough, for the wider picture on when premium models are worth it.
What mistakes should you avoid?
- Testing with made-up examples. They are usually too tidy. Use real work.
- Only counting the price. Review time often matters more than token cost.
- Changing the prompt between models. Keep everything the same except the model.
- Testing once and forgetting. Models and prices change monthly. Re-test.
- Ignoring made-up facts. Check for invented details; our guide to testing an AI model for made-up answers helps.
Frequently asked questions
How many examples do I need to test an AI model?
Five to ten real examples is enough for a small business decision, as long as you include a few awkward cases. Add more for high-volume or high-risk tasks.
What is an eval in AI?
An eval is a test of the work you need done: real examples plus a written standard for a good answer. You run models through it and compare quality and cost.
Can I test models without coding?
Yes. Copy the same prompt and examples into each model in the app you already use and record the results in a spreadsheet.
How often should I re-test?
Whenever a new model, price change or plan renewal comes up, or every few months as a habit.
What if the cheaper model is only slightly worse?
Check whether the difference matters to your customer. If a light edit fixes it in a minute, the cheaper model may still win on total cost.
Related posts
- Do You Need the Most Expensive AI Model? Why Cheaper Is Often Enough
- How to Test an AI Model for Made-Up Answers Before You Rely on It
- Which Claude Model Should You Use? Haiku, Sonnet, Opus or Fable
