Ink and watercolour balance scale with a small sprocket and a large gem level, surrounded by tick and cross test cards, showing a cheaper AI model passing tests

Quick answer

To test whether a cheaper AI model is good enough, give it the same five to ten real examples you would give your current model, judge every answer against a short written standard, and add up the full cost, including the time you spend checking and fixing. If the cheaper model passes your standard at a lower total cost, switch that task. If it fails, you have evidence that the premium model is worth paying for. The whole test takes about an hour and needs nothing more than a spreadsheet.

Key takeaways

  • Use your own work, not public benchmarks. Real examples, including awkward ones, tell you far more.
  • Decide what “good” means first, before you see any answers, so you do not move the goalposts.
  • Count review time. A cheap answer that needs 20 minutes of fixing is not cheap.
  • Re-run the same test whenever a new model or price change arrives.

This guide supports our main article, Do You Need the Most Expensive AI Model? Why Cheaper Is Often Enough, which explains why many businesses are moving everyday AI work to cheaper models. Here we show you exactly how to run the kind of simple test, or “eval”, that makes that decision safe.

📘 Main article: New to the idea of task saturation and why cheaper models are often enough? Read Do You Need the Most Expensive AI Model? Why Cheaper Is Often Enough first, then come back here to test your own tasks.

What do you need before you start?

  • Access to two models: the one you use now and a cheaper one (for example a “mini”, “flash” or mid-tier model from the same company).
  • A spreadsheet or a simple table in a document.
  • Five to ten real examples of the task, with any private customer details removed.
  • About an hour.

💡 Tip: Many AI apps let you switch models in a drop-down menu. Testing within the same supplier is the easiest start: no new account, same settings, just a different model.

Step 1: Pick one task worth testing

Choose a task you do often and that is easy to check. Good candidates:

  • replying to common customer emails,
  • summarising meeting notes or calls,
  • writing product or service descriptions,
  • turning rough notes into a social post.

Avoid starting with rare, high-stakes work such as legal or financial documents. Those are exactly the jobs where a premium model is more likely to be worth it.

Step 2: Collect five to ten real examples

Pull examples from your actual work. Aim for a mix:

  • Three or four typical cases: the everyday version of the task.
  • Two or three awkward cases: long, vague, unusual or emotionally tricky ones.
  • One “trap” case: something where a careless answer would cause a problem, such as a refund request outside your policy.

Nate B Jones stresses this point in the video: tests should reflect real assignments, including the difficult edge cases, not just easy wins.

⚠️ Warning: Remove names, emails, order numbers and anything personal before pasting examples into an AI tool, especially one you have not used before.

Step 3: Write down what “good enough” means

Before running anything, write three to five pass/fail checks. Keep them plain:

  • Is every fact correct (prices, dates, policy)?
  • Does it follow our tone (friendly, British spelling, no hype)?
  • Does it answer the actual question?
  • Would I send it after no more than a light edit?

Writing this first matters. If you decide what “good” looks like after seeing a polished answer, you will tend to favour whichever model you already like.

Step 4: Run both models on the same examples

Use the same prompt, the same instructions and the same examples for both models. Copy each answer into your spreadsheet, one row per example and one column per model.

If you can, hide which answer came from which model before you judge them. A colleague can shuffle the columns, or you can label them A and B.

Step 5: Score the results and count the full cost

Now mark each answer pass or fail against your checks, and record how many minutes you spent fixing it. Then work out the full cost per task:

Cost itemHow to estimate it
Model costToken price or a share of your monthly subscription
RetriesHow many attempts before you got a usable answer
Review and fixing timeMinutes spent × your hourly value
RiskAny answer that could have caused a real problem counts as a fail

You do not need exact figures. A rough estimate is enough to see which model wins. For the token side, our guide to comparing AI models on token cost explains the sums.

Step 6: Decide, then re-test later

Use a simple rule:

  • Cheaper model passes everything and needs no more fixing: switch this task.
  • Cheaper model passes most cases but fails the awkward ones: use it by default and send hard cases to the premium model.
  • Cheaper model fails typical cases or needs lots of fixing: stay with the premium model for now.

Save your examples and checks. When a new model or price change arrives, re-run the same test in minutes. That is what turns a one-off experiment into a habit.

What does a finished test look like?

Here is an illustrative result for a small shop testing customer email replies (example figures only, not real data):

ExamplePremium modelCheaper modelFixing time (cheaper)
Where is my order?PassPass0 min
Change delivery addressPassPass1 min
Product size questionPassPass0 min
Angry, long complaintPassFail (tone too flat)6 min
Refund outside policy (trap)PassPass2 min
Wholesale enquiryPassPass1 min

Verdict: the cheaper model handles the routine replies well. The complaint is the hard case, so that type goes to the premium model or gets a human reply.

What mistakes should you avoid?

  • Testing with made-up examples. They are usually too tidy. Use real work.
  • Only counting the price. Review time often matters more than token cost.
  • Changing the prompt between models. Keep everything the same except the model.
  • Testing once and forgetting. Models and prices change monthly. Re-test.
  • Ignoring made-up facts. Check for invented details; our guide to testing an AI model for made-up answers helps.

Frequently asked questions

How many examples do I need to test an AI model?

Five to ten real examples is enough for a small business decision, as long as you include a few awkward cases. Add more for high-volume or high-risk tasks.

What is an eval in AI?

An eval is a test of the work you need done: real examples plus a written standard for a good answer. You run models through it and compare quality and cost.

Can I test models without coding?

Yes. Copy the same prompt and examples into each model in the app you already use and record the results in a spreadsheet.

How often should I re-test?

Whenever a new model, price change or plan renewal comes up, or every few months as a habit.

What if the cheaper model is only slightly worse?

Check whether the difference matters to your customer. If a light edit fixes it in a minute, the cheaper model may still win on total cost.

author avatar
Garry Knight
I'm Garry Knight, the person behind Prodify Digital. I write about email list building, email marketing, SEO, AI search and the tools that connect them. My aim is to make online marketing easier to understand, so creators and small business owners can make informed decisions about building an audience and keeping people engaged. Here you'll find straightforward guides and product reviews that explain what something does, where it fits and which limitations matter. The focus is on clear explanations and useful next steps—not hype, shortcuts or promises of easy earnings.

Leave a Reply