Pixel-art robot inspecting a stack of documents with a magnifying glass while cracked speech bubbles crumble, illustrating testing an AI model for made-up answers

Quick answer

To test an AI model for made-up answers, give it 20 questions about your own documents where you already know the right answer, include a few questions it cannot answer from those documents, and score each reply as correct, honestly “not sure”, or made up. Divide the made-up answers by the total to get your own hallucination rate. Run the same test on any model you are considering, at the same settings, and repeat it whenever the model changes. It takes about an hour and tells you far more than a vendor’s benchmark.

Key takeaways

  • Benchmarks are not your work. A low published hallucination rate is a good sign, not a guarantee.
  • Use your own documents and write the answer key before you start.
  • Trap questions matter most. A good model says “that is not in the document” instead of inventing something.
  • Score three ways: correct, honest refusal, made up.
  • Keep the test and rerun it every time you switch model or settings.

This guide follows on from our round-up Gemini 4 Hydrogen: Google’s Fast New Flash Model, Anthropic’s “Slow Down” Pledge and This Week’s AI News. Google says its new Gemini 4 Hydrogen model makes more than five times fewer fabricated claims than the Flash model it replaces, and figures quoted in WorldofAI’s video put its hallucination rate at 8%, against much higher rates for rival models.

📘 Main article: Read the full round-up in Gemini 4 Hydrogen: Google’s Fast New Flash Model, Anthropic’s “Slow Down” Pledge and This Week’s AI News for Hydrogen’s benchmarks, pricing and the creator’s hands-on tests, plus the rest of the week’s AI news.

Those numbers are encouraging. But they come from someone else’s test, on someone else’s questions. This guide answers one question: how do you check whether a model makes things up on the work you actually give it?

Why test an AI model for made-up answers yourself?

Because a model’s accuracy depends heavily on the task, and published tests rarely match your task. A model that is excellent at summarising news articles might still invent a clause in your supplier contract or a figure in your sales report.

“Made-up answers” (often called hallucinations) are replies that sound confident but are not supported by the facts you gave the model. They are the most expensive kind of AI mistake, because they look right. A simple test of your own catches them before a customer or client does.

💡 Tip: You do not need any technical skills for this. A spreadsheet and an hour are enough.

What do you need?

  • Two or three real documents you often use with AI: a policy, a price list, a report, a contract or meeting notes. Remove anything confidential first, or use your company’s approved AI account.
  • A spreadsheet with columns for question, correct answer, model reply and score.
  • The models you want to compare, for example your current tool and Gemini 4 Hydrogen.
  • About an hour for the first run. Later runs take 20 minutes.

Step 1: Pick the job you actually rely on AI for

Start with the one job where a wrong answer would hurt. For most small businesses that is one of these:

  • answering customer questions from your policies or FAQs;
  • summarising reports or meeting notes;
  • pulling figures out of invoices, spreadsheets or PDFs;
  • researching facts for articles or proposals.

Test one job at a time. A model can pass on summaries and fail on figures.

Step 2: Build a 20-question test set

Write 20 questions whose answers are clearly in your documents, and write the correct answer next to each one before you ask any AI. Writing the answer key first stops you being swayed by a convincing reply.

Mix the question types:

TypeHow manyExample
Simple fact lookup6“What is our refund window for digital products?”
Numbers and dates5“What was the total in the March invoice from Supplier B?”
Summary or comparison4“What are the three main changes between the old and new policy?”
Detail buried deep in a long document5“What does section 7.3 say about late payment?”

Step 3: Add trap questions

Add five questions that cannot be answered from your documents. These are the most revealing part of the test, because they show whether the model admits it does not know.

  • Missing facts: ask about something the documents never mention, such as a product you do not sell.
  • False premises: build a wrong assumption into the question, such as “Why did we raise prices by 20% in May?” when you did not.
  • Plausible details: ask for a name, date or figure that would normally exist but is not in the text.

A good answer to every trap question is some version of “the documents do not say”. Anything else counts as made up.

Step 4: Run the test the same way every time

  1. Start a new chat for each model so earlier conversations do not leak in.
  2. Upload the same documents to each model.
  3. Use the same prompt (see below) and the same settings. If the model has thinking levels, note which one you used. For Gemini 4 Hydrogen, start with the default, high.
  4. Turn web search off if you are testing answers from your documents, so the model cannot pull in outside information.
  5. Paste each reply into your spreadsheet straight away.

Thinking levels can change accuracy as well as cost. Our guide to choosing an AI reasoning effort level explains the trade-off if you want to test more than one.

Step 5: Score every answer

Score each reply against your answer key using just three labels. Keep it simple so the scoring is consistent.

ScoreMeaningExample
✅ CorrectMatches your answer key; small wording differences are fineStates the right refund window
🟡 Honest refusalSays the answer is not in the documents, or asks for more information“The documents do not mention a May price rise.”
❌ Made upStates something not supported by the documents, including wrong numbers or invented reasonsExplains a price rise that never happened

If a reply is mostly right but includes one invented detail, score it as made up. In real use, that one detail is the part that causes trouble.

Step 6: Work out your made-up answer rate

Your made-up answer rate is the number of ❌ replies divided by 25 (your 20 questions plus 5 traps). For example, 3 made-up replies out of 25 is a 12% rate.

Also look at two smaller numbers:

  • Trap score: how many of the 5 trap questions got an honest refusal. This shows whether the model knows its limits.
  • Refusal rate on answerable questions: a model that says “not sure” to everything is safe but useless.
ModelCorrectHonest refusalsMade upMade-up rateTrap score
Current tool————— / 5
Gemini 4 Hydrogen————— / 5

Copy this table into your spreadsheet and fill it in. If cost also matters, our guide to comparing AI models on token cost adds a cost column.

What prompt should you use?

Use the same prompt for every model and every question:

“Answer using only the attached documents. If the answer is not in the documents, say ‘Not in the documents’ and do not guess. For each answer, quote the sentence or figure you relied on.”

Asking for a quote does two jobs. It makes made-up answers easier to spot, and it tends to reduce them, because the model has to point to evidence. If a model invents a quote, score it as made up and treat that as a serious warning sign.

What is a good result?

For work that customers or clients will see, aim for zero made-up answers on your test and at least 4 out of 5 on the trap questions. For internal drafts that a person always checks, a small number of errors may be acceptable, as long as you know they happen.

Whatever the score, the test tells you how to use the model:

  • No made-up answers, strong trap score: suitable for first drafts and lookups, with spot checks.
  • A few made-up answers: use it only where a person checks every answer.
  • Many made-up answers, or invented quotes: do not use it for this job.

⚠️ Warning: No test makes a model safe for legal, medical or financial decisions on its own. Keep a qualified person responsible for those, whatever the score.

What mistakes should you avoid?

  • Testing with questions the model has seen in public. Use your own private documents, not famous facts.
  • Writing the answer key afterwards. A confident reply can talk you into the wrong answer.
  • Changing settings between models. Compare like with like.
  • Testing once and forgetting. Models are updated often. Rerun your test whenever your provider changes the model.
  • Only checking the easy questions. Long documents and trap questions are where models slip.

If your documents include scanned receipts or forms, see our guide to testing an AI vision model on receipts, invoices and forms. For a broader accuracy check of a single document, read How to Use AI to Check Business Documents for Errors.

Frequently asked questions

What is an AI hallucination?

It is an answer that sounds confident but is not supported by facts, such as an invented figure, quote, policy or source.

How many questions do I need?

Twenty answerable questions plus five trap questions is enough to see clear differences between models. Ten plus three is a useful quick check.

Can I trust published hallucination rates?

Use them as a guide. They come from a specific test set, and your documents and questions may give very different results.

Does turning on web search reduce made-up answers?

It can help with general facts, but it can also bring in outside information that contradicts your documents. Turn it off when you want answers from your own files.

Do higher thinking levels reduce made-up answers?

Often, but not always, and they cost more. Test both levels on the same questions if accuracy really matters.

How often should I rerun the test?

Whenever you change model, settings or provider, and every few months anyway, because models are updated quietly.

The bottom line

Claims like “five times fewer fabricated claims” are worth having, but the only accuracy figure that matters is the one from your own work. Spend an hour building a 25-question test from your documents, score it honestly, and you will know which model you can trust, and for which jobs. Then keep the test and use it every time something changes.

author avatar
Garry Knight
I'm Garry Knight, the person behind Prodify Digital. I write about email list building, email marketing, SEO, AI search and the tools that connect them. My aim is to make online marketing easier to understand, so creators and small business owners can make informed decisions about building an audience and keeping people engaged. Here you'll find straightforward guides and product reviews that explain what something does, where it fits and which limitations matter. The focus is on clear explanations and useful next steps—not hype, shortcuts or promises of easy earnings.
2 thoughts on “How to Test an AI Model for Made-Up Answers Before You Rely on It”

Leave a Reply