Quick answer
To test an AI chatbot before launch, write a test set of 20 to 30 real questions: common ones, awkward ones, follow-ups and questions it should refuse. Run them as multi-turn conversations in a playground, score each answer as correct, partly correct or wrong, and note why. Fix the cause, which is usually a missing or conflicting source or an unclear instruction, then retest the whole set. Launch in one place once the agent answers your common questions correctly and hands over the rest. After launch, read the logs every week and add any new failures to your test set.
Key takeaways
- Build a test set of 20 to 30 real questions, not made-up easy ones
- Include follow-ups, typos, off-topic and out-of-bounds questions
- Score each answer: correct, partly correct or wrong
- Fix the cause (source or instruction), not just the symptom
- Retest the full set after every change
- Go live in one place first, then read the logs weekly
- Every new failure joins the test set for next time
📘 Main article: This guide supports Scale.gg Chatbaser: An AI Chatbot That Learns Your Business First. It focuses on the “test” and “inspect” steps: checking an agent properly before customers see it.
In this guide
A chatbot that gets one in ten answers wrong does more harm than having no chatbot at all. Customers do not know which answers to trust, and your team ends up correcting it by email.
This guide answers one question: how do you test an AI chatbot before it talks to customers? It follows on from Scale.gg Chatbaser: An AI Chatbot That Learns Your Business First and the Chatbaser video on our channel, which shows an interactive playground for multi-turn tests plus logs, charts and exports. The routine below works with any grounded chatbot.
What does testing an AI chatbot actually involve?
It means checking accuracy, behaviour and hand-over against questions you already know the answers to. You are not testing whether the AI sounds clever. You are testing whether it says what your business would say.
| What you check | What good looks like |
|---|---|
| Accuracy | Facts match your approved sources exactly |
| Boundaries | It declines topics it should not answer |
| Hand-over | It points people to a human when it should |
| Tone | It sounds like your brand, not a generic assistant |
| Consistency | The same question gets the same answer each time |
Which questions should you test with?
Use real questions from real customers, plus a few designed to trip it up. Aim for 20 to 30 to start.
- The top ten: your most common questions, copied from your inbox in the customer’s own words
- Awkward wording: typos, slang, very short questions such as “refund?”
- Follow-ups: “What about the yearly plan?” after a pricing answer
- Edge cases: situations your policy covers but rarely comes up
- Out of bounds: questions it should refuse or hand over, such as legal advice or account changes
- Off topic: something unrelated, to check it stays on its job
💡 Tip: Write the expected answer next to each question before you test. Without it, you will judge answers by how confident they sound rather than whether they are right.
How do you run a multi-turn test conversation?
Talk to it the way a real customer would, with several messages in a row. Many chatbots answer the first message well and then lose track of what “it” or “that plan” refers to.
- Open with a vague question, such as “Is this right for me?”
- Add detail in the next message, as a customer would
- Ask a follow-up that depends on the previous answer
- Change your mind or switch topic mid-conversation
- End with a request it cannot fulfil, and check it hands over politely
In Chatbaser, the interactive playground shown in our video is where you run these conversations before the agent goes live. Record each result in a simple spreadsheet: question, expected answer, actual answer, score and notes.
What should you look for in the chatbot logs?
Look for patterns, not one-off oddities. Our video shows logs, charts and exports for checking how an agent is performing. Once you are live, review them weekly for:
- Questions it could not answer, which usually point to a missing source
- Repeated rephrasing by the same person, a sign the first answer did not help
- Confident wrong answers, the most important ones to fix
- Popular new topics you had not planned for
- Hand-over requests, to check people reach a human easily
⚠️ Warning: Logs contain what customers typed, which can include personal details. Limit who can see them, export only what you need and follow your privacy policy.
How do you fix a wrong answer?
Find the cause first, then change the smallest thing that fixes it. Our Chatbaser video shows the fix options: add or remove sources, revise the instructions, retrain and retest.
| Symptom | Likely fix |
|---|---|
| Says it does not know, but you have an answer | Add or clarify the source |
| Gives an old price or policy | Remove or update the outdated source |
| Mixes up two products or plans | Name each one in full in the source text |
| Answers something it should refuse | Tighten the boundaries in the instructions |
| Right facts, wrong tone | Adjust the personality and style settings |
For source problems, our sibling guide on how to prepare knowledge sources so your AI chatbot answers correctly goes into detail. After any fix, rerun the whole test set, because one change can affect other answers.
When is a chatbot ready to go live?
When it answers your common questions correctly and hands over everything else. There is no universal pass mark, so set your own. A sensible starting checklist:
- Every one of your top ten questions is answered correctly
- It refuses or hands over every out-of-bounds question
- No confident wrong answers in the latest full test run
- A clear human contact route appears when needed
- You have a weekly log review booked in your calendar
Launch in one place first, such as a single help page, before adding it across your whole site. If you want a structured way to decide when AI output needs a human check, see How to Set AI Confidence Thresholds for Human Review.
🔎 Unconfirmed / rumour: Our video does not show whether Chatbaser offers automatic accuracy scores, A/B testing of agent versions or live human takeover. Treat any such features as unconfirmed until you see them in the app.
Want the full picture? Read Scale.gg Chatbaser: An AI Chatbot That Learns Your Business First, and see our Starter Bonus Bundle for Scale.gg.
Frequently asked questions
How many test questions do I need?
Start with 20 to 30 real questions, then add every new failure you find in the logs.
How often should I retest my chatbot?
After every change to sources or instructions, and at least once a month even if nothing changed.
What is a multi-turn conversation?
A conversation with several back-and-forth messages, such as a question followed by a related follow-up. It tests whether the chatbot keeps track of context.
Can I test a Chatbaser agent before it goes live?
Yes. Our video shows an interactive playground for testing real multi-turn conversations. Read more in our Chatbaser article.
What is the biggest testing mistake?
Testing only easy questions you wrote yourself. Use real customer wording and include questions the chatbot should refuse.
The bottom line
Testing a chatbot is mostly about honesty with yourself: real questions, written expected answers and a clear score for each. Fix the cause of each failure, retest the full set, launch in one place and keep reading the logs. Each new failure becomes a test, so the agent gets more reliable the longer you run it.
Related posts
- Scale.gg Chatbaser: An AI Chatbot That Learns Your Business First
- How to Prepare Knowledge Sources So Your AI Chatbot Answers Correctly
- How to Set AI Confidence Thresholds for Human Review
[…] Retest after every change, using the routine in how to test an AI chatbot before it talks to customers […]