Build an AI Test Set Before You Change a Business Workflow

A cleaner AI answer does not always mean a better business workflow. A new prompt may sound more confident while dropping a qualification. A different model may route routine enquiries correctly but guess when a request falls outside your policy. Even replacing a source document can change what an assistant treats as true.

Before making those changes, keep a small AI test set: a saved collection of inputs, evidence and expected behaviours that you can rerun consistently. It helps a small team spot regressions—things that worked before but stop working after a change—and make a reviewable decision about whether to proceed.

This guide extends using AI in a small business without losing control. The aim is a portable testing habit for one bounded task. You can begin with a document or spreadsheet; a new evaluation platform is not required.

Choose one job that has an observable result

“Test our AI” is too broad. Define the particular job, its inputs, its allowed sources and the person who accepts the result. An enquiry-routing assistant might suggest a category and explain why. It should not send a reply, promise a price or change a customer record during this test.

Write the job as a sentence: “Given an enquiry and the approved service guide, suggest one allowed route, cite the relevant rule and flag cases that need a person.” That gives the reviewer something concrete to check. It also makes a refusal or request for clarification a valid outcome when the evidence is missing.

Keep a separate test for a different job. Summarising a meeting, classifying an enquiry and drafting a public article require different evidence and failure rules. A strong score on one does not establish suitability for the others.

Build test cases from the work you actually expect

Start with enough cases to cover the important behaviours. There is no universal number that proves reliability. A handful can expose an obvious problem, while a consequential or diverse workflow needs a broader evaluation and appropriate expertise.

Use authorised examples with unnecessary personal or confidential information removed. Synthetic examples can fill gaps, but a reviewer should check that they resemble the situations the team will face. Do not upload private customer records simply to make the test feel realistic.

• Ordinary cases: clear requests that belong to the main categories.
• Edge cases: unusual wording, combined requests or a long input with the key detail near the end.
• Missing evidence: a request for information absent from the approved sources.
• Conflicting evidence: two supplied documents disagree or have different effective dates.
• Boundary cases: a request outside the allowed task, or text that tries to override its instructions.
• Known failures: a previously observed mistake that should stay fixed.

The useful unit is a test card, not just a prompt. Save the source material and the expectation alongside it so another person can score the same case.

Case ID: [stable identifier]
Job and situation: [what this case checks]
Input: [approved, sanitised text]
Source version: [file/date or snapshot]
Expected behaviour: [answer, route, clarification or stop]
Must include: [critical facts or evidence]
Must not do: [guess, disclose, act or exceed scope]
Reviewer: [named owner]
Reason this case belongs: [normal, edge, missing, conflict or past failure]

Define the acceptance rules before looking at the new answers

A polished answer can influence judgement. Write the rules first, while the team is deciding what the task needs rather than defending a favourite tool or prompt.

Separate hard failures from quality differences. For enquiry routing, inventing a service, making an unsupported commitment or ignoring a required human handover could be a hard failure. A slightly awkward sentence may be a quality issue. An average score should never hide a critical boundary failure.

A simple rubric can score three dimensions from 0 to 2: factual grounding, task completion and usefulness to the reviewer. Zero means unacceptable, one means correction is needed and two means acceptable for the stated task. Define those meanings for each dimension, with examples, rather than treating the numbers as self-explanatory.

Choose the release rule with the accountable owner. For instance, a low-risk internal pilot might require no critical failures in the selected set and no deterioration on important cases, followed by supervised use. That is an illustrative local rule, not a recognised safety standard or proof that future outputs will be correct.

Freeze the baseline and record what changed

Run the saved cases through the current workflow and preserve its outputs. Record the prompt version, model identifier where exposed, source versions, tool permissions, relevant settings and test date. If a consumer app hides part of that configuration, record the limitation rather than inventing a model identifier.

Then prepare the proposed version. Change one main component at a time when practical. If you change the prompt, model and source library together, the comparison may show a problem but give little evidence about its cause.

Use the same inputs, approved evidence and scoring rules for both versions. Start each case with the intended context conditions; an old conversation full of unrelated material can make a comparison misleading. Preserve all attempted outputs, including rejected ones, rather than saving only the strongest answer.

Compare behaviour, not just wording

Place the baseline and proposed output beside the case card. Ask whether both satisfy the same reader or business need. Wording can change harmlessly; an omitted condition, invented fact or different action boundary can change the result materially.

Generative systems can vary on repeated runs. Repeat cases that are unstable or important, and record how many attempts passed each rule. One successful demonstration does not erase a failed attempt. Use more systematic sampling and specialist review as the potential consequence grows.

When possible, let a reviewer score the outputs without knowing which version produced them. A second reviewer can examine disputed or critical cases. If an AI tool helps score answers, compare its judgements with human decisions and keep the original evidence available. The tool should not be the sole judge of its own correctness.

OpenAI’s evaluation guidance supports task-specific criteria, representative cases, human feedback and evaluation after changes. NIST’s Measure guidance also emphasises documented test sets, methods and ongoing monitoring. Neither makes a small test set a guarantee of safe deployment.

A worked example: changing an enquiry-routing prompt

Imagine a consultancy uses AI to suggest where incoming enquiries belong. Its approved routes are training, implementation support and human review. Staff read each suggestion before taking action.

The owner prepares twelve sanitised cases: six clear enquiries, two mixed requests, one request for an unavailable service, one missing-price question, one conflict between source versions and one attempt to make the assistant ignore the routing rules. The card for the missing-price question says that the assistant must not invent a price and must send the case to human review.

In this hypothetical comparison, the new prompt improves the explanations on several ordinary enquiries. However, it produces a plausible price on the missing-price case. The team rejects the change because that is a critical failure, even if its average readability score is higher.

The owner repairs the prompt, preserves the failed output and reruns the affected case plus the full saved set. If the revision meets the agreed rules, staff can trial it under the existing review boundary. The price failure remains in the set so a later change cannot quietly reintroduce it.

The example demonstrates a decision process, not a claim about any model’s accuracy. A real team should choose cases and acceptance rules from its own work, risks and evidence.

Keep a short comparison record

Record what changed, which set was used and why the decision follows from the evidence. This can sit beside your AI decision log, with links to the actual case cards and outputs.

Change ID and date: [identifier]
Baseline / proposed version: [configuration references]
Test-set version: [identifier]
Critical failures: [case IDs and evidence]
Quality differences: [case-level notes]
Repeated-run variation: [attempts and outcomes]
Reviewer effort / cost: [observed, if measured]
Decision: [reject, revise, supervised pilot or approve within scope]
Owner and conditions: [person, permitted use, stop rule]
Rollback: [known usable version]
Next review trigger: [change, incident or agreed date]

Keep speed and cost separate from correctness. The AI-assisted work measurement guide explains how to include review and rework without turning limited observations into an invented ROI claim. A faster output that needs more correction may not be an improvement.

Two prompts that support the review

Use the first prompt to propose test cards from a sanitised task description. A knowledgeable owner still needs to approve the cases, expected behaviour and allowed data.

Using only the supplied task description and approved sources, propose test cards for normal, edge, missing-evidence, conflicting-evidence and out-of-scope situations. Give each card an input, expected behaviour, must-include facts and must-not-do actions. Do not invent policies, prices, capabilities or private customer data. Label synthetic cases. Flag expectations that need a human decision.

The second prompt helps organise a comparison. Supply the agreed rubric and evidence; do not ask the evaluator to make up the standard after seeing the answers.

Compare output A and output B against the supplied case card and rubric. For each rule, show pass, fail or uncertain and quote the relevant evidence briefly. Separate critical failures from quality differences. Do not reward confidence or style when evidence is absent. Identify disagreements that need human review. Do not make the release decision.

Maintain the set without teaching to the test

Keep the baseline cases stable enough to compare changes. Add new failure patterns, changed policies and newly relevant tasks with an explanation. Version the set so a result from last month is not compared casually with a different set today.

Reserve some cases for checking the final version rather than using every case while tuning the prompt. Otherwise, a workflow can become good at familiar examples while still failing on fresh work. Include periodic samples from authorised real use and review what the set misses.

Rerun when the model, prompt, retrieval sources, tool permissions or business policy changes. Continue appropriate monitoring between changes. Keep the human approval checkpoints in place; passing a test does not grant a workflow permission to take new actions.

For consequential decisions involving people, money, legal obligations or sensitive data, this lightweight method is only a starting point. Use appropriate professional review and a more complete risk assessment. A small set cannot establish compliance or represent every affected person and situation.

Add an action-boundary case when the workflow can do more than prepare a draft. The business permission review following Anthropic's 9 October report shows how fictional follow-up records can cover normal work, missing input and a blocked delivery action in an isolated environment you control.

Before deciding what belongs in this test set, use an AI workflow dependency map to trace the inputs, permissions, review points and downstream actions that need coverage.

Reader Q&A

What is an AI test set?

It is a saved collection of inputs, approved source material and expected behaviours used to check a particular AI-assisted task consistently. It lets reviewers compare versions against the same rules.

How many cases should a small business use?

There is no universal minimum that proves reliability. Cover the important normal, edge, missing-evidence, conflicting-evidence and boundary cases, then expand the set as diversity and potential consequences increase.

Do I need coding or a paid evaluation platform?

No. A document or spreadsheet can hold the cases, outputs, rubric and decisions for a small manual review. Automation may help at scale, but it does not replace sound cases and accountable judgement.

Should every answer match the baseline word for word?

No. Compare factual grounding, required content, task completion and boundaries. Different wording can be acceptable if it preserves the intended meaning and satisfies the agreed rules.

Can AI grade its own outputs?

AI can help organise a comparison against a supplied rubric, but its judgements need human calibration and review. Keep the source evidence and do not make it the sole judge of correctness or release approval.

When should we rerun the test set?

Rerun after meaningful changes to the model, prompt, sources, tools, permissions or business policy, and when a new failure appears. Continue proportionate monitoring during normal use.

Start with one change you can reverse

Choose a bounded task, write a few strong test cards and agree the critical failures before changing anything. Save the current outputs, compare the proposed version and record the decision. That gives your team an evidence trail it can reuse when the next change arrives.

Sources were checked on 9 October 2026. The process above is platform-independent. OpenAI’s current guidance notes that its Evals platform is being deprecated; verify the current product lifecycle before choosing evaluation software. You do not need that platform to use this manual method.

Garry Knight

I'm Garry Knight, the person behind Prodify Digital. I write about email list building, email marketing, SEO, AI search and the tools that connect them. My aim is to make online marketing easier to understand, so creators and small business owners can make informed decisions about building an audience and keeping people engaged. Here you'll find straightforward guides and product reviews that explain what something does, where it fits and which limitations matter. The focus is on clear explanations and useful next steps—not hype, shortcuts or promises of easy earnings.

See Full Bio