AI Answer Quality: Build a Red-Team Set Before You Trust the Chatbot

A fluent chatbot answer can still be wrong. Before you put an assistant in front of staff, patients, learners or customers, give it questions whose correct handling you can actually judge.

For a small organisation, the first release decision is not “Which model scores highest?” It is “Is this system accurate enough for this defined use, with a safe response when it is not?” That answer needs evidence from your own help-desk tickets, policy documents, recurring enquiries or service scripts—not a supplier demonstration.

Start with one use and one risk statement

Write down the proposed use in a single sentence: “The assistant will answer staff questions about the approved leave policy,” or “The customer tool will explain the published booking process.” Then name what must not happen. An internal assistant must not invent an entitlement. A customer-facing tool must not turn an uncertain eligibility question into a promise. A document tool must not present an unsupported conclusion as if it came from the source.

This boundary keeps the test set focused. A system may be acceptable for finding a paragraph in a controlled document and unacceptable for interpreting a disputed case. Do not hide that difference inside one overall score.

Build 20 to 30 real, redacted questions

Choose questions that people actually ask. Remove names, account numbers, patient details, telephone numbers and other identifying information, but preserve the wording and context that make the question difficult. Keep the original case reference in a restricted internal register so a reviewer can confirm that the redacted version remains representative.

Do not make every question neat. A useful starting set contains five kinds of case:

• Ordinary: a direct question answered by the current approved source.
• Ambiguous: wording with two reasonable meanings, such as “How long is the notice?” without saying which process.
• Missing information: a question that cannot be answered until a date, product, location or account detail is supplied.
• Outdated policy: a question whose old answer conflicts with the current version of a policy, price list or service notice.
• Deliberately misleading: a prompt that asserts a false rule and asks the assistant to confirm it.

Add at least one case that must be refused or escalated. If every test invites an answer, you are not testing whether the system knows when not to answer.

Label every case before running the chatbot

Do not ask the AI to define its own marking scheme after it has answered. A process owner should prepare a small answer contract for each case:

• Expected answer: the fact, steps or safe non-answer the organisation expects.
• Acceptable variation: wording or ordering that is different but still correct and complete.
• Source location: the document name, version, page, section, URL or record that a reviewer can open.
• Escalation condition: the missing detail, risk or boundary that requires a named human role.
• Harm if wrong: what could result—a delayed service, wrong charge, privacy incident, unsafe advice, lost entitlement or reputational damage.

“Answer mentions the policy” is not an expected answer. Write enough detail for two reviewers to make the same judgement. For a return-policy question, specify the time window, exclusions and the action the customer should take. For a staff question, specify whether the assistant should answer, ask for a missing date or send the case to HR.

Use a small red-team table

The table below is a template. Replace the examples with your own redacted cases and the actual source locations. The examples are deliberately generic: do not treat them as your organisation’s policy.

Record five observations, not one flattering percentage

Run the same questions with the same source snapshot, prompt and relevant settings. For each response, record:

1. Answer correctness: correct, partly correct, wrong, or safely unanswered.
2. Citation quality: source present, source verifiable, source relevant and source version current.
3. Refusal behaviour: whether the assistant avoids guessing when the case is outside scope or unsupported.
4. Handoff quality: whether it names the right human route and gives the reviewer enough context.
5. Reviewer time: the time needed to check, correct and release the answer.

Counts and notes are often more useful than a single percentage: “three cases used an old document”, “two ambiguous questions received an answer without clarification”, or “the reviewer needed four minutes to verify each customer response”. If you do calculate a rate internally, define the denominator and keep the underlying cases. Do not publish an invented accuracy percentage that disguises critical failures.

Make polished answers work harder for their trust

Read every confident answer as a claim to verify. Open the cited source. Confirm that it exists, is the right version and supports the exact conclusion. Look for omitted conditions, changed dates, invented links, unit errors and an answer that quietly turns “it depends” into “yes”. A citation is useful evidence only when it is verifiable and relevant.

Then test the failure response. Does the assistant say what information is missing? Does it avoid exposing private material? Does it hand off to the correct team? Does the handoff carry the question, relevant source and reason for escalation, or does the person have to start again?

Set a release gate and a refresh calendar

Before release, agree which cases are blocking. A wrong answer about a low-risk formatting task may need correction. A wrong answer about a payment, medical matter, legal obligation, employment entitlement, privacy issue or customer commitment may require refusal and human review. Do not average away a high-harm failure.

Version the test set with the source snapshot, model or tool configuration, prompt, run date and reviewer. Keep failed cases; they are your most valuable regression tests. Re-run the set after a policy, price list, product, service process, prompt, model or integration changes. Add a new case whenever users find a failure that the set did not anticipate.

For a small team, a monthly review may be enough for a stable internal document assistant; a customer-facing system or frequently changing price list may need review after every approved change. Choose the interval from change risk and service impact, not convenience.

The decision is not whether the chatbot sounds intelligent. It is whether the organisation can show what it tested, what it accepts, what it refuses, what it escalates and who is accountable when the source or answer changes. Trust is conditional. A maintained red-team set makes that condition visible.