Website chatbot testing: a practical launch checklist

Test a website chatbot against a small, repeatable set of real questions before launch. For each question, define the expected facts, acceptable sources and correct fallback. Include ambiguous requests, outdated information and requests for a person. A fluent answer passes only when it is accurate, relevant and within the assistant's actual permissions.

A chatbot can sound impressively confident while being wrong about your opening hours. Confidence is not a testing strategy, even when it arrives in a friendly speech bubble. This guide gives a small team a practical evaluation worksheet, a launch decision process and a routine for checking changes after the assistant is live.

Define what the assistant is allowed to do

Write a narrow scope before testing. Perhaps the assistant should explain services, answer questions from approved documents and direct visitors to the correct contact route. Separate those informational tasks from actions such as changing bookings, approving refunds or accessing account information. Do not imply that an assistant can perform an action merely because it can describe the process.

List topics it should decline, qualify or pass to a person. Include information that changes frequently and questions requiring individual judgement. Scope gives reviewers a consistent standard: they can assess whether an answer is appropriate without debating the assistant's role every time. It also helps keep the knowledge base focused on material the assistant genuinely needs.

Build a question set from real needs

Start with common support questions, sales enquiries and confusing parts of the website. Remove personal information from examples before using them as test material. Include the language people actually use, not only the polished wording in your FAQ. A customer may ask “Can I change it?” long before they explain what “it” means.

Create several categories: straightforward facts, multi-part questions, ambiguous requests, unsupported topics and requests for human help. Add a few paraphrases of important questions. The aim is not to guess every possible sentence. It is to test representative situations that reveal whether the assistant uses the right evidence and behaves sensibly when the evidence is insufficient.

Write expected outcomes before reading the answers

For each test, record the facts that must appear, the source that supports them and anything the assistant must not claim. A good expected outcome describes meaning, not an exact script. This allows natural language variation while preserving the important requirements. It also reduces the temptation to accept a plausible answer simply because it sounds polished.

Add columns for the actual answer, pass or fail, severity and the fix owner. Keep the original wording of a failed test so it can be repeated after a change. A summary such as “pricing problem” is too vague to reproduce reliably.

Test the knowledge base, not just the prompt

Many poor answers begin with poor source material. Look for duplicate policies, obsolete pages, missing headings and documents that assume internal knowledge. If two sources disagree, decide which one is authoritative and correct the material. Asking the assistant to “be more accurate” does not resolve a conflict that the organisation has left unresolved.

Make important answers easy to locate and understand. A clear page stating the current process is more useful than a collection of old announcements. Keep ownership and review dates for changing information. When a source is removed or updated, verify how the product refreshes its knowledge rather than assuming the new version is available instantly.

Include uncertainty and boundary tests

Ask questions the assistant cannot answer from its sources. A good response should acknowledge the limitation and suggest an appropriate next step. It should not fill the gap with a confident invention. Test partial knowledge too: perhaps the assistant can explain the general process but cannot confirm a particular customer's status.

Try requests that ask the assistant to ignore its role, reveal private information or claim authority it does not have. Use harmless test data. Check that public answers stay within the intended scope. These tests are not proof against every possible attack, but they can reveal obvious failures before visitors encounter them.

Check the human contact route end to end

A fallback is useful only if it leads somewhere real. Click the contact link, check the destination and confirm what information the visitor is asked to provide. If the assistant claims a person will respond, make sure the actual process supports that statement. Do not promise instant help when the destination is an inbox reviewed during limited hours.

Where the product supports a handoff, inspect what context reaches the human team. Where it only provides a link or address, write the response accordingly. The test is about the configured experience, not a feature you saw in another vendor's demonstration. A modest, accurate contact route is better than an imaginary support department.

Use severity as well as pass rate

A percentage can hide a serious failure. An assistant that passes many simple questions but invents a refund promise has a different launch risk from one that merely writes an awkward greeting. Classify failures by consequence: factual error, unsupported action, privacy issue, broken route, incomplete answer or presentation problem.

Decide which failures block launch and which can be corrected during a limited pilot. Set these rules before reviewing the final score. For important questions, repeat tests because generated responses can vary. Do not treat one successful run as a guarantee that the assistant will always produce the same answer.

Worked example: a service business pilot

Imagine a fictional business testing thirty questions across services, hours, booking steps and support routes. Twenty-six answers meet the expected criteria. Two are incomplete, one uses an outdated opening-hours page and one claims to confirm a booking even though the assistant has no booking access. The raw pass rate alone would miss the importance of that last answer.

The team corrects the conflicting hours source, clarifies the booking boundary and improves the missing content. They rerun the failed tests and a sample of previously successful questions. The booking response now explains how to request a booking without pretending it has been confirmed. The team then runs a limited pilot with staff reviewing early conversations.

The example demonstrates a decision process, not a universal passing score. Your threshold should reflect the assistant's role and the consequences of error. Keep the evidence behind the decision so a future update can be compared with the same standard.

Test the visitor experience on a phone

Check the assistant where visitors will use it. On a small screen, inspect whether the launcher blocks important controls, whether the conversation can be read comfortably and whether links open correctly. Test keyboard navigation and visible focus as well as touch. A correct answer can still be difficult to use in a cramped or confusing interface.

Review long answers, lists and contact details. The assistant should answer the question without overwhelming the reader with unrelated information. Test a slow connection and a failed request where practical. The interface should make the state understandable rather than leaving the visitor wondering whether to click again or abandon the conversation.

Repeat the evaluation when the product changes

Keep the test set as a living checklist. Rerun relevant questions when sources, instructions, models or integrations change. Add new cases when real conversations reveal a gap. Keep personal data out of the shared test set and use anonymised wording where possible. The aim is to preserve useful lessons without copying sensitive conversations unnecessarily.

Assign someone to review failed or unresolved interactions at a manageable interval. Group problems by cause and fix the source of repeated failures. Ten similar unanswered questions may need one better knowledge page, not ten increasingly complicated instructions. Track the change and its test result so improvements remain understandable over time.

Frequently asked questions

How many questions should be in the first test set?

Start with enough to cover the important categories and the consequences of failure. A focused set of representative questions is more useful than hundreds of duplicates. Expand it as real conversations reveal new situations.

Can we judge answers only by whether they sound natural?

No. Assess factual support, relevance, boundaries and the next action as well as tone. Fluent language can conceal a wrong fact or an unsupported promise, so reviewers need expected outcomes and approved sources.

Does a successful test guarantee every future answer?

No. Testing provides evidence under the conditions checked. Repeat important cases, monitor early use and retest changes. Keep the assistant's scope appropriate to the reliability you can actually demonstrate.

Do we need a specialist testing tool to use this method?

No. A spreadsheet and manual conversations can support a small initial evaluation. Use Chaat.ai's available setup and testing controls for the actual product, while keeping the worksheet as your team's independent record of expected and observed behaviour.