A chatbot can be polite, fast and wrong. We test whether it answers from your own policy and documents, hands over to a person when it should, keeps track of a conversation, and keeps working after you change the model or the prompt.
We build a golden question set with the correct answers from your policy, then check each reply against it.
It hands over when asked, when it is unsure and when the topic is out of scope, and passes the context on.
The voice fits your brand and the moment, including when the customer is upset.
It remembers what was said, handles a change of mind and does not mix up two customers.
Simple attempts to make it ignore its rules. Deeper attacks are covered by AI red teaming.
The same questions rerun after a model, prompt or knowledge-base change.
Tickets are created, fields filled and customers identified correctly in your systems.
The core of chatbot testing is a list of real questions, each with the answer your company would give. Customers ask in their own words, so for each question we add rephrasings, typos and questions that look similar but have a different answer. A reply is checked against your documents, not against what sounds plausible.
A language model gives non-deterministic output: the same question can get a different wording next time (ISTQB CT-AI 4.1.3). So we judge a reply by whether it is correct and within policy, and we run the important questions more than once. You keep the set and can extend it as your policy changes.
JRNY, our sales CRM, uses an LLM to write outreach emails and replies and to summarise inbound mail. Azorel, our Slack and chat agent, drafts posts, replies and emails. In both, the prompt asked for the right thing and the model drifted anyway.
The model ignored its own banned-phrase rule in about 1 email in 12 to 20 (2 to 3 leaks in 39 runs), and production had no check. Now every draft is checked and flagged on the approval screen.
In a Romanian cold pitch the model turned a team size into a client count. Now the facts are passed in, a backstop check catches the rest, and the tests were written red first.
A draft referred to a call that was never held, because today's date was not passed to the model. Now the date is passed and the prompt guards against it.
We changed every default to fail closed to human approval. All live campaigns are now on approval.
The lesson. A prompt is a request, not a control. We added a deterministic check on the send path, kept a person as the last gate, and run an evaluation set of 22 cases (deterministic checks plus an LLM judge) on every prompt version.
Which topics the bot covers, which it must refuse and what it can do in your systems.
Real questions with correct answers from your policy, plus variants, edge cases and out-of-scope requests.
Automated runs for repeatable questions, and an engineer reading the conversations where judgement is needed.
Each wrong answer with the question, the reply, the correct answer and the risk.
The same set after every model, prompt or knowledge-base change.
Findings ranked by risk, with the conversation for each.
The golden questions and correct answers, ready to rerun when something changes.
Each failure logged in BugBoard or your tracker.
For hostile users, see AI red teaming. If the bot speaks on the phone, see voice agent testing. If it calls tools, see AI agent and MCP testing. The wider view is on AI quality assurance.
Checking that a chatbot gives correct answers, stays within policy, remembers the conversation, hands over to a person and keeps doing so after changes.
With a golden question set: real questions and the correct answers from your own documents. Each reply is compared to the correct answer.
Yes. The question set is handed over and reruns after each model, prompt or knowledge-base change, so you see what moved.
Yes. We check that tickets, fields and customer records are created and matched correctly.
Yes, it is the same work under another name. It covers chat and, with voice agent testing, calls.
Send us a few real customer questions. In a complimentary call we will show you how we would test them.
