An AI feature can pass every test you wrote and still give a wrong answer tomorrow. We test AI systems for wrong answers, drift, bias and misuse, using methods from the ISTQB Certified Tester AI Testing syllabus, and we report what we found in a form your team can act on.
Grounded answers, golden question sets, escalation to a human, tone and regression after changes.
Chatbot testing →Real-caller behaviour, latency, handover, hallucinated policy, load, and classic menu testing.
Voice agent testing →Tool calls, runaway loops, permissions, tool poisoning and regression on model change.
Agent and MCP testing →Jailbreaks, prompt injection, leakage and tool abuse, mapped to the OWASP LLM Top 10.
AI red teaming →Classic tests compare an output to one expected result. For AI-based systems that often fails. The ISTQB CT-AI syllabus (4.1.3) lists why: outputs are probabilistic and can vary for identical inputs, specifications are often incomplete, the task may be too complex to check by hand, "correct" can be subjective, and self-learning systems change what correct means over time.
So we do not promise a pass or fail on every answer. We agree thresholds and tolerance ranges with you, test against risk, and say plainly what the results do and do not show. Testing shows the presence of defects, not their absence.
For a plain-language guide see our AI testing guide. For the tools we use ourselves see QA with AI. For AI that does the testing, see agentic QA testing.
LLM answers that sound right and are not. We cross-verify output against trusted sources (CT-GenAI 3.1.2).
No single right answer to compare with. We set thresholds, or use back-to-back and A/B comparison (CT-AI 6.1.9, 6.1.10).
Same prompt, different answer. Temperature, context window and system prompt all change output, so we test across them (CT-AI 4.2.1).
The input data or its meaning changes after release and accuracy slips. We test for both kinds (CT-AI 6.1.7).
We attack the system on purpose to provoke harmful output, then you fix what we find (CT-AI 4.2.2).
Change an input in a known way and check the output moves as it should. Useful when no oracle exists (CT-AI 6.1.5).
Behind many chatbots sits a language model that answers from your documents, a setup called retrieval-augmented generation (RAG). Two things can fail: the search that picks the documents, and the model that writes the answer from them. We test both separately, so you know which one to fix.
Retrieval. For each question in an evaluation set, did the right document come back? Answer. Given the right document, is the reply correct, complete and limited to what the document says? We build the evaluation set with you from real questions, score the replies against the correct answers, and hand the set over so you can rerun it when the model, the prompt or the documents change.
A second AI can help score replies at scale, but ISTQB cautions that using one AI to check another can repeat the same bias (CT-AI 4.2.1), so an engineer reviews what matters.
Which AI features, which harms matter, which thresholds you accept. Risk-based, as CT-AI 4.3.2 asks.
Representative prompts and data, plus edge cases and adversarial examples, agreed with you.
Black-box checks, metamorphic tests and red teaming, scripted where repeatable and reviewed by an engineer where judgement is needed.
Failures by risk, with the prompt, the output and the expected behaviour, written up in BugBoard or your tracker.
Drift checks on live data so a slow decline is seen by you before your users see it.
We run AI in our own products and write down what goes wrong. Four short write-ups:
You receive a written report, the evaluation set handed over for reruns, and each finding logged in BugBoard or your tracker.
Prompt injection and data exposure are covered by AI red teaming. The AI Security Toolkit scans the application around the AI with 18 AI agents on every scan.
We never build your product. The testing is independent of the people who wrote the model integration.
Testing and monitoring systems that contain machine learning or generative AI, where outputs are probabilistic and expected results are hard to define. It adds checks for data quality, bias, drift and adversarial input to normal functional testing.
Black-box prompts across normal, edge and hostile input, with thresholds for pass and fail, plus red teaming and review by an engineer. The model parameters and context window are part of what we vary.
The difficulty of knowing the correct expected result for an input. In AI systems it comes from non-determinism, incomplete specifications and subjective correctness (ISTQB CT-AI 4.1.3).
A technique that derives follow-up test cases from a passed one by changing the input in a known way and checking the output changes as the relation says it should (ISTQB CT-AI 6.1.5).
Yes. Drift testing compares live inputs and outcomes with what the model was trained on and flags change against a threshold (CT-AI 6.1.7).
Tell us what it does and what a wrong answer would cost you. In a complimentary call we will propose what to test first.
