QA glossary
What is red teaming in AI testing?
Red teaming an AI system means attacking it on purpose to make it produce harmful or unwanted output, such as leaked private data, unsafe advice or instructions it was told never to give. What the testers manage to provoke is then used to make the system harder to misuse before real users find the same weakness.
An example from QA practice
A bank adds an AI assistant to its app. Before launch, testers spend sessions trying to make it misbehave: they ask for another customer's balance, hide an instruction inside an uploaded document, and over a long conversation slowly talk it into giving investment advice it must not give. Each success is logged with the exact prompts, so the team can add a guardrail and the testers can try the same attack again.
What it covers
The ISTQB AI Testing syllabus (CT-AI v2.0, section 4.2.2) describes red teaming as a systematic, often black-box form of fault attack. It is especially important for generative AI because the range of possible inputs and outputs is so large. It covers:
- Security: attacks from outside, including indirect prompt injection and malicious content hidden in documents the system retrieves.
- Safety and harm: harmful output under ordinary use, with no attacker involved, such as unsafe medical advice.
- Privacy, fairness and misinformation: the same probing applied to leaks, bias and made-up facts.
The syllabus places it after internal quality checks and before deployment, often as multi-turn conversations, and notes it is becoming a regulatory expectation for some AI systems, for example under the EU AI Act.
How BetterQA does it
Red teaming is a step in our AI risk testing. Our engineers probe a client's AI feature with adversarial prompts and record every one that works. For the rest of the application, the AI Security Toolkit coordinates AI agents and open-source scanners on every scan, and an engineer reviews every AI-generated finding before it reaches the report. We only test, we never build: we show what the system can be made to do, and the client's team decides the fix.