QA glossary
What is a flaky test?
A flaky test is an automated test that passes and fails on the same code, with no change to the code or the test. Its result cannot be trusted, so it hides real failures and teaches a team to ignore red builds.
An example from QA practice
A checkout test clicks the Pay button as soon as the page loads. On a fast build machine it passes. When the payment widget loads slowly, the button is not there yet and the test fails. A developer re-runs the pipeline, it goes green, and nobody looks further. Three weeks later a real regression breaks the button, and that failure is re-run away too.
Common causes
Most flakiness traces back to a short list:
- Timing: fixed waits instead of waiting for a condition, so the test races the page.
- Shared data: tests that depend on leftovers from other tests or on the order they run in.
- Outside services: a real network call, a third-party sandbox or a clock the test does not control.
- Unstable locators: locators tied to generated IDs or to element order that varies between page loads.
- Load on the runner: parallel jobs competing for the same machine.
What to do with one
Do not hide it behind automatic retries; a retry makes the build green and the cause invisible. Move the test out of the blocking set, give it an owner and a deadline, find the cause, then fix the test or delete it. Flakiness can be reduced and contained, but any suite that drives a browser will see it again, so the working habit is to detect it early and keep it visible.
How BetterQA does it
Flows, our browser test tool, has self-healing: when a locator stops matching because the page was changed on purpose, it finds a replacement and repairs the test, so a real change does not leave a pile of red tests that people learn to ignore. That is a different problem from flakiness: timing, data and ordering problems still need an engineer. Flows exports its tests to Playwright, Cypress or Selenium, so the tests are not locked into Flows.
Related terms
Spotting the patterns that make browser tests unreliable.