Regression testing: how to choose what to run

Regression testing is re-running tests over software that has changed, to confirm the change did not break something that used to work. The ISTQB glossary puts it as testing a previously tested program after modification, to make sure defects have not been introduced in unchanged areas.

That definition is not the hard part, and it is not why anyone is still reading. The hard part arrives about eighteen months into a product's life, when the suite takes four hours, the release window is forty minutes, and somebody has to decide which tests do not run. Every team reaches that point. Most handle it by quietly running whatever finishes in time, which is a decision nobody made and nobody can defend.

This is about making that decision deliberately.

Why the suite always outgrows the window

Regression suites grow monotonically and nothing removes from them. Each escaped defect adds a test. Each new feature adds its own coverage plus a few cases guarding its interaction with older features. Nobody is ever assigned to delete tests, and deleting one feels like removing a safety net even when the test has not failed in three years.

So runtime rises, and two things happen. Teams start running the suite less often, usually nightly instead of per-merge, which means a regression now sits undetected for up to a day and lands on top of other changes that make it harder to isolate. And flaky tests accumulate, because a longer suite has more chances to be flaky, until a red run stops meaning anything and people rerun it until it goes green.

A suite in that state is worse than a smaller one. It costs the full four hours and provides almost no signal, because nobody believes its failures.

What belongs in a regression suite

The useful question is not "does this test add coverage" but "if this broke in production, what would happen". Four properties decide it, and they can be assessed by anyone who knows the product.

Blast radius. What is the worst outcome if this path silently breaks? Money moving incorrectly, data loss, a security or access-control failure, or anything a user cannot undo sits at the top and stays in every run, regardless of runtime budget. A misaligned icon does not.

Reach. How many users touch this, and can they route around it? Login is on every session's path and has no workaround. An export button used by nine admins a month has both lower reach and an obvious manual fallback.

Change frequency. Code that changes often breaks often. Your version control already knows which files those are, and the answer is frequently not where people assume:

git log --since='12 months ago' --name-only --pretty=format: -- src/ \
  | sort | uniq -c | sort -rn | head -20

Run that, then ask which tests cover the top of the list. In most repositories a handful of files carry a disproportionate share of the churn, and they are the ones worth guarding hardest.

Defect history. Areas that have broken before break again. If you have a defect tracker with components on the tickets, the count of production defects per area over the last year is the single best predictor you own, and it is one query away.

A test that scores low on all four is a candidate for deletion, not for a slower pipeline. That sentence is the one most teams need permission to act on.

Four selection strategies

Once selection is deliberate, there are four recognised ways to do it, and most working teams end up combining the last two.

Prioritisation deserves the emphasis because it is the cheapest to adopt and it degrades gracefully. If your suite is ordered by risk, then a run that is cut short has still executed the most valuable tests, and you can say exactly which ones did not run. An unordered suite cut short has tested an arbitrary subset and you can say nothing.

Tagging is usually enough to implement it. Most runners support it directly:

# per merge: critical paths plus anything touching what changed
pytest -m "critical or payments" --maxfail=1

# nightly: everything, highest risk first so a timeout still means something
pytest -m "critical" && pytest -m "not critical"

Change-based selection can be approximated without specialist tooling by mapping directories to tags, and an approximation is worth having:

CHANGED=$(git diff --name-only origin/main...HEAD)
TAGS="critical"
echo "$CHANGED" | grep -q '^src/payments/'  && TAGS="$TAGS or payments"
echo "$CHANGED" | grep -q '^src/auth/'      && TAGS="$TAGS or auth"
pytest -m "$TAGS"

One caution that is easy to get wrong. Selection based on changed files is blind to indirect coupling, which is precisely where regressions live: a change in a shared utility can break a feature whose directory was not touched. Treat change-based selection as what runs per merge, never as what runs before a release. The full prioritised suite still has to run at some scheduled point, or the strategy has simply stopped testing the interactions.

A worked prioritisation

Take an e-commerce product with a 340-case suite that now runs for three hours against a forty-minute window. Scoring by the four properties above produces something like this.

Tier 4 is the point of the exercise. Roughly a third of a mature suite is usually redundant, dead or cosmetic, and removing it is what makes the rest affordable. Delete it in version control where it can be recovered, rather than leaving it skipped, because a permanently skipped test is a comment pretending to be a safety net.

The resulting arrangement runs 22 minutes per merge inside a 40-minute window, everything nightly, and the whole suite before a release. Nothing in tier 1 or 2 is ever cut.

Keeping the suite from rotting

Selection solves today's runtime. Keeping it solved takes a few standing habits.

Treat flakiness as a defect with an owner. A test that fails intermittently is not a minor annoyance, it is an attack on the credibility of every other test in the run. Track the flake rate, and quarantine anything above a small threshold into a separate non-blocking job with a ticket and a name against it. A quarantine with no exit date is just a slower delete, so give it one.

Budget the runtime, and enforce it. Decide the per-merge ceiling, put it in CI, and fail the build when it is exceeded. Without a ceiling the suite grows until it hits whatever people will tolerate, which is always more than what is useful.

Review the suite on a schedule. Once a quarter, list the tests that have not failed in a year and ask whether each still guards something real. Some do, because they cover a path that genuinely never breaks and would be catastrophic if it did. Many do not.

The numbers worth watching are few. Suite runtime and its trend. Flake rate. Escaped defects, meaning production defects in areas the suite claims to cover, which is the only measure of whether the suite is doing its job. Coverage percentage is not on that list, because a suite can cover every line and assert nothing useful.

Manual and automated

Regression testing is the strongest case for automation in testing, because it is repetitive, unchanging and frequent, which is exactly the profile humans handle badly. A regression suite that is manual will be cut under deadline pressure every time.

It does not follow that everything should be automated. Automation is worth its maintenance cost where the behaviour is stable; on a screen being redesigned every sprint, an automated test is a recurring bill for a result you already expect to change. Exploratory regression around a redesigned area, done by a person, finds things a scripted test cannot, because the scripted test only asserts what someone thought of in advance.

The practical split is to automate tiers 1 and 2 completely, automate tier 3 where it is stable, and keep a short exploratory pass on whatever changed most this release.

Where it sits in the process

Regression testing runs during the execution phase of the software testing life cycle, after a build has passed its smoke test and after the targeted sanity check on whatever changed. Those two are fast checks on one build and one change; the regression suite is the broad net, and it is the one that catches damage nobody predicted.

Which is the thing worth ending on. The whole reason regression testing is hard to prioritise is that its value is invisible when it works. A suite that has caught nothing this quarter looks like an expense, right up until the quarter it does not run.

Sources

The ISTQB glossary defines the term, and ISO/IEC/IEEE 29119 covers test selection and documentation.

If the practical blocker is that nobody owns the suite, that is the usual shape of a regression testing engagement with BetterQA: score the existing suite against the four properties above, delete the third of it that earns nothing, and get the rest running on every merge.

Built by BetterQA