What is test automation? A practical guide

Test automation is the use of software to run tests, compare the actual results with the expected ones, and report the difference, without a person doing each step by hand. It covers the automated checks themselves and the machinery around them: the test data, the environments, the pipeline that runs the checks on every change, and the reports people act on.

Most teams do not struggle with the definition. They struggle with the decisions that follow it: what to automate, at which level, who maintains it, and how to stop a suite from turning into something nobody trusts. That is what this guide is about.

What test automation is, and what it is not

An automated test is a script or program that sets up a situation, performs an action and checks the outcome against a defined expectation. The test passes or fails with no human judgement in between. That last property is both its strength and its limit.

It is fast, repeatable and tireless. DORA’s research on test automation as a capability puts the case plainly: “Manual tests and inspections are not reliable, because people are poor at repetitive tasks like manual regression tests…” Running the same regression checks after every merge is exactly the job people do badly and machines do well.

What automation cannot do is notice something it was not told to look for. A script checks that the total is 118.00. It does not notice that the total is displayed in the wrong currency, that the button now overlaps the footer on a small phone, or that the new flow is simply confusing. That is the work of exploratory testing, and it stays with people.

So test automation is not “testing without testers”. It is a way to hand the repetitive checking to machines, so that testers spend their time on the questions machines cannot answer.

You will see “test automation”, “automated testing” and “automation testing” used interchangeably. Strictly, test automation is the wider discipline, including data setup, environment provisioning and reporting, while automated testing is the act of running automated checks. In search results and job titles they mean the same thing.

The levels you can automate at

The same behaviour can usually be checked at more than one level. Picking the lowest level that can see the defect is most of the skill.

Google’s testing team suggested a 70/20/10 split between unit, integration and end-to-end tests as “a good first guess”, and Martin Fowler’s practical test pyramid walks through why: broad UI tests are slow to run, slow to diagnose and fragile, so you want few of them. The exact ratio matters less than the shape. When most of a suite sits in the UI layer, that is usually because the UI was the only layer the testers were given access to, not because the risks live there.

What to automate first

Good candidates share a few traits. A check is worth automating when:

• It runs often. Every merge, every deployment, every release. A check run twice a year rarely pays back its build cost.
• The expected result is unambiguous. A status code, a stored value, a calculated total. If two testers could disagree on whether it passed, a script will not settle it.
• It is stable. The feature is not being redesigned next sprint.
• A failure would be expensive. Payments, permissions, data integrity, the sign-up path.
• It is tedious or error-prone by hand. Fifty input combinations, cross-browser checks, large data sets.

In practice the first automated suite is almost always the regression pack: the checks that confirm yesterday’s features still work after today’s change.

What to leave manual

• Exploratory testing, where the tester designs the next test based on what the last one revealed.
• Usability and visual judgement: is this clear, does it look right, would a customer understand this error.
• One-off checks, such as a data migration that runs once.
• Features still changing shape. Automate the API underneath and wait for the UI to settle.
• Anything where building the check costs more than it will ever save. Do the arithmetic below before assuming automation is cheaper.

A worked example: automating a login and password reset pack

A team tests login and password reset by hand before every release. The manual checklist has twelve items and takes about three hours, because resetting a password means waiting for emails and creating fresh accounts. They release twice a week.

The mistake would be to script those twelve items exactly as written, all through the browser. Instead, they sort each item by the lowest level that can see the defect:

Eight checks move below the UI, three stay in the browser, one stays with a person. The API checks run in seconds on every merge; the three browser journeys run on every deployment to staging.

Now the arithmetic, with the numbers from this example rather than an industry benchmark. The manual pack costs about 3 hours per release, twice a week: 6 hours a week. Suppose building the automated version takes 40 hours, and keeping it working takes 1.5 hours a week as the product changes. The weekly saving is 6 minus 1.5, so 4.5 hours, and the build cost is recovered after about 9 weeks. Change the inputs for your own situation. If the pack runs once a month, or the screens change every sprint, the same calculation can come out negative, and that is a real answer.

The saving in hours is not even the main benefit. The team now finds a broken reset link minutes after the change that broke it, instead of on release day.

How automated tests go wrong

Flaky tests

A flaky test passes and fails on the same code. Google reported in 2016 that about 1.5% of all its test runs had a flaky result, that almost 16% of its tests showed some flakiness, and that about 84% of the pass-to-fail transitions it saw involved a flaky test. In a suite like that, most of the time a test turns red, someone first has to work out whether the test or the code is at fault, and Google notes that real failures in flaky tests commonly get ignored along the way. People learn to rerun until green, and at that point the suite has stopped protecting anyone.

The most common cause is timing. A study of 201 commits that likely fix flaky tests, across 51 open-source projects, found that 45% of the categorised fixes (74 of 161) were “async wait” problems: the test checked a result before the asynchronous work producing it had finished. The cure is to wait for the specific condition you are about to assert, never for a fixed number of seconds.

Automating manual test cases one for one

Manual test cases are written for people. They contain steps like “check the page looks correct”, and they route everything through the UI because that is the only interface a manual tester has. Translated literally, they produce a slow, brittle UI suite. Automate the checks, not the documents: decide what each case is trying to prove, then pick the level, as in the worked example.

Record and playback as the finished product

Recorders are good for a first draft. Left unedited, the output depends on exact page structure and fixed timings, so it breaks with the next layout change. Treat a recording as something a tester rewrites, with stable selectors and real assertions.

Nobody owns the suite

Tests break as the product changes, and that is normal. A suite without a named owner and time set aside for maintenance is still there a year later, half of it disabled. The upkeep is part of the cost from day one, not a surprise. Our list of common test automation mistakes covers more of these patterns.

Green that means nothing

A test can pass because it checks nothing that matters: it asserts that a page loaded, not that the data on it is right. We wrote up a project where exactly this happened, in every test passed and nothing worked. A good habit is to break the feature on purpose once and confirm the test goes red. If it stays green, it was never testing that feature.

One habit that keeps a suite honest

Give every automated test a one-sentence answer to “if this fails, what is broken for a customer, and who acts?” Write it in the test name or its first comment. “Customers cannot reset a forgotten password; owner: accounts team.”

This sounds like paperwork. In practice it does three jobs. It forces the author to know what the test is for, which kills a surprising number of tests before they are written. It turns a red run into a message someone can act on without reading code. And it makes review easy: a test whose sentence nobody can write is a candidate for deletion.

Pair it with a rule for what happens after a failure. A check that fails twice in a row on the same code is a bug report, filed with the run’s logs and screenshots attached, in whatever tracker your team uses. We use our own, BugBoard. A failure that only lives in a chat message is forgotten by the next release. A flaky test goes onto a quarantine list with an owner and a fix-by date, and if the date passes, it is rewritten at a lower level or removed.

How to start

1. List the checks your team already repeats every release, with how long each takes by hand.
2. Sort each one by the lowest level that can see the defect, as in the table above.
3. Automate the most expensive failures first, at the API level where possible, and run them on every change.
4. Add a handful of UI journeys for the paths that must go through the real interface. Our guide to end-to-end testing covers how to choose them.
5. Put the suite in the pipeline so it runs without anyone remembering to start it. DORA’s guidance is to “create and curate fast, reliable suites of automated tests which are run as part of your continuous delivery pipelines”, and to keep them fast.
6. Measure trust, not coverage. Track how often a red result is a real defect. If that falls, fix or remove tests before adding more.

Choosing a framework comes after these decisions, not before. Our article on choosing a test automation framework covers that step, and the software testing life cycle guide shows where automation fits alongside planning and manual testing.

Frequently asked questions

Does test automation replace manual testers? No. It replaces repetitive manual checking. Exploratory testing, usability judgement and deciding what to test still need people, and someone has to design and maintain the automated checks. We look at where this is heading in is test automation the future of quality assurance?

How much of our testing should be automated? There is no correct percentage. Automate what runs often, has a clear expected result and would be expensive to miss. For many products that is most regression checks and very little of the exploratory work.

Which tool should we use? The one that fits your stack and your team’s skills. Playwright, Cypress and Selenium cover web UIs, Appium covers mobile, and your language’s unit test framework covers the lower levels. Tool choice matters much less than choosing the right level and keeping the suite reliable.

Where BetterQA fits

BetterQA is an independent software testing company in Cluj-Napoca. We design automation strategies, build suites at the right levels and take over suites that have stopped being trusted, through our test automation services. If you want an outside view of your current approach before building anything, our QA consulting team can review it.

Sources

• DORA, Capabilities: Test automation.
• Google Testing Blog, Just Say No to More End-to-End Tests, 2015.
• Google Testing Blog, Flaky Tests at Google and How We Mitigate Them, 2016.
• Luo, Hariri, Eloussi and Marinov, An Empirical Analysis of Flaky Tests, FSE 2014.
• Ham Vocke, The Practical Test Pyramid, martinfowler.com.