Automating test executions
Automation pays when it targets the right tests and the scripts are maintained like production code. We help you pick the candidates, build a framework your team can keep, and run it in your CI/CD pipeline.
Automate, or keep manual
- AutomateRegression, data-driven, CI checks
- Keep manualExploratory, usability, one-offs
- MeasureTime saved against upkeep
- MaintainScripts treated as code
What is test automation?
Test automation uses tools and scripts to run test cases without manual intervention. Done well, it cuts repetitive testing, makes continuous testing in CI/CD pipelines possible and gives fast feedback on every code change.
The hard part is choosing which tests to automate and keeping the scripts maintained as carefully as production code.
When to automate
| Scenario | Description | Recommendation |
|---|---|---|
| Regression testing | Tests rerun after every code change to confirm existing features still work. The frequency makes automation essential. | Automate |
| Repetitive tests | Tests run across many builds or releases. Automation removes the repetition and the slips that come with it. | Automate |
| Data-driven tests | Tests with many input combinations, such as boundary values and equivalence partitions. | Automate |
| CI/CD pipeline tests | Tests run on every commit or pull request, giving feedback without holding developers up. | Automate |
| Cross-browser tests | The same checks on many browser, OS and device combinations, which takes too long by hand. | Automate |
| Performance tests | Load, stress and endurance tests that need sustained execution and many simulated users. | Automate |
When to keep testing manual
Some tests cost more to automate than they save, or need a person to judge the result:
| Scenario | Description | Recommendation |
|---|---|---|
| Exploratory testing | Testers learn the system while testing it. This needs human judgement and domain knowledge. | Manual |
| Unstable UI | Features whose design still changes often. Script upkeep costs more than it saves until the UI settles. | Manual |
| One-time tests | Tests run once or rarely, such as a migration check. Building the script takes longer than running it by hand. | Manual |
| Usability testing | User experience, visual design and ease of navigation. These need human perception. | Manual |
Automation implementation process
Identify automation candidates
Go through the existing suite for high-value candidates: tests run often, on stable features, in high-risk areas.
Select tools and framework
Choose tools from your technology stack and your team's skills, and set the framework design (page object model, data-driven or hybrid).
Develop test scripts
Write maintainable, reusable scripts with proper error handling, logging and reporting.
Integrate with CI/CD
Add the tests to your pipeline, with triggers, parallel runs and failure notifications.
Maintain and scale
Update scripts as the application changes, fix or remove flaky tests, extend coverage and track whether the suite still pays for itself.
ROI considerations
ROI = Savings / Investment
Investment
Set-up (tools, framework, training, infrastructure), developing the scripts, and the upkeep of every script after that: maintenance, flaky-test fixes, framework upgrades and run time.
Savings
The same test cases run in much less time than by hand, so they can run more often. CT-TAS counts time to run a test manually and automated, the number of test cases and the number of test runs.
Break-even point
The run count or sprint after which the savings have paid back the investment. If the planned project is shorter than that turning point, automating is not worth it and manual execution saves time and effort (CT-TAS 5.1.1).
Long-term value
Time saved adds up release after release, and testers get time back for the exploratory work automation cannot do.
The investment depends heavily on how long the automated tests take to run, so we put each test at the right level of the test pyramid (many fast, isolated tests at the bottom, few end-to-end tests at the top). That shortens run time and improves ROI (ISTQB CT-TAS 3.1.1 and 5.1.1). We estimate the break-even point for your suite as the first step of the work.
Benefits and pitfalls
Key benefits
Automation gives measurable gains when it is applied to the right tests and maintained.
- Faster feedback on code changes
- The same steps every run
- Parallel execution across environments
- Unattended runs overnight
- Better regression coverage
- More tester time for exploratory testing
- Detailed execution logs and reports
- Scales across projects and teams
Common pitfalls
These are the mistakes behind most abandoned automation efforts.
- Automating unstable or changing features
- Poor script maintainability
- Aiming to automate every test
- Ignoring flaky tests
- Weak locators that break often
- Skill gaps and no training
- No framework or coding standards
- Treating scripts as throwaway code
Industry standards
Based on the ISTQB test automation syllabi
Our automation practice follows ISTQB guidance on automation strategy (CT-TAS), tool selection and implementation patterns (CTAL-TAE).
For a full automation service, see test automation; for what to rerun after each change, see regression testing.
How BetterQA does test automation
We start with an audit of your current tests, automate the candidates that will pay back, and keep the suite maintained as your product changes.
Engineers who test for a living
BetterQA, founded 2018, 50+ QA engineers. People plan and run the test automation and report what the results mean for your release.
Bugs you can act on
Every defect we find is written up in BugBoard, our own test management tool, or in your tracker, with steps to reproduce, the expected and actual result, and the environment.
A certified process
ISO 9001 for quality management and ISO 27001:2022 for information security. See our certifications.
How an engagement works
- €20-45 per hour
- First month is a trial
- First two weeks complimentary if you are not happy
- 30 days' notice after that
Automating test execution: common questions
What does automating test execution mean?
Running test cases with tools and scripts instead of by hand, usually triggered from a CI/CD pipeline on each change.
Which tests should be automated first?
Stable, frequently repeated tests in high-risk areas, such as the regression checks on your main user journeys.
Which tests should stay manual?
Exploratory and usability testing, one-off checks, and features whose design still changes every week.
How do you calculate test automation ROI?
Savings divided by investment, where the investment includes set-up, script development and ongoing maintenance (ISTQB CT-TAS 5.1.1). It improves the more often the tests run and the more stable the product is, and there is a break-even point before which manual execution is cheaper.
Why do automation projects fail?
Usually because they automate unstable features, ignore flaky tests, or treat scripts as throwaway code instead of maintaining them.
What does it cost?
The cost depends on the size of the suite and how much of it is worth automating. Our rate is €20-45 per hour. For ongoing work the first month is a trial, the first two weeks are complimentary if you are not happy, and after that it is 30 days' notice.
Ready to automate strategically?
Send us your current test suite. We will mark what to automate, what to keep manual, and what the automation would cost to build and keep.