Automating test executions
Automation pays when it targets the right tests and the scripts are maintained like production code. We help you pick the candidates, build a framework your team can keep, and run it in your CI/CD pipeline.
Automate, or keep manual
- AutomateRegression, data-driven, CI checks
- Keep manualExploratory, usability, one-offs
- MeasureTime saved against upkeep
- MaintainScripts treated as code
What is test automation?
Test automation uses tools and scripts to run test cases without manual intervention. Done well, it cuts repetitive testing, makes continuous testing in CI/CD pipelines possible and gives fast feedback on every code change.
The hard part is choosing which tests to automate and keeping the scripts maintained as carefully as production code.
When to automate
| Scenario | Description | Recommendation |
|---|---|---|
| Regression testing | Tests rerun after every code change to confirm existing features still work. The frequency makes automation essential. | Automate |
| Repetitive tests | Tests run across many builds or releases. Automation removes the repetition and the slips that come with it. | Automate |
| Data-driven tests | Tests with many input combinations, such as boundary values and equivalence partitions. | Automate |
| CI/CD pipeline tests | Tests run on every commit or pull request, giving feedback without holding developers up. | Automate |
| Cross-browser tests | The same checks on many browser, OS and device combinations, which takes too long by hand. | Automate |
| Performance tests | Load, stress and endurance tests that need sustained execution and many simulated users. | Automate |
When to keep testing manual
Some tests cost more to automate than they save, or need a person to judge the result:
| Scenario | Description | Recommendation |
|---|---|---|
| Exploratory testing | Testers learn the system while testing it. This needs human judgement and domain knowledge. | Manual |
| Unstable UI | Features whose design still changes often. Script upkeep costs more than it saves until the UI settles. | Manual |
| One-time tests | Tests run once or rarely, such as a migration check. Building the script takes longer than running it by hand. | Manual |
| Usability testing | User experience, visual design and ease of navigation. These need human perception. | Manual |
Automation implementation process
Identify automation candidates
Go through the existing suite for high-value candidates: tests run often, on stable features, in high-risk areas.
Select tools and framework
Choose tools from your technology stack and your team's skills, and set the framework design (page object model, data-driven or hybrid).
Develop test scripts
Write maintainable, reusable scripts with proper error handling, logging and reporting.
Integrate with CI/CD
Add the tests to your pipeline, with triggers, parallel runs and failure notifications.
Maintain and scale
Update scripts as the application changes, fix or remove flaky tests, extend coverage and track whether the suite still pays for itself.
ROI considerations
ROI = (Time saved - Maintenance cost) / Initial investment
Initial investment
Tool licences, framework setup, script development, team training and infrastructure.
Time saved
Less manual execution, faster feedback and tests that run in parallel.
Maintenance cost
Script updates as the application changes, framework upgrades, flaky test fixes and infrastructure upkeep.
Long-term value
Time saved adds up release after release, and testers get time back for the exploratory work automation cannot do.
The break-even point depends on how often the tests run and how stable the product is, so we estimate it for your suite as the first step of the work.
Benefits and pitfalls
Key benefits
Automation gives measurable gains when it is applied to the right tests and maintained.
- Faster feedback on code changes
- The same steps every run
- Parallel execution across environments
- Unattended runs overnight
- Better regression coverage
- More tester time for exploratory testing
- Detailed execution logs and reports
- Scales across projects and teams
Common pitfalls
These are the mistakes behind most abandoned automation efforts.
- Automating unstable or changing features
- Poor script maintainability
- Aiming to automate every test
- Ignoring flaky tests
- Weak locators that break often
- Skill gaps and no training
- No framework or coding standards
- Treating scripts as throwaway code
Industry standards
Based on the ISTQB test automation engineering syllabus
Our automation practice follows ISTQB guidance on automation strategy, tool selection and implementation patterns.
For a full automation service, see test automation; for what to rerun after each change, see regression testing.
How BetterQA does test automation
We start with an audit of your current tests, automate the candidates that will pay back, and keep the suite maintained as your product changes.
Engineers who test for a living
BetterQA, founded 2018, 50+ QA engineers. People plan and run the test automation and report what the results mean for your release.
Bugs you can act on
Every defect we find is written up in BugBoard, our own test management tool, or in your tracker, with steps to reproduce, the expected and actual result, and the environment.
A certified process
ISO 9001 for quality management and ISO 27001:2022 for information security. See our certifications.
How an engagement works
- €20-45 per hour
- First month is a trial
- First two weeks complimentary if you are not happy
- 30 days' notice after that
Automating test execution: common questions
What does automating test execution mean?
Running test cases with tools and scripts instead of by hand, usually triggered from a CI/CD pipeline on each change.
Which tests should be automated first?
Stable, frequently repeated tests in high-risk areas, such as the regression checks on your main user journeys.
Which tests should stay manual?
Exploratory and usability testing, one-off checks, and features whose design still changes every week.
How do you calculate test automation ROI?
Time saved minus maintenance cost, divided by the initial investment. It improves the more often the tests run and the more stable the product is.
Why do automation projects fail?
Usually because they automate unstable features, ignore flaky tests, or treat scripts as throwaway code instead of maintaining them.
What does it cost?
The cost depends on the size of the suite and how much of it is worth automating. Our rate is €20-45 per hour. For ongoing work the first month is a trial, the first two weeks are complimentary if you are not happy, and after that it is 30 days' notice.
Ready to automate strategically?
Send us your current test suite. We will mark what to automate, what to keep manual, and what the automation would cost to build and keep.