Testing software when every client runs a different version
A lot of B2B software is not one product running in one place. It is deployed per client, upgraded on each client's schedule, and configured differently for each of them. On any given day, one customer is on the release you shipped last month, another is two releases behind, and a third is still on a version you would rather forget. When a bug report arrives, it belongs to the version that client has, not to the code on your main branch. A tester who checks the newest build, cannot reproduce, and closes the ticket has answered the wrong question.
That changes what testing has to do. Testers need to reproduce on the exact version and setup the client runs, decide whether a fix in the newest release also has to go back to older ones, and make sure that backport does not break something else. This guide covers how to set that up so multi-version software testing stays manageable instead of turning into guesswork.
What do testers need before they can reproduce a client bug?
A supported-version matrix
Start with one document that lists every version you still support, which clients run it, its end-of-support date, and anything unusual about it: a different database, an old integration, a feature that was later removed. When it is out of date, testers guess, and guesses favour the newest version.
The matrix also forces a conversation that many teams avoid: how many versions can you realistically support? Every version on the list is a version someone has to be able to build, run and test. If the honest answer is that nobody can stand up a three-year-old release any more, that version is not supported in practice, whatever the contract says.
Environments you can switch to a given version on demand
Reproducing on the client's version should take minutes, not a request to the infrastructure team. The usual building blocks are:
- Tagged builds. Every release produces an artifact you can pull again by version number, months later, without rebuilding from an old branch.
- Containers. Each supported version runs as an image, so a tester can start version 4.2 next to version 5.1 on the same machine.
- Infrastructure as code. The environment around the application (database version, queues, third-party stubs) is defined in files kept per release, so the surroundings match the version too.
- Seed data per version. A database schema that matches the release, with test accounts and sample records, so testers are not migrating data by hand.
None of this is exotic. The point is to treat reproducing on an old version as routine work.
What must every bug report from a client carry?
In multi-version software, a bug report without a version is close to useless. Before a tester touches it, the report should state:
- The exact version and build the client runs, not "the latest" or "the one from spring".
- Configuration and client-specific settings, including feature flags, enabled modules, custom fields and integrations that are switched on for that client.
- The environment around it: database, browser, operating system, and whether it is the client's production, staging or a sandbox.
- Evidence: a screenshot or screen recording of what the client saw, ideally with the version number visible on screen.
Much of this information gets lost between the client and the bug tracker. A client tells a consultant or support agent, who summarises it in an email, which later becomes a ticket. By then the version is "probably the current one" and the custom setting that caused the bug is gone.
The fix is to make the path from report to ticket short, and to make the version a required field rather than a courtesy. This is one reason we built BugBoard. Support staff or consultants can hand it the screenshot or screen recording the client sent, and it turns that into a structured bug report with steps, expected and actual results. A tester checks it and adds the version and configuration before it counts, and then it is pushed to Jira, or Azure DevOps, with that information intact. Whatever the tool, a ticket without a version goes back to whoever filed it.
When does a fix in the newest version also need backporting?
Once a bug is reproduced on the client's version, the next question is where else it lives. Testers should check the same scenario on every supported version, not just the one in the report. Often the bug exists in several, sometimes it was already fixed in a newer release, and occasionally it only appears in the newest one, which is a different and more urgent problem.
That check produces the information the team needs to decide on a backport: applying a fix made in newer code to an older release. The decision is a product call, but testers make it possible by showing exactly which versions are affected. The other inputs are severity, how many clients on each version would hit it, and how far the code has moved since.
Backports are where regressions hide. The fix was tested against the newest code, then adapted for older code under pressure. Treat every backport as its own change: reproduce the bug on that version, apply the fix, confirm it is gone, and run the regression tests for that version around the changed area. Do not assume that because it passed on the newest release, it passes everywhere.
Which versions get regression testing on each release?
Running the full suite on every version for every release sounds safe and rarely survives a real schedule. Decide in advance, per type of release, what gets tested where.
Tag tests by the version a feature exists in
A test suite for multi-version software needs to know which features exist in which versions. Tag each test with the version where its feature was introduced, and where relevant, the version where it was changed or removed. Then regression against version 4.2 selects the tests that apply to 4.2, instead of running everything and sorting expected failures from real ones. This is easier when the tests are automated, and it is a sensible thing to ask of any test automation effort from the start.
Set a rule per release type
A workable default, which you can adjust to your risk:
- New major or minor release: full regression on the new version, plus upgrade testing from each supported version to it.
- Patch on the newest version: regression around the changed area, plus a smoke test of the main user journeys.
- Backport to an older version: the reproduce-fix-confirm cycle above, plus regression for that version around the changed area.
- Security fix across versions: confirm the fix on every supported version it touches, since a fix that lands in some versions and not others leaves the gap open.
Upgrade paths deserve their own attention. Clients rarely move one version at a time. Test the jumps clients actually make, from the oldest supported version to the newest, including data migration and settings carried over, because that is when client-specific configuration tends to break. How this fits into the wider release process is covered in our article on the QA role in release management.
How do release notes and end-of-support help testing?
Release notes are written for clients, but they are also one of your best testing documents. Notes that list, per version, what changed, what was fixed and which known issues remain let a tester see in minutes whether a reported bug is already fixed in a later release, already known, or new.
An end-of-support policy is the other half. Every version you support adds environments, tests and backport decisions. A clear, published policy that says how long each version is supported, and what happens after, keeps the matrix from growing without limit. It also gives support a fair answer for a client on a retired version: here is the release where it is fixed.
From the testing side, the policy only works if it is enforced. A version that is officially retired but still quietly patched for one client is still a supported version, with all the cost that implies, minus the tests.
Where an outside QA team fits
Most of this work is steady and easy to postpone: keeping the matrix current, old environments runnable, and reports triaged against the right version. It is the first thing to slip when the team is busy with the next feature.
That is where we usually come in. We work in your tools, your Jira, your repository and your CI, and what we build stays yours. A tester checks every generated bug report and test case before it counts, and nothing on our platforms is used to train AI models on your data. We can run the testing function for you as managed testing, or add engineers under your own QA lead. Rates are $25 to $45 an hour, and a team is usually working on your product about two weeks after the first call.
Frequently asked questions
Supporting more versions than your team can test?
Tell us which versions your clients run and where bugs slip through. We will tell you what we would set up first.
Book a call See our testing servicesNeed help with software testing?
BetterQA provides independent QA services across manual testing, automation, security audits, and performance testing. ISO 27001, 9001, 14001 and 13485 certified.
