QA trial, pilot or proof of concept: what the first two weeks should prove
Most buyers who talk to a QA vendor end up asking for the same thing in different words. A trial. A pilot. A proof of concept. An intro period. Sometimes they ask for it and then ask what it is called. The name matters less than the shape, because a badly set up trial tells you almost nothing, and a well set up one tells you more than every sales call put together.
This guide is written for the person running the evaluation, often someone who will have to summarise it for management afterwards. It covers what to agree before the trial starts, what to judge while it runs, what to ignore, and how to write it up.
Trial, pilot and proof of concept are mostly the same thing
In product development a proof of concept shows that an idea can work at all. In buying QA services the term is used more loosely, and so are the others. What buyers mean by all of them is a short, real piece of work on their own product, done before any long contract, so they can judge the vendor on results instead of slides.
The differences that do matter are practical. Who pays, how long it runs, what happens at the end, and whether the people on the trial are the people you would get afterwards. Ask about those four things and the label stops mattering. Some vendors charge for a pilot and some do not. Some run it for a week and some for a month. A week is usually too short to see anything beyond the first round of obvious bugs. Two weeks is the shortest period we have found that shows how a team works once the novelty has worn off.
What to agree before day one
A trial that starts without these in place spends its first week waiting for access, and you end up judging the vendor on your own onboarding.
- One area of the product, named. Not "the app". A feature, a user journey or a module that matters to you and that changes often enough to be worth testing.
- Access. A test environment, test accounts with the right roles, and, if the vendor will look at code or automation, the repository. Sign the NDA first so nobody waits on it.
- One contact on your side. Someone who can answer "is this a bug or intended" within a day. Without that person the testers guess, and you end up judging their guesses.
- What you will judge. Write it down now, using the next section. It stops the decision at the end turning into a vague feeling.
- The named people. Ask whether the engineers on the trial are the ones who would stay on the account. If not, the trial is a demonstration, not a test.
What a good trial shows you
Bug reports a developer can act on without asking
Pick five reports at random and hand them to a developer. Can they reproduce each one from the steps alone? Is the expected behaviour stated? Are the environment and evidence attached? Reports that need a conversation to understand cost your developers time on every single defect, for as long as the engagement lasts.
Severity you agree with
Look at how the vendor ranked its findings. If the top of the list is cosmetic and a broken payment edge case sits at the bottom, the team found bugs but did not understand your product. Getting priority and severity right is the clearest early sign of judgement.
Questions about your requirements
The best testers ask uncomfortable questions in the first week: what should happen here, who is allowed to see this, is this behaviour intended. A trial that produces a list of questions about your requirements alongside the bug list is a good sign. It means someone is reading the product, not just clicking through it.
Coverage of what you said mattered
Compare where the effort went with the area you named before day one. A vendor that drifts to easier parts of the product is telling you how it will behave when nobody is watching.
How they talk to you
Note how often you heard from them without asking, and whether bad news arrived early or at the end. That cadence is what you are buying for the next year.
Do not judge a QA trial by the number of bugs
The bug count is the number every vendor will lead with, and it is the least useful one. On a product nobody has tested, any competent team will find a lot. Buyers often know this already. The number of issues is rarely what worries them. What worries them is whether the person on their product is any good, and whether they will be able to tell.
A high count can even be a warning: forty minor findings and no requirement questions suggests a team optimising for a number. Ten well-written, correctly ranked findings and three good questions about a requirement nobody had thought through is a better trial.
Likewise, do not weigh a polished tooling demo too heavily on its own. Tools matter, and ours do real work on every engagement. But a demo proves the vendor is capable. The trial is where you find out whether hiring them reduces your risk, which is the thing you are actually buying. Our article on what testing vendors will not tell you before you sign covers the questions a demo cannot answer.
How to write it up for management
If you are comparing several vendors, the decision probably happens in a meeting you have to prepare for. Keep the write-up to one page per vendor, in the same order for each, so they can be compared side by side. State the area tested and for how long. Give three example findings with their severity, in plain language. List the requirement questions the team raised. Say how the people were to work with, and whether they would be the same people going forward. Finish with the price and the commitment it assumes.
That page is easier to defend than a score out of ten, and it holds up when someone asks why you chose one vendor over another. If you want a fuller checklist for the wider evaluation, our guide to evaluating a QA company covers technical capability, security and red flags.
The BetterQA trial
Ours is simple, and it is the same for every client. Two weeks, free if you are not happy with what we deliver. We invoice from week three. In those two weeks we embed, read the product, write the first tests and file real bugs. At the end you decide whether to continue.
It exists because QA quality is very hard to judge from a proposal, and we would rather be judged on bugs we found in your product. A team can usually be working on your product in about two weeks, and after the trial you get a named team, the same people month to month. If the trial goes well, the engagement continues either as a fully managed testing function or as engineers working inside your team, both described on our QA outsourcing page.
Frequently asked questions
Put us on your product for two weeks
Name the area that worries you most. We will test it, file real bugs and tell you what we would do next. You decide whether to continue.
Talk to our teamNeed help with software testing?
BetterQA provides independent QA services across manual testing, automation, security audits, and performance testing. ISO 27001, 9001, 14001 and 13485 certified.
