If you pay testers for every bug they report, you will get a lot of bug reports. That part works exactly as designed. What you will not reliably get is the handful of bugs that cost your business money, because the payment model never asked for them. Counting bugs measures how hard people worked. It does not measure how much risk is left in your product.
I had this conversation again recently with the founders of a two-sided marketplace startup. They were weighing a crowdsourced testing service against a dedicated team, and the per-bug model looked attractive on paper: you only pay for results. The problem is what “results” means once money is attached to it.
What per-bug pay actually rewards
A tester paid per unique bug has one rational strategy. Report as many distinct issues as possible, as fast as possible, before someone else does. A misaligned icon on one Android screen size is a unique bug. A typo in a tooltip is a unique bug. A button that is two pixels off in Safari is a unique bug. Each one earns the same as a broken checkout.
I am not criticising the people doing the work. They are responding to the incentive in front of them. If I were paid that way, I would do the same thing.
The result is predictable. You can write your instructions carefully and still receive 100 reports, of which maybe one matters to the business. Somebody on your side then has to read all 100, reproduce the interesting ones, close the duplicates and argue about the rest. That triage time is a cost nobody puts in the quote.
There is a quieter cost too. A big number feels like progress. A dashboard showing 340 bugs found looks like a product that has been tested hard. It might be. It might also be a product where nobody ever tried to sign up with a real email address and pay.
On a live marketplace, volume has side effects
This is where the founders and I spent most of the call, and where they changed their minds.
Their platform was already live, with real people on both sides. Testing a two-sided marketplace properly means creating activity on both sides: posting jobs, applying for them, messaging, agreeing terms. Now imagine dozens of crowd testers, each paid per unique finding, all doing that at once on production.
They will create listings. Lots of them. Some will look plausible, some will be nonsense, some will be edge cases built to trigger an error. Real users browsing the marketplace have no way to tell a test listing from a genuine one. Someone looking for work applies to a job that only exists so a tester could check a validation message. Someone hiring sees a wall of junk profiles and decides the platform is not serious.
One of the team said it better than I did: they had not seen it that way before, but the reputation of the platform could be damaged before it ever had a chance. For a young marketplace, trust is most of the product. Testing that erodes it is not a cheap way to find bugs. It is an expensive way to lose users.
If you do have to test in production, the rules matter more than the testers. Agree a load limit, tag every test record so it can be found and deleted in one go, and decide whether test listings are hidden from real users. That is a separate article, but the point here is simple: a payment model that rewards volume is the wrong fit for an environment where volume does damage.
The bugs that cost money are few and boring
When I ask a founder which bugs would actually hurt, the list is short and rarely exciting.
- A new user fills in the signup form, gets no confirmation email, and never comes back.
- A payment goes through on the provider’s side but is never recorded on yours, so the work looks unpaid.
- Two people agree terms, and one of them never sees the agreement.
- The platform slows to a crawl on the one morning a partner sends you a thousand visitors.
- Personal data, an ID document or a phone number, is readable by someone who should not see it.
None of these will win a bug-hunting leaderboard. Several of them do not even show an error. The signup “works” from the tester’s point of view; the email just never arrives. You only find these by walking the journey the way a real user would, end to end, and checking that the business outcome happened, not just that the screen changed.
These are also the bugs that a per-bug tester has the least reason to chase. A full end-to-end journey through signup, matching, agreement and payment takes time, and it might produce zero findings. In a per-bug model, an hour with zero findings is an hour unpaid. So that hour goes to the screens where findings are easy.
Test against business loss first
The way we approach it is to start from the other end. Before anyone opens the app, we ask what has to go wrong for the business to lose money, users or trust, and we write that list down with the product owner.
For the marketplace on that call, the list was roughly: a worker can sign up and complete a profile, a client can post a job, the two can find each other, agree terms, meet, and the payment and completion record end up where they should. If that cycle breaks anywhere, nothing else matters much.
Then we test that cycle first, repeatedly, on the devices and networks the real users have. After that we widen out to performance, data protection and the long tail of cosmetic issues. The cosmetic issues still get logged. They just do not get to set the priority.
This changes what a report looks like. Instead of “we found 340 bugs”, the useful summary is closer to “the signup to first payment journey works on these devices and fails on this one, here is why, and here is what it costs you”. That is a sentence a founder can act on, and it is a sentence an investor might ask to see.
Counting bugs is not a quality metric
I understand why bug counts are tempting. They are easy to measure, easy to compare, and they go up when people are busy. But a rising count can mean the product is getting worse, or the testers are getting more thorough, or the testers are getting better at splitting one issue into five. The number alone cannot tell you which.
A better question for any testing arrangement, crowdsourced, in-house or outsourced, is this: what is the riskiest thing about our product, and has anyone checked it? If the answer is a list of 340 findings and nobody can tell you whether a new user can pay, the money went to the wrong place.
We have written before about how dedicated teams compare with managed crowdtesting, and about writing bug reports that developers actually fix. The short version of both: fewer, better findings, tied to something the business cares about, beat a big pile every time.
Our view
Crowdsourced testing has its place. Wide device coverage for a stable product, with clear scope and nothing at stake in production, can be a good fit. What I would not do is pay per bug on a young, live product where trust is fragile and the important journeys have never been walked end to end.
Decide what failure would cost you. Test that first. Count later, if you still want to.
This is how our functional testing works: https://betterqa.co/functional-testing-services/