Our query-count test could not fail: it counted zero against a ceiling of 25
A ceiling far above the true number hides a counter that reads zero.
BetterFlow has a test whose only job is to catch a page that runs one database query per row. We deleted the code that prevents that, and the test stayed green.
The test that passed with the fix deleted
The test loaded a page of 25 rows and asserted that the number of queries stayed under a ceiling of 25. A page that runs one query per row would run about 25 and fail it.
During a deliberate break-things sweep we removed the eager load that prevents exactly that. The test passed.
Why it counted zero
A profiler sits at the outermost layer of the request pipeline, is on in every environment but production, and clears the query log when each request finishes. The test turned the log on, made a request and counted the log afterwards, which was always empty.
Comparing zero with a ceiling of 25 can never fail, on the one test whose whole job was to see something nothing else in the app can see.
How we found it
The sweep killed 28 of 35 mutants, and the survivors were the deliverable. Deleting the eager load and seeing the test stay green was the clue. The same review found two more tests that proved nothing, one of which used fixtures where a partial match and an exact match returned the same row.
What we changed
The count now comes from a listener that nothing downstream can clear. The test also has a positive control, a check that the counter saw at least one query, and a precondition that the page really carries 25 rows. Any report asserting a query count has to use the listener.
What to take from it
- Break the code on purpose. A test you have never seen fail is a hope.
- Add a positive control. Prove the instrument sees something before you read its silence.
- Be wary of a generous ceiling. It hides a counter that reads nothing.
Want to know your performance tests can fail?
We delete the thing under test and watch the suite. The first month is a trial.
