Our test suite stayed green after we deleted the thing it watched

The symptom nobody could see

Delete a success counter and the nightly line reads "N processed, 0 success, 0 skipped, 0 failed" while every fetch worked. It looks like a quiet night. No test would fail, because the tests only covered failures.

How the gap got there

One fix repaired BugBoard's success counter. A second change restored the one the first fix had deleted. Twenty-one minutes later a third change shipped the GitHub and Jira halves with five test fixtures, every one a failure. So both sibling counters could be deleted with the whole suite green.

The positive control

The follow-up added one successful-response test per provider and set the bar: each new guard has to be seen turning red against its own deleted line, or it is not worth merging.

The review found the review's own gaps

An adversarial reviewer reproduced the proof exactly, then found four things wrong with what it claimed.

• The skip counter was still deletable with all 174 tests across two test folders green, because every assertion read "0 skipped".
• Every fixture had one project and one user. So "add one" and "set to one" looked identical, and so did "counts users" and "counts projects". A second member separates them.
• A comment called a fixture "a quiet day". It is also, indistinguishably, a Jira integration that is completely down: all four sub-fetchers catch errors and return empty lists, so the parent reports success either way. The comment now says so.

What to take to your own team

• Break it on purpose. Delete the line the test claims to guard and watch it go red.
• Vary the fixture. A test with one of everything cannot tell "one" from "all".
• Ask what the check reports when the thing it watches is dead. If that equals success, you have a green light on a dead system.