Our scheduled job ran green 60,611 times and sent zero emails
Nothing failed, so nothing alerted. Healthy output and broken output were the same bytes.
Hireo is our recruiting platform. Its welcome-email sequence enrolled every new signup for about seven months and never sent anyone anything. The job that was meant to send them reported success every single time.
Every run was green
The scheduler woke up on time, looked for emails to send and answered that it had processed zero. Zero is a valid answer, so nothing failed and nothing alerted.
Meanwhile every new signup was being enrolled in the welcome sequence, and every enrolment sat at step zero. The queue of outgoing emails held the same five rows from one internal test, the whole time.
The cause was one line
The database function that queues the emails checked whether it had found its template with a test that reads as "a row exists". On a whole row, that test actually asks whether every column in the row is filled in.
All five welcome templates had an empty optional column, so the check was false for each of them and the loop that queues emails never ran. The step that enrols the person sat above the loop, which is why enrolments piled up and emails did not.
Fixing it was the risky part
Repairing the function alone would have sent five out-of-date messages, written before a rebrand, to the next person who signed up. So the fix was one migration with three parts: switch the trigger off, repair the function anyway so it is not a trap for whoever turns it back on, and cancel the stranded enrolments so nobody from February gets a surprise.
Turning a welcome sequence back on is a product decision. It should not be a side effect of a bug fix, so we left that call to a person.
The documents said it worked
Three documents in the repository opened with "fully implemented" or "production ready". We measured them. One described this sequence. One described onboarding analytics whose event table had no rows and no code that writes to it. One described A/B testing with every experiment still in draft and no user ever assigned.
We deleted all three. An absent document invites a look. A confident one closes the question.
The same flaw in four more places
We searched the codebase for the same whole-row test. It appeared in four more places, in three other functions that check whether something already exists. Those touch live features, and a wrong answer there creates duplicates instead of reusing records, so we reported them and measured before changing anything.
What to take to your own team
- Monitor the effect, not the run. "The job ran" is not "the emails went out". Count outcomes.
- Ask what the job reports when it does nothing. If that looks the same as success, add a check that can tell them apart.
- Treat "complete" documents as claims. Measure the table, not the title.
- Search for the class. One wrong test usually has siblings.
Want someone to check that your jobs do what they report?
We test the effect of scheduled work, not just whether it ran. The first month is a trial.
