Our upload page refused every visitor for about two weeks and no alert fired
A service that says no politely looks healthy. An alert on crashes watches the run, not the effect.
Hireo is our recruiting platform. Its public page for uploading a CV started refusing every legitimate visitor after one setting was typed in the wrong format. Our error capture, which only sees thrown errors, had nothing to report, and the first we heard came from a person outside the team.
One setting, typed the way the guide said
The upload function checks which websites may call it, by comparing bare hostnames. The migration guide in our own repository told operators to enter the allowed sites with a web prefix, so no comparison could ever match.
When we probed it afterwards with three legitimate origins, all three were refused. We have no log that shows the start date, so the length is our estimate: about two weeks.
Why nothing alerted
The refusal paths returned a response on purpose instead of throwing an error. Our error capture only sees thrown errors, so the function looked perfectly healthy while it said no to everyone.
Two more faults made it worse. The origin check ran after the one-time token was claimed, so a refused first attempt used the token up and the retry came back with a different message that looked like an unrelated bug. And the page matched only one word in the reply and showed a generic message for everything else.
How we found out
A person outside the team told us that every CV submission failed. No monitor found it. That is the finding.
What we changed
The setting was put back to bare hostnames. The origin and rate-limit checks now run before the token is claimed. Every silent refusal branch now reports itself, and the fix makes the origin-rejected case raise an error. The page shows the server's own message, and the guide was corrected. Probes before and after showed legitimate origins going from refused to accepted while a hostile origin stayed refused, and the token was no longer burned.
We have not added an alert on zero successful uploads. That would watch the effect, and it is the next step.
What to take from it
- Make every refusal loud. A deliberate no is still a failure if you did not intend it.
- Do not burn a one-time token before you validate. The retry then hides the cause.
- Watch the effect. Uploads arriving is the real health check.
Want to know whether your service can say no without anyone noticing?
We test the refusals as well as the successes. The first month is a trial.
