Our fact-checker kept blocking true statements about ourselves
A guard that stops invented facts can also stop true ones, and then people learn to word around it.
JRNY is our sales CRM. Before an AI-drafted email goes out, a claim check looks for statements about the recipient that nobody has verified. It does its job. It also kept refusing to let us say true things about BetterQA.
What the check is for
AI-written outreach can state things about a company that nobody checked. The claim check compares each proper noun and number in a draft with the evidence on the company record, and refuses to send what it cannot support.
What it refused
- Our own review rating. A sentence saying we are rated well on a review site was blocked, along with the word "EU", a region like "US" and "UK".
- Our own product names. An email to a signed client was refused because it named our tools.
- Words describing our service. A partner follow-up promising to run our scanner on their site was blocked for the words SEO and Google.
- The recipient's own words. A partner's product name, written on their own website and in their own emails, was refused until a person typed in a confirmed fact by hand.
The workaround was to strip every proper noun until the gate passed. That trains senders to paraphrase around the check, which weakens it for the claims it exists to catch.
Patching the edges does not end
In about six weeks, 42 commits touched the one file holding the check, and each fixed one more edge. The root cause, found when we planned a proper fix, was a pattern: our exemption for statements about ourselves matched three exact sentences, so one extra clause broke it.
The check also had no way to say who a sentence was about. A sentence about us and a sentence about the recipient need different evidence, but the check treated them the same.
The fix, in the order we chose
- Warn, do not block. For now, emails go out. The check still runs and lists the words it would have blocked, in the send result and in a log. Blocking comes back once false alarms are rare.
- Decide who the sentence is about first. Sentences that clearly speak as us clear only by restating one list of verified facts, and only warn. Sentences about the reader or unknown subjects keep the strict check.
- Shadow first. The first part of the new design changes no verdicts. It records what it would have decided so we can compare before switching it on.
What to take to your own team
- Count false alarms. A guard people work around is a guard that is off.
- Ask whether you are patching a class. Dozens of commits on one file is the tell.
- Fail closed, but know the cost. Our default stayed blocking. Warning only was a deliberate, reversible decision by a person.
Want a guard that catches the claims that matter?
We test checks on both sides: what they let through and what they wrongly stop. The first month is a trial.
