Why your AI writer still invents facts after you told it not to
Short answer
A prompt is a request, not a control. The model follows it most of the time and drifts the rest. We measured a banned phrase leaking into about 1 email in 12 to 20. What worked was a deterministic check on the send path, with a person approving last. Test the check, not only the prompt.
We use language models to write real messages. JRNY, our sales CRM, drafts outreach emails and replies and summarises inbound mail. Azorel, our Slack and chat agent, drafts posts, replies and emails. In both, we told the model exactly what to do and what never to say.
It mostly did. This article is about the part where it did not, and what we changed.
The prompt asked, and the model drifted
JRNY had a banned-phrase rule: phrases the model must never use in outreach. We measured how well the model kept it. It ignored its own rule in about 1 email in 12 to 20, which was 2 to 3 leaks in 39 runs. In production there was no check at all, so nobody would have seen it.
A rule in the prompt lowers the chance of a failure. It does not remove it, and the chance is not zero just because you wrote it in capitals.
What drift looked like in practice
- A number became a claim. "50+ engineers" turned into "we helped 50+ clients" in a Romanian cold pitch. It sounds close, and it is a different claim.
- A meeting that never happened. A draft referred to a call that was never held. The model had not been given today's date, so it invented a past conversation to fit.
- A credential reworded. The model paraphrased a real supplier arrangement with NATO into "NATO approved", which we do not claim and must not.
- The wrong name. The model derived first names itself and produced regional variants. The name is now passed in as written.
- Promises nobody made. Reply drafts said "someone from the team will reach out", or recalled events the thread never held.
- Wrong register. The Romanian welcome came out in the informal "tu", and a duplicated "Subject:" line appeared in the body in 2 of 3 live runs.
Each of these reads fine at a glance, and each went out under our name.
A deterministic check on the send path
For each failure we added a check that does not depend on the model behaving. A banned-phrase check now runs on every draft and flags the result on the approval screen. A facts line gives the model the true numbers, and a backstop check catches a claim that slips past it. For unapproved commitments there is a check in Romanian and English, plus grounding of names against the thread.
We wrote the tests red first: each test failed on the old behaviour before we fixed it. For the credential wording the test fails on the bad phrase, so the paraphrase cannot come back quietly.
A deterministic check is blunt and it is meant to be. It does not understand the message. It only knows what must never leave.
Every default chose to send automatically
The most useful thing we found was not in the model at all. Every default in the pipeline chose automatic sending. We changed them all to fail closed to human approval, and all live campaigns are now on approval.
Related: model reasoning with no message was once queued as the draft. It is now a failed generation, shown as one. And inbound email text is marked as untrusted before it reaches the classifier, so a message cannot give the model instructions.
Human approval stays last
The check is a net. A person still reads the draft, sees the flags and decides. Azorel works the same way: comment text fed to the model is marked untrusted and capped in length, and human approval stays the last gate.
We also run an evaluation set of 22 cases on every prompt version, with deterministic checks and an LLM judge. The prompt went through many versions this year, from v13 to v16.9, and each one is re-measured before it is trusted.
When the AI reads instead of writes
Writing is only half the use. JRNY also summarises inbound email. Those auto-summaries inverted the meaning of an email, and were saved to the CRM as if they were fact. Once a summary was attached to the wrong email. Enrichment attached contacts from a different company with the same name.
A wrong draft is stopped by a person before it goes out. A wrong summary is different: it is stored, and the next person trusts it. So the rule for stored AI text is stricter. Label it as machine-written, and never let it overwrite something a person entered.
A short test list for any AI that writes in your name
- Give the model a case with the date missing, and see whether it invents one.
- Give it a true number that is easy to turn into a bigger claim, and read the output.
- Give it a real credential or agreement, and check the wording does not drift toward something stronger.
- Run the cases that matter more than once. One pass proves little when the output varies.
- Check what happens when generation fails: the failure should be visible, not queued as a draft.
- Look at the defaults. If nothing is specified, the safe choice is a person approving.
These cases test the whole path from prompt to send, where the harm happens.
Frequently asked questions
Does your AI write in your name?
Send us a few real drafts. In a complimentary call we will show how we would test them.
Book a complimentary call Chatbot testing
