Your prompt says never. How often does the AI do it anyway?
For Founders, CTOs and heads of customer teams whose AI drafts messages that a customer will read.
JRNY, our sales CRM, drafts outreach emails and replies with a language model. We measured it and found it used a phrase it had been told never to use in roughly one draft in 12 to 20, and production had no check for it at all.
- AI drafting
- Our own sales CRM
- Facts
- Claims
- Rules
- Actions
What was at risk
A meeting that never happened
The model started writing relative-time lines such as references to yesterday. A guard had to forbid inventing a date, a call, an invite or a meeting not in the input.
A credential upgraded
Asked to rewrite a real credential, a basic ordering agreement, the model drifts toward a stronger phrase the company does not hold.
A banned phrase used anyway
The model broke its own banned-phrase list in roughly one draft in 12 to 20.
A draft that goes out unapproved
The database default, the normaliser fallback, the save function and the setup screen all chose automatic sending.
Five questions we ask of any AI that drafts for your customers
- CheckWhich of your prompt rules also has a check that does not rely on the model?
- CountHave you counted how often the model breaks each rule, or only assumed it obeys?
- DefaultDoes every default fail to human approval?
- EmptyWhat happens when the model returns its working notes and no email?
- DateDoes the model know today's date, or does it invent one?
How we found it
- Where
- JRNY, our own sales CRM, which drafts outreach emails and replies with a language model. A prompt eval of 22 cases.
- How
- We measured, because we had only assumed. In a handful of eval runs we counted how often the model used a phrase it had been told never to use.
- What counted
- A prompt is an instruction the model follows most of the time. The send path needs a check that does not.
- How we workedWe tested the credential wordingWhere it showed upWe took the credential facts out of the writing prompt and wrote prohibitions instead, with a test that fails on the wrong wording. The test failed against the old prompt and passes now.
What we found, and what it costs
- Finding1 of 4Banned phrases: used in roughly one draft in 12 to 20What it costs youProduction had no check for it at all, so nothing would have caught it.
- Finding2 of 4Working notes queued as the draft when the model produced no emailWhat it costs youIn one version, 6 of the 22 eval cases produced notes and no email.
- Finding3 of 4Four places that all defaulted to sending without approvalWhat it costs youA draft could go out with no human having approved it.
- Finding4 of 4An invented meeting, and a stronger credential than the company holdsWhat it costs youBoth are statements a customer would read as fact.
The model was never told today's date, which is why it started writing relative-time lines.
What changed
The date is now passed in. One generated phrase list feeds both the eval and a check on every draft, and a draft that breaks it shows a red badge naming the phrase on the approval screen. It flags; it never rewrites silently. Every default now fails closed to approval, and a model answer with notes and no email is a failed generation.
The longer write-up: why a prompt is a request, not a control.
Want your AI drafts checked before they reach a customer?
We measure what the model does, and add the checks the prompt cannot give you. The first month is a trial.
