What breaks when AI agents use MCP servers: lessons from running our own
Short answer
MCP servers rarely crash. They answer, and the answer is wrong, empty or from the wrong place. In our own journal of 221 entries across 13 tools, the most common failures were acting on the wrong project, reporting success while doing nothing, and reading a capped list as a total. Test what changed, not what the tool said.
We build MCP servers for our own tools and our AI agents use them every day: BugBoard, Flows, JRNY, Hireo, a site scanner, the AI Security Toolkit, Maestro and others. Because we are the first users, we write down every surprise. After a while the journal became the best test plan we own.
Here is what it taught us, with the failure classes in the order they came up most often.
Why we keep a journal of our own tools
Every time an agent used one of our tools and something surprised us, we wrote it down. The count so far is 221 entries across 13 tools. From those entries we filed 819 GitHub issues: 697 are closed and 122 are open.
That count is the cost of putting an agent in front of a tool. A person who gets a vague error stops and asks. An agent carries on with whatever it was given, and builds the next step on top of it.
Wrong project, and nobody told the agent
Several of our tools hold one key per project. With more than one key in play, a tool sometimes acted on the wrong project without saying so. A test suite was created in the default project instead of the intended one. Reads crossed project boundaries while writes did not, so the data looked right on the way in and wrong on the way out.
The test that finds this is dull and effective: use two projects with different data, run the same call with each key, and check where every write lands and which records every read returns.
The tool said success and did nothing
A tool answered "Updated" while dropping the fields. A re-read showed nothing had changed. Security scanners reported success while scanning nothing, because one interpreter they depended on was dead. In our mobile test flows, AI assertions defaulted to optional, so a failed check still passed. That is now refused when the flow is created.
We worry about this class most: an agent cannot tell a real success from an empty one unless the tool lets it. Every write test should read the record back afterwards and compare it with what was sent.
A capped list read as a total
A list capped at 100 was read as the total. A "real zero" was reported while 23 items existed, because a default filter hid them. A health check reported 0 dead links when it had stopped at its cap. One result read "18/14 passed (128.6%)".
A person sees a number like that and frowns. An agent writes it into a status report. Test every count against a number you know: seed 101 records and check that the tool does not say 100.
The server claims one version and runs another
A server reported a current version while running month-old code. In another case a fix landed but a restart did not pick it up, and two probes gave a false green. If you test the reported version, you test a label.
After any fix, check the behaviour the fix changed, on the running server, not the version string.
The rest of the list
- Schema versus behaviour. A field required at runtime was not marked required. A tool advertised a status the database rejects. A number arrived as a string.
- Opaque errors. A bare "Failed", raw database codes, and a remediation hint that pointed the wrong way.
- AI text stored as fact. Email auto-summaries inverted the meaning of the email and were saved to the CRM, once attached to the wrong email. Enrichment attached contacts from a different company with the same name.
- Sensitive data in tool output. A scan in parse-only mode echoed plaintext credentials from request bodies back into the agent's context.
- Output too large. One call overflowed the agent's context window.
- False liveness. Status said "connected" while nothing listened, or "not connected" for a healthy extension.
One thing we did not find. We found no prompt injection arriving through tool output in our journals. We still test for it, because absence in our journal is not absence in your server.
A short test list for your own MCP server
- Run every write with two projects, accounts or tenants, and check where it lands.
- Read back after every write and compare with what you sent.
- Seed more records than any cap and check totals, filters and "zero" results.
- Check the behaviour a fix changed on the running server.
- Send bad input and read the error as the agent would. Is it clear enough to act on?
- Look at what a tool returns for secrets, size and meaning, as well as whether it ran.
Frequently asked questions
Building an agent or an MCP server?
Tell us which tools it can call. In a complimentary call we will outline what to test first.
Book a complimentary call AI agent and MCP testing
