Testing a phone AI agent: what changed after real calls
Short answer
We tested a phone AI agent in rounds of real test calls. Each round checked how it answers, how fast it answers and what information it has, then we called again. The biggest wins were speed (about 2 s to about 0.4 s to first answer), no silence while tools run, and a human taking calls first to learn what callers ask.
Maria is the inbound phone agent for Arctic Works, a car-service business in Cluj that we are related to. It answers calls in Romanian, switches to English, checks open slots, books, looks up and cancels appointments, and sends a WhatsApp confirmation and an alert to staff. The voice is ElevenLabs and the telephony runs through Twilio and Infobip.
We did not run a formal campaign. We made test calls, the findings were fixed, and we called again. This is what changed.
Rounds of calls, three questions each
Each round of test calls looked at the same three things:
- How the agent answers. Is the reply short, correct and on topic?
- How fast it answers. How long does the caller wait, and where does silence start to feel like a dropped call?
- What information it has. The knowledge base, the schedule data and the tools it can call.
After each round those three things were tuned and we ran the next round of calls. It is a loop. We do not have a pass rate to quote, and we would not trust one.
From about 2 seconds to about 0.4
At first the caller waited about 2 seconds for the first answer. Testing pointed at the slot lookup. Batching it brought time to first answer down to about 0.4 seconds.
A two-second gap at the start of a call sounds like nobody is there.
The agent went quiet while a tool ran
When the agent called a tool, it fell silent until the result came back. Callers heard dead air. The fix was to preload the schedule at call start, add a "never go silent" rule, and give the agent a short line to speak before each tool call.
Two small fixes, found only by listening to calls. A transcript shows no silence.
The booking system said nothing was booked, on every call
The webhook reported "no booking made" on every call. The voice platform sent the parameters in a different field than its documentation suggests. The same fix revealed a second problem: bookings had been saved under a placeholder name. The agent now asks the caller's name.
Other integration surprises were of the same kind: a signature format that did not match, nested payloads, and the caller's number buried in nested fields. Calls under 3 seconds were also filtered out of notifications, because those were ghost calls nobody needed an alert for.
Shorter answers, fewer choices
Answers were too long and offered too many slots. Brevity rules were added: one question per reply and fewer slots offered. A business-hours check was added, and the agent no longer says "our colleagues are busy" during business hours.
One pass over the prompt fixed 20 issues. The code went through 4 rounds of review. One call path is capped at 20 concurrent calls, a limit worth knowing before you plan for load.
The agent was paused so a person could take the calls
The best decision was not technical. The agent was paused for a period so that a person took the calls. That showed what callers ask. Only then were the prompts tuned on real intent.
The agent also speaks only Romanian and English and politely declines any other language. This is a design choice for this business. On any agent, say a sentence in a third language and check what happens.
What we would check on any phone agent
These come from what we saw, not from a standard.
- Silence. Make a call that forces a tool lookup and listen for dead air at every step.
- Time to first answer. Call several times and note how long the start takes. One call tells you little.
- The data it writes. After a test booking, open the booking system and read the record. Is the name, the number and the time right?
- Length. Count how many options it offers in one turn. Fewer options are easier for a caller to hold in mind.
- Out-of-scope callers. Ask something unrelated, speak a language it does not support, and call outside business hours.
- Notifications. Check that staff are alerted for real calls and not for ghost ones.
Then change something and run the calls again. A single round of calls shows you the first set of problems, and fixing them shows you the next set.
Frequently asked questions
Putting a voice agent live?
Tell us who calls and what they ask. In a complimentary call we will outline the first test set.
Book a complimentary call Voice agent and IVR testing
