Evaluating an AI receptionist: eight calls and a scorecard
A good voice wins the demo. What decides the purchase is what lands in your calendar. Put the same eight calls to every provider, read the record behind each one, and score what actually happened.
AI receptionist scorecard
Download the scorecard (CSV)| # | Call | Look for | Score |
|---|---|---|---|
| 01 | Known answer“What time do you open on Saturday?” | The answer matches your sheet, including its limits, and the agent can point to the source. | |
| 02 | Unknown answer“Do you offer payment plans?” | The agent says it does not have that answer and offers your agreed next step. Nothing is invented. | |
| 03 | Correction“Tuesday morning, please. Actually, make that Wednesday, after two.” | The readback, the booking and the confirmation all carry the corrected detail. The first version never reaches your system. | |
| 04 | Interruption“Sorry, no, it’s forty-two Alder, not forty.” | The agent stops, takes the correction and keeps the request without starting over. | |
| 05 | Completed action“That works. Please book it.” | Exactly one correct record exists in your system, and the spoken confirmation matches it. | |
| 06 | System failure“Can you book me in for Thursday at ten?” | The caller hears the real result and the next option. Nothing is called booked that is not, and a retry never creates a second booking. | |
| 07 | Human handoff“Can I speak to someone, please?” | Your person hears a brief before the call connects. A decline brings the caller back to the agent with a next step. | |
| 08 | Repeat request“Hi, I called yesterday about my booking.” | The caller is recognized, no duplicate is created, and they hear only what they are allowed to hear. |
Score 0 = wrong outcome 1 = needs repair 2 = evidence matches
Test the work, not the voice
Every provider sounds good in its own demo. The demo runs its own script on its own material, and it ends before anyone opens a calendar. A fair test turns that around: your workflow, your material, your test system, the same for everyone.
Pick one workflow you can check from end to end: booking an appointment, moving one, opening a service request. Hand every provider the same pack: the information sheet the agent answers from, the details a booking needs, who takes over and when, and a test calendar or sandbox you can open afterwards. Name two people: one to judge the conversation, one to check the record.
Write down what you tested: the agent’s version, the voice, the language and the systems it was connected to. Run each provider twice, once as delivered and once after the tuning they would do for you, and keep the two sheets apart. The scorecard CSV carries every instruction, so the person running the calls doesn’t need this page open.
Put the same eight calls to every provider
Each call on the scorecard tests one thing a front desk does every day. Set the calls up by what they need, and run every one more than once: keep each recording, and change only the detail the call is testing.
Calls 01 · 02 Your information sheet
Ask one question the sheet answers, and ask where the answer came from. Then ask for a price or a policy the sheet doesn’t cover.1
- 1Coverage checkPut your customers’ questions to the knowledge before launch and see which ones it can’t answer.Knowledge
Calls 03 · 04 A correction
Change a date, a name or an address before you agree to the booking. Then cut in with a correction while the agent is still talking, on the phone setup your callers really use.
Calls 05 · 06 Your test system
Ask for a booking or an update in your test system, and agree when the agent reads it back. Then run it on test records twice more: once with your system refusing, once with it not answering in time.2
- 2Error mappingEach error your system returns is mapped in the action itself: its status, whether to retry, and the sentence the agent says.Handoff to your systems
Call 07 A person to transfer to
Ask for a person twice: once with someone who accepts, once with someone who declines or doesn’t answer.3
- 3Transfer outcomesEvery transfer ends in one of seven typed outcomes, read from the carrier’s own hangup cause.Transfers and handoff
Call 08 The same number
Call back about the same request from the same number, and pass the identity check you agreed on.
Swap the booking for your own work and the eight calls stay the same:
Read the record behind each call
What the agent said is half the evidence. The other half is what it did: which lookup it ran, what it sent your calendar, what came back, and whether the words it spoke match. Ask each provider to open that record in front of you, for every one of the eight calls.
Transcript
- Caller
Hi, it’s Jamie Rowan. I need to move my appointment to Wednesday, after two if you can.
Find openingshttp · find_openings
- sent
- date 2026-09-30 · after 14:00
- returned
- 2:30 PM · 3:15 PM
- Agent
Wednesday I have 2:30 or 3:15. Which suits you?
- Caller
Two-thirty.
- Agent
Moving your appointment to Wednesday, September 30 at 2:30 PM. Shall I confirm it?
- Caller
Yes, please.
Move appointmenthttp · move_appointment
- sent
- APT-20417 → 2026-09-30 14:30
- returned
- 200 · updated
- Agent
Done. You’re booked for Wednesday at 2:30, and the confirmation is on its way by text. Moved
Session details
- Session
- Today, 10:12 AM · Phone · 03:48
- Agent
- Front desk
- Session overview
- Moved Jamie Rowan’s appointment (APT-20417) to Wednesday, September 30 at 2:30 PM after the caller confirmed. Confirmation texted.
- Outcome
- Resolved
- Cost
- 03:48 × $0.156 = $0.59
Check three things against it:
- the booking in your test calendar is the one in the record;
- the confirmation the caller heard matches both;
- nothing ran that the caller didn’t agree to.
A provider that can show you this for a test call will show it to you for every real one. That is the record your team works from on a Monday morning, and the one you open when a customer says “but the agent told me”.
Score it the same way every time
Score each call against the record, not the impression it left. Three rules keep one provider’s total comparable with the next:
- a blank is an untested call, neither a pass nor a zero;
- where you ran variants, keep the lowest score;
- compare totals out of 16 only once every call has been run, as a way to compare, not a bar to clear.
The scorecard shouldn’t retire the day you sign. In Zumu, the same kind of checks run on every conversation after launch,4 each verdict citing the turn it came from, and a check you correct learns from the correction.
- 4Quality checksJudged by a model, by a rule or by a number, grouped into packs you switch on per agent. Repeated failures cluster, so one fix covers the pattern.Conversation records
Ask what running it will take
The demo is one afternoon. The agent will run for years, through new prices, new staff and new systems. Ask every provider the same questions about those years. Here are Zumu’s answers.
- Who keeps the knowledge current?
- You do: edit a document or a page on your site, and the agent’s knowledge re-syncs on the schedule you set.
- Can we undo a change?
- Every edit is a revision you can restore.
- Can we try a change on some callers first?
- Branch the agent and split traffic to test the change against the original.
- Can we test before a customer hears it?
- Scripted test conversations with mocked tools run before you publish. The handoff guide shows one.
- What does a minute cost?
- $0.156 per AI-handled minute on phone calls, with no seat fees, and every conversation shows its itemized cost. The rate card is public.
- What happens when a person takes over?
- AI billing stops.
Then put every provider on your list through the same eight calls, Zumu included. The scorecard decides from there.
Further reading
- Compare: Zumu vs ElevenLabsKeep the voice you chose, and run the whole front line on Zumu.
- Compare: Zumu vs Intercom FinYour inbox and your phone line on one agent, at published rates.
- Guide: After-hours call handlingA plan for every call that comes in after you close.
- Guide: Handoff to your systemsThe action contract behind calls 05 and 06, error by error.
