How we score voicemail transcription quality, and what we cut

Introduction

Voicemail transcription sounds like a solved problem. Several large ASR models can transcribe clean studio audio at near-human accuracy. The problem is that voicemail audio is almost never clean. Callers call from cars, hallways, construction sites, and speakerphones. Accents vary widely. Proper nouns — names, companies, addresses, product codes — are exactly the tokens that matter most and the ones models handle worst. This is what our eval harness is designed to surface before a model ever touches production traffic.

Why word error rate alone isn't enough

Word Error Rate (WER) is the standard academic metric for transcription quality. It measures what fraction of words in a transcript don't match the reference. The problem with using WER as your only gate is that errors are not uniformly costly. A transcript that gets 'um' and 'uh' wrong costs you nothing. A transcript that gets the callback number wrong costs you the job.

We run WER as a baseline check, but our gating metric is what we call Entity-Critical Error Rate (ECER): the fraction of named entities — phone numbers, addresses, personal names, company names, dollar amounts — that are transcribed incorrectly. A model that looks acceptable on WER can fail badly on ECER, particularly for accented speech and proper nouns outside the training distribution.

The eval dataset

Our eval set is built from real voicemail audio with customer consent and full anonymisation before it enters the pipeline. We maintain roughly 4,000 samples stratified across five noise environments (quiet indoor, car, outdoor ambient, background music, speakerphone reverb), three accent clusters (General American, Southern American, non-native English), and four message types (appointment request, complaint, callback request, general inquiry).

Each sample has a human-verified transcript that was reviewed by two annotators with disagreements resolved by a third. The annotation protocol specifies that proper nouns must be transcribed as spoken, not corrected to the most likely intended entity — if a caller says 'Jonson' we write 'Jonson', not 'Johnson'. Models that silently normalise proper nouns look better on standard eval sets and fail in production exactly where it matters most.

What we test and how

• WER on full transcript — baseline gate, must pass before any other evaluation runs
• ECER on named entities — gating metric; any regression here blocks a candidate model
• Latency at p95 — transcription should complete within 8 seconds of voicemail end for real-time notification delivery
• Degradation under noise — WER delta between quiet and noisy samples; models that degrade non-linearly under moderate noise fail this gate
• Proper noun recall — specifically tested against a list of 500 common business names, street types, and phone number patterns
• Hallucination rate — fraction of transcripts that include content not present in the audio; critical because a hallucinated callback number is worse than no transcript

What we cut and why

We evaluated four candidate models in the last cycle. Two were cut at the ECER gate — both were strong on WER but performed poorly on proper nouns in the non-native accent cluster, which represents a significant share of our customer base. One was cut on hallucination rate: it produced confident, grammatically correct transcripts that occasionally included words not present in the audio at all. The fourth passed all gates and is running in production.

The human-in-the-loop escape valve

When a transcript scores below our confidence threshold on any segment, we flag it for human review rather than delivering it with a misleading confidence indicator. Customers see a 'low confidence' badge on flagged transcripts. In our current production model, roughly 3% of transcripts are flagged, and of those, about 60% are corrected by human review before delivery. The alternative — delivering all transcripts without confidence signals — has a lower support ticket rate right up until a customer loses a job because we got a callback number wrong.

What's next

The area with the largest remaining gap is speaker diarisation on multi-party voicemails — messages where a caller hands the phone to a colleague or partner to add additional context. Current models handle this poorly, and the downstream effect on CRM logging is significant. We're running a separate eval cycle on diarisation candidates now, with the goal of shipping an improved version in Q2.