The opening piece in this series named four things language coverage misses, and each got its own piece: code-switching, register, accent, and context. Every one of them ended with the same advice: don't trust the demo, test on your own calls. This piece closes the series the way the Fast to Deploy capstone closed the last one — by turning the argument into something you can run. Here's what a local-fluency test actually looks like.
Why does a voice agent pass the demo and stumble in production?
Because the demo and production are different tests. A demo is usually clean audio, a speaker with a neutral accent, one language at a time, and replies phrased the way the script expects. Production is an 8kHz phone line, background traffic, a regional accent, a switch into English on the product name, and "after payday" instead of a date. Each of the four gaps in this series is invisible under demo conditions and obvious under real ones. The research has shown the same pattern for years: Koenecke et al. (PNAS, 2020) found commercial speech recognisers that looked strong on average misheard one group of speakers almost twice as often as another. An average hides exactly the customers you most need to test for.
What should the test set be built from?
Your own recordings, sampled the way your traffic actually arrives. Pull real calls from the lines the agent will take over — redacted for personal data before anyone outside your team hears them — and stratify them on purpose: by market and region, by language mix, by use case, and by time of day and channel. Then make sure the hard tail is in there and not filtered out as "bad audio": the elderly customer, the provincial accent, the call from a moving vehicle, the customer who answers every question with a question. A test set made only of tidy calls will tell you the agent works on tidy calls. Keep a few hundred calls as a fixed benchmark so you can compare vendors, and versions of the same vendor, against the same yardstick. Voice AI evaluation benchmarks for ASEAN languages covers how public benchmarks fit alongside a set like this — useful for a first filter, never a substitute for your own traffic.
How do you test each of the four dimensions?
Accent. Report recognition accuracy per accent segment, not one blended number. If most of your inbound volume in Vietnam comes from the south, a model tuned on northern speech can post a respectable average and still fail the majority of your callers. Look hardest at word errors on the load-bearing words — names, amounts, product terms — which is the point WER is not enough for business calls makes in detail. The accent piece covers which segments matter most for inbound versus outbound.
Code-switching. Include utterances where the switch lands on the key noun or verb — "Pwede ba i-waive yung late fee?", "Boleh tolong check my bill?" — and check two things: did the agent keep the intent, and did it reply in a mix the customer would recognise, rather than forcing them back into one language? Check the transcript too: mixed-script transcripts that mangle the switched words will quietly break QA and analytics downstream.
Register. This one can't be scored by a script. Have native speakers listen to the agent's side of the call and rate whether the formality fits: the right Thai particles, the right Bahasa "you", the right Vietnamese form of address for the customer's apparent age, po and opo where a Filipino customer would expect them. On outbound, weight the first ten seconds most heavily — that's where register decides whether the customer stays on the line.
Context. Test with the replies your script doesn't expect, then check the record, not just the conversation. When the customer says "habis gajian" or "sa katapusan", is a usable promise-to-pay date stored? When they say "bukas na lang po", is it marked as a likely soft refusal or as a firm callback? The context piece lists the patterns worth seeding into the set.
Is word error rate the right number to measure?
It's the first number, not the last. A useful local-fluency scorecard has four layers, and each catches failures the one below it misses. Recognition: word errors, broken down by accent and language segment. Understanding: whether the intent and the key details — amount, date, product, payment method — were captured correctly. Conversation: how often the customer had to repeat themselves, how often the call escalated, and how long it took. Outcome: whether the result the system logged matches what a human reviewer says the customer actually agreed to. The last layer is the one most evaluations skip and the one the business is judged on. An agent can score well on the first three and still fill your CRM with promises nobody made.
Who should judge the calls?
People who speak the way your customers do — not translators working from transcripts, and not reviewers from another market who share the language on paper. A reviewer from Jakarta will miss what sounds off to a caller in Medan; a fluent second-language speaker will hear a grammatical sentence and miss that it was too blunt. Give reviewers a short rubric per dimension so scores are comparable, and have two people score a slice of calls independently so you know how much the judgement itself varies. That review step doesn't go away after launch: it becomes the human review loop that catches what automated checks can't. QM then carries the same rubric across every live call, so the sample your reviewers check is chosen by risk rather than at random.
What happens to local fluency after go-live?
It drifts, because the market does. A new e-wallet takes off, a product gets a nickname, a holiday moves, a campaign starts reaching a region your test set never covered. Fluency that was measured once at launch quietly decays from there. The fix is to keep the test set alive: add the calls that failed in production every week, rerun the benchmark on every model or scenario change, and watch the per-segment numbers rather than the average. That's the same discipline the AI ops loop applies to deployment changes, and agent observability is what makes the failing calls easy to find. It's also where this series hands over to the next one. Testing tells you the agent is fluent on a thousand calls. Keeping it fluent across a million is a reliability problem — and that's what the next series, Reliable at Scale, takes on.
How does WIZ.AI support this kind of testing?
The platform is built to be tested on your traffic, not ours. Language Engine runs on the conditions the test set is designed to reproduce — 8kHz telephony, mid-call language switching, regional accents — and its scenario customisation is where findings from the test get fixed: pronunciation and recognition for your product names and places, intents for the replies your customers actually give, and the context rules that turn "after payday" into a usable date. Those changes apply under both deployment shapes: Wizlynn for inbound service and Talkbot for outbound campaigns. QM scores live calls against the same dimensions after go-live, and WIZ.AI's regional industry corpus means the starting point is speech from the markets you serve rather than a generic multilingual base.
What should you ask a vendor before you sign?
Three questions, one for each stage of the test. First: will you run the evaluation on our redacted recordings before contract, and report results per accent and language segment rather than as one average? Second: will you show us the logged outcome next to the transcript for every test call, so we can check what the system recorded, not just what it heard? Third — the question every piece in this series has ended on — when the test or production finds a gap, who fixes it and how fast: our team on the platform, or a change request back to you? A vendor that is confident about local fluency should welcome all three.
Coverage is a list on a slide. Local fluency is a result on your own calls — and the only way to know you have it is to test for it. See how scenario tuning and live call scoring work on the Language Engine and QM pages, or book a demo and bring a set of your own recordings to test against.
References
- Racial disparities in automated speech recognition — Koenecke et al., PNAS, 2020
- SeaEval for Multilingual Foundation Models: From Cross-Lingual Alignment to Cultural Reasoning — Wang et al., NAACL 2024
- SEA-LION: Southeast Asian Languages in One Network — AI Singapore, 2025
- Locally Fluent: Why Language Coverage Isn't Local Fluency — WIZ.AI
- Locally Fluent: Code-Switching Is the Call, Not the Edge Case — WIZ.AI
- Locally Fluent: Register Is Not Optional — WIZ.AI
- Locally Fluent: The Accent Is the Test, Not the Exception — WIZ.AI
- Locally Fluent: Context Is What the Script Can't Translate — WIZ.AI
- ASR Evaluation: WER Is Not Enough for Business Calls — WIZ.AI
- Voice AI Evaluation Benchmarks for ASEAN Languages — WIZ.AI
- Language Engine: mid-call language switching and scenario customisation — WIZ.AI
- QM: compliance-driven quality monitoring for every call — WIZ.AI
Related insights
Book Demo