Mid-utterance code-switching — changing language inside a single sentence — is the most common and the hardest form of language mixing in Southeast Asian customer calls. "Saya mau cancel order yang kemarin." "Pwede po bang i-check yung balance ko?" "我要 check 一下 my bill." For speech recognition, each switch point is a moment where the model has to change its expectations about sounds, words, and grammar. This piece looks at how contact center ASR handles that, technically, and how to evaluate it.

Why are switch points where errors cluster?

Because the model's predictions are strongest when the next word is in the same language as the last. At a switch, the language model's expectations are wrong, and the acoustic model has to recognize sounds from the other language — often adapted to the speaker's first-language pronunciation. Errors tend to cluster on the words around the switch. In customer service, those are frequently the product, the action, or the amount, which is why switch-point accuracy matters more than overall error rate.

How do modern systems recognize mixed speech?

Mainly with joint multilingual models that recognize both languages in one pass, with a shared vocabulary covering both. This avoids the delay and errors of choosing a language first. Bilingual language modelling learns how the languages mix — which kinds of words tend to be switched, and where. Contextual biasing boosts expected switched terms — product names, actions, payment methods — so the recognizer prefers them when it hears something close. Large multilingual models like MMS provide a strong base; adaptation on code-switched call data does most of the work for specific language pairs.

Where does code-switched training data come from?

It's scarce. A systematic survey of code-switching research found limited resources for most language pairs, with a few exceptions such as the SEAME corpus for Mandarin-English in Singapore and Malaysia. In practice, the best data is transcribed customer calls from the same market and domain. Teams also generate synthetic code-switched text to improve language modelling, and splice or synthesize audio to augment training — useful, but no substitute for real speech.

How should switched words be written in transcripts?

Consistently. Decide whether switched English words are written in Latin script or the local script (relevant for Thai, Chinese, and other non-Latin languages), how affixed forms like "i-cancel" or "di-refund" are spelled, and how product names are written. Consistency matters for intent detection, QA rules, search, and analytics. Mixed-script transcripts covers the conventions.

How should mid-utterance code-switching be evaluated?

With a dedicated test set of real code-switched turns, labeled with language tags per word or segment. Measure overall error rate, but also error rate on switched segments and on words adjacent to switch points. For pairs that mix character-based and word-based writing, such as Mandarin-English, researchers use a mixed error rate that counts characters for one language and words for the other. Above all, measure whether the intent and key entities were captured correctly. A system can have acceptable overall error rates and still miss the switched word in most sentences.

What should buyers ask vendors?

Which language pairs the system recognizes jointly, not just which languages it supports. Error rates on code-switched test data, not just monolingual benchmarks. How switched words are written in transcripts. How quickly custom switched vocabulary can be added. And, ideally, results on a sample of your own code-switched calls. Code-Switching Is the Call, Not the Edge Case covers the business side.

How does WIZ.AI handle mid-utterance code-switching?

Language Engine recognizes major Southeast Asian language pairs jointly — including Indonesian, Thai, Tagalog, Vietnamese, Malay, and Mandarin with English — adapted on code-switched customer-operations speech, with contextual biasing for enterprise vocabulary and consistent transcript conventions. It runs in real time in Wizlynn and Talkbot.

The switch point is where generic ASR breaks and where your customer's meaning lives. Book a demo and test it on your mixed-language calls.