Urdu IVR bot that handles code-mixed speech

Three ways to see it

1. 1

   An IVR is the voice menu a customer hears when they dial the helpline. Until recently, IVRs in Pakistan were strictly DTMF, the press-1-for-this menus. Modern IVRs use Automatic Speech Recognition to listen to the caller's spoken words and Natural Language Understanding to decide what they meant. For a Pakistani operator the hard part is not ASR for clean Urdu. The hard part is ASR for actual customer speech, which is code-mixed Urdu, Punjabi, Sindhi, Pashto, English, often with a poor signal because the caller is calling because they have a poor signal.

2. 2

   Way one to think about it: code-mixing is the language, not a deviation from it. A trained urban Pakistani speaker switches between Urdu and English mid-sentence without conscious thought. A bot that demands 'speak only Urdu' or 'speak only English' breaks the natural fluency and frustrates the caller. The right design accepts the mix as input and reasons about intent on the meaning, not the surface tokens. Whisper-large-v3, Google Chirp, and the open Urdu-Hindi finetunes from MBZUAI are the current strong baselines.

3. 3

   Way two: regional accent and dialect variation is real and unequal. ASR accuracy on Karachi-Urdu speakers is meaningfully higher than on Mehsud-Pashto-tinted speakers because the training data is unbalanced. If you build an IVR without measuring per-region performance, you will quietly under-serve KP and Balochistan. The fix is targeted data collection: record consented calls from each circle, get them transcribed by native speakers, and finetune. This is operational work, not just research.

The why-tree

Try this with Claude

AI-edge prompt: 'I am the head of customer experience at a Pakistani mobile operator. Design a voice IVR architecture using OpenAI Whisper for ASR, an open-source NLU model for intent, and a fallback to human in under 8 seconds. Cover (1) per-region accuracy measurement plan, (2) consent and recording compliance for PTA and PDP Act 2025, (3) latency budget per turn, (4) escape-hatch UX, (5) the three KPIs the call centre director should track in month one.'