ASR · Phase 8: Indian languages and accents
Vistaar
Vistaar: Diverse Benchmarks and Training Sets for Indian Language ASR
In one sentence
Vistaar gathers 59 Indian-language ASR benchmarks (language × domain combinations), evaluates public and commercial systems on all of them, and shows that Whisper fine-tuned on ~10.7k hours of public Indic data (IndicWhisper) is best on 39 of the 59, with an average WER reduction of 4.1.
Why this paper exists
Indian-language ASR results used to be reported on one or two test sets, usually clean read speech. That hides how models behave on what users actually produce: news, phone calls, spontaneous talk, noisy rural recordings, different accents of the same language. Two problems followed:
- No fair comparison. Each paper picked its own test set, so you could not tell which model was better for your use case.
- Off-the-shelf models looked better than they were. Whisper (Whisper) and commercial APIs are tuned on mostly high-resource data. On many Indian languages they do poorly, but there was no broad scoreboard to show it.
Public training data also existed but was scattered across many sources, with no standard "train on all of this" recipe.
The core idea
Evaluate broadly, train on everything public. One benchmark suite spanning many datasets and domains per language shows real-world gaps; one combined public training set shows how far you get by fine-tuning a strong pretrained model (Whisper) on Indian data.
How it works
1. The benchmark suite
- Collects existing test sets across 12 languages, giving 59 benchmarks in all (one per language-dataset pair).
- Sources span domains and styles. Datasets in the mix include Kathbath (IndicSUPERB, read speech on phones), FLEURS and Common Voice (read crowd-sourced speech), IndicTTS (studio speech), MUCS (challenge sets), and Gramvaani (noisy rural phone recordings in Hindi). Check the paper's tables for the complete list.
- Systems evaluated: 3 public models and 2 commercial APIs (the abstract's count); check the paper for which ones. Public ones include Whisper and IndicWav2Vec-style models (IndicWav2Vec).
2. The training set
- ~10.7k hours of public labelled data across the 12 languages, combining existing corpora (for example Shrutilipi, a large set of news-bulletin audio with transcripts mined from All India Radio, plus Kathbath train, and others; check the paper).
- Test sets are kept separate from training data so benchmarks are not contaminated.
3. IndicWhisper
Whisper (multilingual, pretrained by OpenAI)
|
fine-tune on Vistaar train (~10.7k h, 12 langs)
(one model per language; check the paper)
|
IndicWhisper
|
evaluate on 59 benchmarks -> WER per cell- Same architecture as Whisper: Transformer encoder-decoder on 30 s log-mel windows, trained with cross-entropy on the transcript tokens.
- Input: 80-channel log-mel features (128 for large-v3) at 100 frames/s, downsampled 2× by the encoder's conv front end to 50 frames/s.
4. The metric
WER = (S + D + I) / NS,D,I: substituted, deleted and inserted words when aligning hypothesis to reference.N: number of words in the reference.- For Indian languages, text normalization before scoring matters a lot: punctuation, numerals, nukta and other script variants can change WER substantially. Vistaar publishes its evaluation code so everyone normalizes the same way.
Build the intuition
Analogy: a cricket player's average on one home pitch tells you little. Vistaar is a full tour: home and away, flat and green pitches, day and night. A batter who averages 60 at home and 15 abroad is not a 60-average batter.
Why does off-the-shelf Whisper struggle? Think about its training mix: 680k hours of web audio, overwhelmingly English and other high-resource languages. Many Indian languages got a tiny slice, often with noisy transcripts. Its BPE tokenizer was built mostly for Latin script, so Indic scripts split into many byte-level tokens, making each word long to decode and easy to misspell. Fine-tuning on ~10k hours of real Indic speech fixes the acoustic and language model side; it cannot fix the tokenizer, which is a reason later models trained Indic-specific tokenizers.
What would happen if you only tested on Kathbath? A model trained on read speech would look excellent; on Gramvaani's noisy phone audio, the same model's WER can be several times higher. The 59-cell grid exposes exactly that.
Tiny example: reference "मेरा नाम राहुल है" (4 words), hypothesis "मेरा नाम राहुल" (one deletion). WER = 1/4 = 25%. If the hypothesis instead wrote a common spelling variant of one word, an unnormalized scorer counts a substitution; that is why normalization rules matter.
Numbers worth remembering
- 59 benchmarks, 12 languages.
- ~10.7k hours of public training data.
- IndicWhisper: lowest WER on 39 of 59 benchmarks, average 4.1 WER reduction (vs the best other system per benchmark; check the paper for the exact comparison).
- Evaluated 3 public and 2 commercial systems.
- Data, code and models are open-sourced.
Where it shows up today
- Vistaar's benchmark list and normalizer are a common way to report Indic ASR results, including for AI4Bharat's IndicConformer models and multilingual Parakeet / Canary-style models (Fast Conformer, Canary).
- IndicWhisper checkpoints are a standard baseline for Indian voice agents.
- For Indian-accented English, pair it with Svarah (Svarah). For the TTS side, see Indic TTS (AI4Bharat) and IndicVoices-R (IndicVoices-R).
What it got wrong, and what came next
- Whisper inherits Whisper's limits: offline 30 s windows, hallucination on silence, slow autoregressive decoding. Streaming Indic ASR needs other architectures (Cache-aware streaming Conformer).
- Training data skews to read and broadcast speech; conversational, code-mixed (Hinglish, Tanglish) and very noisy speech remain hard. IndicVoices (from the same group, see IndicVoices-R) later collected large-scale natural speech.
- Coverage: 12 languages leaves out many of India's 22 scheduled languages; later AI4Bharat work extended coverage.
- WER itself is harsh on agglutinative languages (Tamil, Malayalam, Telugu), where one long word carries what English says in several; CER (character error rate) is often reported alongside.
Check yourself
Q1Why build 59 benchmarks rather than one big test set?
Averaging hides behaviour per language and domain. Separate cells show that a model good on read speech can be poor on phone audio, or good in Hindi and weak in Odia. You choose a model by the cells that match your users.
Q2Give three reasons off-the-shelf Whisper does poorly on many Indian languages.
Little Indic audio in its training mix, noisy or mismatched web transcripts for those languages, and a tokenizer that splits Indic scripts into many tokens. Domain mismatch (phone calls, rural recordings) adds to it.
Q3Why does text normalization matter more for Indic WER?
Indic scripts allow variant spellings, nukta forms, numerals in digits or words, and punctuation choices. Without consistent normalization, correct transcripts get counted as substitutions, and different papers' numbers stop being comparable.
Q4What is IndicWhisper, in one line?
Whisper fine-tuned on ~10.7k hours of public Indian-language labelled speech, which gives the lowest WER on 39 of Vistaar's 59 benchmarks.
Q5What would happen if test utterances leaked into the training mix?
WER on those benchmarks would be optimistically low, since the model memorised them. Results would no longer predict real-world performance. Keeping train and test sources separate prevents that.
Q6For a Hindi voice agent on phone calls, which Vistaar cells would you look at first?
Hindi cells from phone or noisy conversational data (such as Gramvaani) and Kathbath's unknown-speaker split. Clean studio or read news cells matter less for that product.
When you're back at a laptop
- Clone AI4Bharat's Vistaar repo, run its evaluation script with an IndicWhisper Hindi checkpoint and with stock
openai/whisper-smallon Kathbath Hindi and one noisy set. - Print WER with and without the normalizer to see how much it moves.
- Check whether
whisper-smallin fp16 fits on your 4 GB RTX 3050 (it should) and measure real-time factor.