ASR · Phase 8: Indian languages and accents
IndicSUPERB
IndicSUPERB: A Speech Processing Universal Performance Benchmark for Indian languages
In one sentence
IndicSUPERB releases Kathbath, 1,684 hours of labelled read speech in 12 Indian languages from 1,218 speakers, and uses it to benchmark six speech tasks, from ASR to speaker and language identification.
Why this paper exists
SUPERB (Speech processing Universal PERformance Benchmark) is the English benchmark for comparing self-supervised speech models across many tasks. Indian languages had nothing comparable: labelled datasets were small, uneven across languages, and often from few speakers, so results were hard to compare and could reflect speaker memorisation.
The core idea
Collect a large, balanced, many-speaker labelled dataset with a clean protocol, define standard splits and tasks on it, and run the common models through all of them. Then everyone can report numbers on the same footing.
How it works
- Kathbath collection: contributors read sentences (from Indian-language text corpora) aloud into their phones, giving realistic phone-mic audio in natural home conditions. Transcripts are the prompted text, with validation.
- Coverage: 12 languages (including Hindi, Bengali, Tamil, Telugu, Marathi, Kannada, Malayalam, Gujarati, Odia, Punjabi, Urdu and Sanskrit; check the paper for the exact list).
- Splits: test sets with known speakers (seen in training) and unknown speakers (never seen), so you can measure generalisation to new voices.
- Six tasks: ASR, speaker verification, speaker identification (mono and multilingual), language identification, query by example, keyword spotting.
- Models compared: a FBANK (filter-bank features) baseline vs self-supervised models such as IndicWav2Vec (IndicWav2Vec) and generic multilingual ones.
Build the intuition
Analogy: before a common board exam, every school graded its own way, so you could not compare students. Kathbath is the shared exam paper, with a section of unseen questions (unknown speakers).
What would happen if you only tested on known speakers? A model could score well by learning those voices, then fail on your customers.
Numbers worth remembering
- Kathbath: 1,684 hours, 12 languages, 1,218 contributors.
- 6 benchmark tasks.
- Language-specific fine-tuned models beat the FBANK baseline on most tasks, with a large gap on language identification; self-supervised models trained on large data did best on speaker identification.
Where it shows up today
Kathbath is a standard Indic ASR test set: it is one of the Vistaar benchmarks (Vistaar) and a common evaluation for IndicWhisper, IndicConformer and multilingual Parakeet-style models. You will use its test-known / test-unknown splits to measure WER for Indian voice agents.
What it got wrong, and what came next
- Read speech is cleaner and more fluent than real calls: no hesitations, code-mixing or crosstalk. IndicVoices (see IndicVoices-R) collected spontaneous and conversational speech.
- One domain and style per utterance; Vistaar (Vistaar) combined many datasets to cover domains like news, conversations and phone audio.
Check yourself
Q1What is the purpose of the test-known vs test-unknown speaker split?
Test-known measures performance on voices seen in training; test-unknown measures generalisation to new voices. The gap tells you how much the model relies on memorising speakers, which matters for products with new users every call.
Q2Why record on contributors' phones rather than in a studio?
Phone microphones and home noise resemble real deployment audio. A studio set would overstate accuracy for voice agents and apps.
Q3Name three non-ASR tasks in IndicSUPERB.
Speaker verification, language identification and keyword spotting (also speaker identification and query by example). They test whether representations carry speaker and language information, not just words.
When you're back at a laptop
- Download Kathbath's Hindi test-known and test-unknown splits from Hugging Face.
- Run Whisper (Whisper) and an IndicWav2Vec model on 100 utterances of each split, and compare WER between splits and models.