Reading Room

TTS · Phase 9: Data, evaluation and Indian languages

IndicVoices-R

IndicVoices-R: Unlocking a Massive Multilingual Multi-speaker Speech Corpus for Scaling Indian TTS

Sankar, Anand, Srinivasa Varadhan, Thomas, Singal, Kumar, Mehendale, Krishana, Raju, Khapra · AI4Bharat (IIT Madras) · 2024

Deep read18 minDay 10 of 10#86 of 86

In one sentence

IndicVoices-R takes IndicVoices, a large but noisy ASR corpus of natural Indian speech, cleans it with denoising and speech-enhancement models into 1,704 hours of TTS-quality audio from 10,496 speakers in 22 languages, and adds a benchmark for how well TTS models generalize to unseen speakers.

Why this paper exists

Indian TTS had a speaker problem. The best studio data (IndicTTS, used in Indic TTS (AI4Bharat)) has about two speakers per language. That is enough for a fixed voice, but zero-shot cloning (VALL-E, F5-TTS) needs thousands of speakers to learn what "a voice" is. English solved this with LibriTTS and Emilia LibriTTS → Emilia. India has huge amounts of speech in ASR corpora, but ASR data is recorded on phones, in homes and streets, with noise, reverberation and low bandwidth. A TTS model trained on it learns to generate the noise too. So the data existed, but in a form TTS could not use.

The core idea

ASR data has exactly what TTS lacks (many speakers, natural speaking style, all 22 scheduled languages) and lacks what TTS needs (clean audio). Restore the audio instead of re-recording it: run denoising and speech-enhancement models over the corpus, filter what still falls short, and the result can match studio TTS datasets on quality metrics. The enhancement models were trained on English, which works because noise and reverberation are largely language-independent.

How it works

The source corpus

IndicVoices (AI4Bharat) is a large ASR dataset collected across India's districts, with read, extempore (spontaneous) and conversational speech in all 22 scheduled languages, from many thousands of speakers, recorded in everyday conditions.

The pipeline

IndicVoices ASR audio (noisy, varied mics)
   │ select utterances, keep transcripts
   ▼
denoise / source separation
   ▼
speech enhancement / restoration
  (bandwidth, dereverberation)
   ▼
quality filtering (automatic scores,
  thresholds per utterance)
   ▼
IndicVoices-R: 1,704 h, 10,496 speakers,
  22 languages, TTS-ready

Check the paper for the exact enhancement models and filter thresholds. Terms:

  • Denoising removes additive noise (fans, traffic).
  • Speech restoration goes further: it repairs bandwidth limits (phone-quality 8 kHz-like audio has no high frequencies) and reverberation, effectively regenerating clean speech conditioned on the degraded input.
  • Quality filtering discards clips where enhancement failed or introduced artifacts, using automatic quality metrics such as SNR estimates and MOS predictors UTMOS.

Showing it is TTS quality

The paper compares IndicVoices-R's quality metrics with gold-standard TTS sets (LJSpeech, LibriTTS LibriTTS → Emilia, IndicTTS) and reports comparable quality. A typical quality score is an automatic MOS or SNR estimate, averaged over clips (check the paper for which ones).

The benchmark

A benchmark for speaker generalization, at three levels:

  • zero-shot: the target speaker is never seen in training; you clone from a short prompt,
  • few-shot: a little target-speaker data is available for adaptation,
  • many-shot: plenty of target-speaker data.

Speaker similarity is measured with a speaker-verification embedding:

SIM = cos(e_prompt, e_output)
    = (e_p · e_o) / (|e_p| · |e_o|)

e_p and e_o are the speaker embeddings (fixed-length vectors from a speaker-verification model) of the reference speaker and the synthesized audio. Intelligibility is measured by ASR error rate on the output, and naturalness by listening tests or predicted MOS.

The key experiment

Take an English-pretrained TTS model and fine-tune it on (a) IndicTTS only, or (b) IndicTTS plus IndicVoices-R. Adding IndicVoices-R improves zero-shot speaker generalization, because the model has now seen thousands of Indian voices instead of a couple per language. The authors release the data, code, and a TTS model covering all 22 official languages.

Build the intuition

Analogy: an art school with two models to draw learns those two faces perfectly and nothing else. Bring in ten thousand faces, even from slightly blurry photos you have cleaned up, and students learn what faces in general look like, so they can draw a new one from a single glance.

What would happen if you trained on raw IndicVoices without restoration? The model would learn the noise and room as part of each voice. Since the prompt determines the "recording conditions" in zero-shot models (VALL-E copies the prompt's acoustic environment VALL-E), outputs would sound like a phone call in a busy street. Restoration removes conditions while keeping identity and style.

What would happen if enhancement changed the voice? A restoration model that regenerates speech can drift timbre or smear phonemes, especially for sounds absent from its English training data (retroflex consonants, aspirated stops). That is why the filter step and an ASR check on restored audio matter: a clip that is clean but now says something different is worse than useless.

Numbers worth remembering

  • 1,704 hours of speech.
  • 10,496 speakers.
  • 22 Indian languages (all scheduled languages).
  • Quality comparable to LJSpeech, LibriTTS and IndicTTS on the paper's metrics.
  • Benchmark levels: zero-shot, few-shot, many-shot speaker generalization.
  • Published at the NeurIPS 2024 Datasets and Benchmarks track.

Where it shows up today

  • Training data for open Indian multi-speaker TTS, including AI4Bharat's description-controlled Indic Parler-TTS Parler-TTS and zero-shot IndicF5 (built on F5-TTS F5-TTS); check each model card for the exact training sets.
  • The "turn ASR data into TTS data" pattern parallels Emilia-Pipe LibriTTS → Emilia and LibriTTS-R, and runs in the opposite direction to ASR pseudo-labelling pipelines like Granary Granary and Distil-Whisper Distil-Whisper.
  • The ASR side of the same ecosystem: IndicWav2Vec IndicWav2Vec, Kathbath IndicSUPERB, Vistaar and IndicWhisper Vistaar.

What it got wrong, and what came next

  • Restoration artifacts: enhancement models trained on English can introduce subtle phonetic damage in Indian languages; restored audio is clean but not studio audio. Better multilingual restoration, or training codecs and TTS models that tolerate mild noise, are the follow-ups.
  • Uneven coverage: hours and speakers are not balanced across 22 languages; low-resource languages still have thin data.
  • Code-mixing and agent speech: most utterances are monolingual; Hinglish and Indian-English customer-support speech need dedicated data and normalization Text normalization challenge.
  • Data, not architecture: the paper's lesson is that clean, multi-speaker, natural data is the bottleneck for Indian TTS; architectures (CosyVoice → CosyVoice 2, F5-TTS, CALM (Pocket TTS)) are mostly language-agnostic once data exists.

Check yourself

Q1Why can't you train a zero-shot cloning model for Indian languages on IndicTTS alone?

IndicTTS has about two speakers per language. Zero-shot cloning needs the model to learn a general space of voices from thousands of speakers, so that a short prompt can place a new voice in that space. With two speakers the model memorizes them and cannot generalize.

Q2Why is ASR data unsuitable for TTS as-is, and how did IndicVoices-R fix it?

ASR data is recorded in everyday conditions with noise, reverberation and limited bandwidth, and a TTS model learns to reproduce those conditions. IndicVoices-R ran denoising and speech-enhancement models over the audio, then filtered by quality, producing clean audio while keeping the many speakers and natural style.

Q3The enhancement models were trained on English. Why does that mostly work, and where could it fail?

Noise, reverberation and bandwidth loss are acoustic, not linguistic, so removing them transfers across languages. It can fail on sounds rare in English, such as retroflex and aspirated consonants, where a generative restoration model may "correct" them toward English-like sounds. Checking restored audio with an Indian-language ASR model catches such damage.

Q4What are the three levels of the speaker-generalization benchmark, and which matters for a voice agent?

Zero-shot (unseen speaker, clone from a prompt), few-shot (a little adaptation data) and many-shot (plenty of target data). A product with one brand voice is a many-shot or few-shot problem; a product that clones customer or agent voices on demand is zero-shot.

Q5What did the fine-tuning experiment show?

An English-pretrained TTS model fine-tuned on IndicTTS plus IndicVoices-R generalized better to unseen speakers than one fine-tuned on IndicTTS alone. Speaker diversity, not just hours, drives zero-shot quality.

Q6How would you check that a restored corpus is actually TTS quality?

Compute automatic quality scores (predicted MOS, SNR) and compare their distribution with known TTS sets like LJSpeech and LibriTTS. Transcribe the restored audio with ASR and compare to the original transcripts to catch content damage. Then train a small TTS model on it and run a listening test, since the final judge is the synthesized speech.

When you're back at a laptop

  • Download a few hundred IndicVoices-R Hindi and Tamil clips from Hugging Face; listen to several speakers and note residual artifacts.
  • Score a sample with UTMOS UTMOS and DNSMOS, and compare with IndicTTS and LJSpeech clips.
  • Run a raw IndicVoices clip through an enhancement model (e.g. DeepFilterNet or a restoration model), transcribe before and after with IndicConformer or IndicWhisper Vistaar, and compare CER.
  • Try IndicF5 or Indic Parler-TTS on Hindi with a short prompt of your own voice; compute SIM with a WavLM speaker-verification model.
arXiv ↗PDF ↗