Indic TTS (AI4Bharat)

Why this paper exists

By 2022 English TTS was near human quality, but Indian languages lagged badly. The public data, mainly the IndicTTS database (studio recordings from IIT Madras and partner institutes), has only a few speakers per language and a modest number of hours each. Existing Indian-language systems were older pipelines (HMM-based or early neural) with audible quality problems. Nobody had systematically answered the practical questions an engineer faces:

• Which acoustic model: autoregressive (Tacotron 2 Tacotron 2), duration-based (FastPitch, the NVIDIA cousin of FastSpeech 2 FastSpeech 2), flow-based (Glow-TTS Glow-TTS) or end-to-end (VITS VITS)?
• Which vocoder?
• One model per language, per language family, or one for all?
• Train male and female voices separately or together?

The core idea

In low-resource TTS, the recipe matters more than the novelty. Hold the data fixed, vary one design choice at a time, evaluate with native-speaker listening tests, and ship the winner as open models. The answer turned out to be a fast, stable, non-autoregressive model with explicit durations, a GAN vocoder, and sharing data across speakers of the same language.

How it works

The data

IndicTTS covers 13 languages from two families:

• Indo-Aryan: e.g. Hindi, Bengali, Marathi, Gujarati, Odia, Assamese, Rajasthani.
• Dravidian: Tamil, Telugu, Kannada, Malayalam.
• plus Manipuri and Bodo (other families). Check the paper for the exact list.

Each language has roughly one male and one female professional speaker reading prompts in a studio.

The winning pipeline

Indic text (native script)
   │ normalize, split into characters
   ▼
FastPitch acoustic model
  text encoder (Transformer)
   ├─ duration predictor ─┐
   ├─ pitch predictor  ───┤  added to hidden states
   └─ speaker embedding ──┘
   │ length regulator (repeat by duration)
   ▼
  mel decoder (Transformer) → 80-band mel
   ▼
HiFi-GAN V1 vocoder → waveform

• Characters, not phonemes: Indic scripts (abugidas, where each symbol is a consonant plus an inherent or marked vowel) map to sound far more regularly than English spelling, so character input works without a G2P (grapheme-to-phoneme) model. Hindi's schwa deletion (the inherent "a" is dropped in some positions) is one place where this is imperfect.
• FastPitch: predicts a duration (number of mel frames) and an average pitch per input symbol. The length regulator repeats each symbol's hidden vector d_i times, so the decoder generates all frames in parallel. Durations at training time come from an aligner (the NeMo/Coqui recipes use an attention-based or MAS-style aligner One TTS Alignment).
• HiFi-GAN V1: the largest HiFi-GAN configuration HiFi-GAN, a GAN vocoder that converts mel frames to audio in one pass.
• Joint male + female training: one model per language, conditioned on a speaker ID, trained on both speakers' data. Each voice benefits from the other's coverage of the language.

The losses

FastPitch trains with regression losses:

L = L_mel + α · L_pitch + γ · L_dur
L_mel   = || mel_pred - mel_true ||²
L_pitch = || p_pred - p_true ||²   (per-symbol mean F0)
L_dur   = || log d_pred - log d_true ||²

mel is the 80-band log-mel spectrogram, p the average pitch (F0) over each symbol's frames, d the duration in frames, α and γ weights. The paper also tests supplementary losses and training schedules on top of the base recipe (check the paper for which helped). HiFi-GAN is trained separately with adversarial + feature-matching + mel L1 losses.

Evaluation

MOS from native speakers of each language, comparing their models against existing open and commercial Indian-language systems, plus ablations for each design choice.

Build the intuition

Analogy: if you only have two singers per language, you get a better choir by letting them rehearse together than by training each alone, but adding singers from a different language with a different accent can muddy things.

Why would per-language-family or all-language models not win clearly? Pooling data helps when languages share sounds and scripts, but Indian languages use different scripts, so a character-level model sees disjoint input symbols for each language and shares little at the input. Pooling also mixes accents and prosody, and each language's own data was enough for a decent model. Sharing across the two speakers of the same language, however, shares everything: script, phonology, prosody.

Why FastPitch over Tacotron 2 in this setting? With little data, attention-based alignment is fragile: it skips or repeats syllables, especially in long agglutinative Dravidian words. Explicit durations cannot skip. FastPitch is also parallel, so inference is fast on CPU.

Numbers worth remembering

• 13 Indian languages, from the IndicTTS database.
• Two families compared: Indo-Aryan and Dravidian.
• Best setup: monolingual FastPitch + HiFi-GAN V1, trained jointly on male and female speakers.
• Reported significant MOS improvements over existing models in all 13 languages (check the paper's tables for values).
• Models released openly (AI4Bharat Indic-TTS repo, and on the Bhashini platform).

Where it shows up today

• The AI4Bharat Indic-TTS models (Coqui-TTS based) are still a common baseline and a fast CPU option for Indian-language voice agents.
• Later AI4Bharat work moved toward more data and more speakers: expressive data in Rasa Rasa, massive multi-speaker data in IndicVoices-R IndicVoices-R, description-controlled Indic Parler-TTS Parler-TTS, and zero-shot cloning with IndicF5 (F5-TTS F5-TTS fine-tuned).
• It is the TTS counterpart of the ASR paper with the same title, IndicWav2Vec IndicWav2Vec.

What it got wrong, and what came next

• Few speakers: two voices per language cannot do zero-shot cloning or voice variety. IndicVoices-R IndicVoices-R scaled to thousands of speakers.
• Read, neutral speech: studio reading sounds formal; expressive and conversational speech came with Rasa Rasa and in-the-wild style data like Emilia LibriTTS → Emilia.
• No code-mixing: Hinglish ("aapka EMI due hai") and English words in Indic script need text normalization and mixed-script handling (Text normalization challenge) that a monolingual character model lacks.
• Two-stage mel + vocoder pipeline: later end-to-end and flow-matching models (VITS, F5-TTS) improved naturalness, at higher compute.

Check yourself

When you're back at a laptop

• Clone AI4Bharat/Indic-TTS, download the Hindi FastPitch + HiFi-GAN checkpoints, and synthesize five sentences on CPU. Measure RTF.
• Feed it Hinglish, numbers ("₹1,50,000") and dates; note each failure and which front-end step would fix it.
• Swap in a different vocoder if one is available and compare by ear.
• Transcribe outputs with IndicWhisper or IndicConformer Vistaar and compute CER.