Reading Room

TTS · Phase 9: Data, evaluation and Indian languages

LibriTTS → Emilia

LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speech; Emilia: An Extensive, Multilingual, and Diverse Speech Dataset for Large-Scale Speech Generation

Zen, Dang, Clark, Zhang, Weiss, Jia, Chen, Wu (LibriTTS); He, Shang, Wang, ... Wu (Emilia) · Google (LibriTTS); CUHK-Shenzhen / Amphion team (Emilia) · 2019

Skim12 minDay 10 of 10#81 of 86

In one sentence

LibriTTS re-cut the LibriSpeech audiobooks into a clean 24 kHz, sentence-level, punctuated corpus that became the standard academic multi-speaker TTS set; Emilia, five years later, built 100k+ hours of spontaneous, in-the-wild speech in six languages with an open pipeline, because read-speech data produces read-sounding voices.

Why this paper exists

LibriTTS (2019). LibriSpeech (~1,000 hours of English audiobooks) was the default ASR corpus, and TTS researchers wanted to reuse it for multi-speaker work. It was a poor fit for TTS:

  • audio was 16 kHz, too low for good synthesis;
  • text was upper-cased with punctuation removed, so a model could not learn that a comma means a pause or a question mark means rising pitch;
  • utterances were cut at silences, not sentence boundaries, so segments started or ended mid-sentence;
  • some recordings were noisy.

Emilia (2024). By 2024 zero-shot TTS models (VALL-E, Voicebox, F5-TTS) were trained on tens of thousands of hours, mostly audiobooks (LibriLight, MLS). Audiobook narration is careful and even. Models trained on it sound like someone reading aloud, not someone talking, which is exactly wrong for a voice agent. Spontaneous speech exists in huge amounts on video platforms, but it is messy: background music, overlapping speakers, no transcripts.

The core idea

Both papers say the same thing at different scales: the data defines the voice. LibriTTS fixes the format of an existing corpus so TTS can use it. Emilia goes to the source of natural conversational speech and builds a repeatable pipeline that cleans, segments and transcribes it automatically.

How it works

LibriTTS

Built from the original LibriVox MP3s and Project Gutenberg texts that LibriSpeech came from (not from LibriSpeech's processed files):

  1. Audio resampled to 24 kHz.
  2. Text split at sentence boundaries, keeping both the original text (case and punctuation) and a normalized version (numbers and abbreviations expanded; see Text normalization challenge).
  3. Utterances with significant background noise removed.
  4. Same speaker splits as LibriSpeech: train-clean-100, train-clean-360, train-other-500, plus dev/test clean/other.

Emilia-Pipe

raw video audio (talk shows, interviews, debates...)
  → standardize (resample, mono, loudness)
  → source separation (remove music / background)
  → speaker diarization (who spoke when)
  → VAD-based segmentation (short single-speaker clips)
  → ASR transcription (Whisper-family)
  → filtering (language ID, quality score, ...)
  → (clip, transcript, speaker) training rows

Each step uses an off-the-shelf model, so the pipeline is cheap to run on new data and new languages. Terms: diarization is labelling which speaker is talking at each moment; VAD (voice activity detection) finds where speech is. The quality filter uses an automatic MOS-style predictor (see UTMOS; check the paper for the exact model and thresholds).

Build the intuition

Analogy: a voice model is a mimic who learns from whatever it hears. Train it only on audiobook narrators and it will narrate everything, including "Your balance is 4,200 rupees". Train it on people chatting and it picks up hesitations, fast and slow stretches, and laughter.

Thought experiment: what would happen if you trained a TTS model on LibriSpeech's original text (no punctuation)? It would have no cue for where to pause or which sentences are questions, so prosody would be flat and pauses would be guessed from length alone. This one formatting fix is a large part of LibriTTS's value.

Numbers worth remembering

  • LibriTTS: 585 hours, 2,456 speakers, 24 kHz, English.
  • LibriTTS paper: models trained on it reached naturalness MOS above 4.0 for five of six evaluation speakers.
  • Emilia: 101k+ hours across six languages (English, Chinese, German, French, Japanese, Korean), with English and Chinese the largest share.
  • A later extension (Emilia-Large, adding YODAS-derived data) grew it to roughly double that size; check the Amphion repo for the current figure.

Where it shows up today

  • LibriTTS (and LibriTTS-R, a speech-restored version) is the standard training and test set for multi-speaker papers: vocoders like BigVGAN, codecs like EnCodec, and many zero-shot TTS baselines.
  • Emilia is the training set of F5-TTS F5-TTS and MaskGCT MaskGCT, and is widely used for open zero-shot models.
  • Emilia-Pipe's shape (separate, diarize, VAD, ASR, filter) is the same one used to turn noisy ASR data into TTS data for Indian languages in IndicVoices-R, and mirrors ASR pseudo-labelling pipelines like Granary Granary and WhisperX's VAD + alignment WhisperX.

What it got wrong, and what came next

  • LibriTTS is still read audiobook speech, English only, and some audio is not studio clean. LibriTTS-R improved the audio quality; Emilia addressed the speaking style.
  • Emilia's transcripts come from ASR, so they contain errors; the filtering step trades data volume for label quality. Licensing of video-sourced audio limits commercial use (check the dataset licence).
  • Neither covers Indian languages. For those, see IndicTTS-based work Indic TTS (AI4Bharat), expressive data in Rasa and IndicVoices-R IndicVoices-R.

Check yourself

Q1Name three things about LibriSpeech that made it a poor TTS corpus, and how LibriTTS fixed each.

It was 16 kHz, which LibriTTS raised to 24 kHz by going back to the original MP3s. Its text had no case or punctuation, so LibriTTS kept the original text alongside a normalized version. Its segments were cut at silences, so LibriTTS split at sentence boundaries instead. It also dropped noisy utterances.

Q2Why would an audiobook-trained TTS model sound wrong in a voice agent?

Audiobook narration is slow, careful and evenly paced, with formal prosody. A model reproduces the style of its training data, so it reads replies like a narrator. Conversational agents need spontaneous rhythm, shorter phrases and casual emphasis, which is what Emilia's in-the-wild data provides.

Q3Walk through the Emilia-Pipe stages and say why each is needed.

Standardization makes all inputs the same format. Source separation removes music and noise that the TTS model would otherwise learn to generate. Diarization and VAD cut the audio into single-speaker speech clips, because a training clip with two voices teaches nonsense. ASR provides transcripts since none exist, and filtering removes clips with low quality, wrong language or likely transcription errors.

Q4What is the main risk of using ASR-generated transcripts as TTS training labels?

Transcript errors teach the model to say words that are not in the text, or skip words that are. This shows up as higher WER on synthesized speech. Filtering by ASR confidence or by agreement between models reduces it at the cost of discarding data.

When you're back at a laptop

  • Download dev-clean of LibriTTS from OpenSLR. Compare an .original.txt with its .normalized.txt file and list what normalization changed.
  • Stream a few Emilia samples from Hugging Face (amphion/Emilia-Dataset), listen, and note the difference in style from LibriTTS.
  • Build a mini Emilia-Pipe on one YouTube-style podcast clip: Demucs for separation, pyannote for diarization, Silero VAD, Whisper for transcripts. Keep only single-speaker clips of 3-15 s.
LibriTTS arXiv ↗Emilia arXiv ↗