FastSpeech

Why this paper exists

Tacotron 2 (Tacotron 2) and Transformer TTS (Transformer TTS) are autoregressive: one decoder pass per mel frame, so latency grows with utterance length. Their alignment is learned soft attention, so on hard sentences they skip words, repeat words or never stop. And you cannot ask them to speak 20% faster. All three problems come from the same root: the model decides alignment implicitly, frame by frame.

The core idea

Make alignment explicit. TTS alignment is monotonic and every phoneme maps to a contiguous block of frames, so it can be described completely by one integer per phoneme: its duration in frames. Predict those durations, expand the phoneme sequence to frame length, and the rest is a parallel sequence-to-sequence mapping of equal lengths. No attention between text and audio is needed at all.

How it works

phonemes -> embedding + PE -> 6 x FFT block (encoder)
                                   |  h (N x d)
                       duration predictor -> d_i
                                   |
                  length regulator: repeat h_i d_i times
                                   |  (T x d), T = Σ d_i
                     + PE -> 6 x FFT block (decoder)
                                   |
                         linear -> mel (T x 80)

• FFT block (feed-forward Transformer): self-attention followed by a 1-D convolution network (instead of the usual position-wise MLP), because neighbouring phonemes and frames are strongly related.
• Duration predictor: two 1-D conv layers with ReLU, layer norm and dropout, then a linear layer that outputs a scalar per phoneme. It predicts log duration (more stable) and is trained with MSE against the target durations.
• Length regulator: LR(H, D, α). Repeat each hidden vector h_i exactly d_i times. α scales the durations for speed control: α = 1.2 makes speech 20% slower, α = 0.8 faster. Pauses between words can be lengthened by scaling only the durations of space/punctuation tokens.
• Where durations come from: an autoregressive Transformer TTS teacher is trained first. For each training utterance, they pick the encoder-decoder attention head whose alignment is most diagonal (highest "focus rate") and count how many frames attend most strongly to each phoneme.
• Sequence-level knowledge distillation: the student is trained on the teacher's generated mels, not the real ones. Teacher outputs are less variable (more "average"), which is an easier target for a non-autoregressive model.
• Vocoder: WaveGlow (Parallel WaveNet + WaveGlow).

Build the intuition

Toy input: phonemes [k, æ, t] ("cat") with encoder outputs h_k, h_æ, h_t and durations [2, 4, 3].

LR(α=1):   h_k h_k h_æ h_æ h_æ h_æ h_t h_t h_t   (9 frames)
LR(α=0.5): round([1, 2, 1.5]) -> e.g. [1, 2, 2]  (5 frames)

At 80 frames per second, 9 frames is about 0.11 s. The decoder now has one input vector per output frame and can compute all of them at once. Nothing can be skipped (every phoneme has at least one frame) and nothing can repeat (each phoneme's block is used exactly once, in order).

Why distillation? Real speech of the same text varies a lot; a parallel model predicting all frames independently, given only the text, tends to average those variations into blurry mels. Training on the teacher's output, which is itself a smoothed, deterministic version, hides that one-to-many problem. FastSpeech 2 (FastSpeech 2) attacks it properly instead.

Analogy: autoregressive TTS is a singer reading the score live and deciding note lengths as they go; FastSpeech first writes the note lengths into the score, then the whole orchestra plays every bar at once.

Numbers worth remembering

• Data: LJSpeech, ~24 hours, 13,100 clips of one female English speaker.
• Speed: about 270x faster mel generation and 38x faster end-to-end (with WaveGlow) than the autoregressive Transformer TTS baseline.
• Quality: MOS roughly on par with the autoregressive teacher (check the paper's table).
• Robustness: on a set of particularly hard sentences, FastSpeech had almost no skipped or repeated words while the autoregressive models had many (check the paper for the error rates).

Where it shows up today

• Explicit durations plus length regulator is the backbone of FastSpeech 2 (FastSpeech 2), FastPitch (NVIDIA NeMo), Glow-TTS (Glow-TTS), VITS (VITS), Grad-TTS (Grad-TTS), Matcha-TTS (Matcha-TTS) and Kokoro via StyleTTS 2 (StyleTTS → StyleTTS 2).
• Duration models are why on-device TTS such as Piper (VITS) and Kokoro can expose a "speed" slider.
• The opposite direction, dropping durations entirely, is E2 / F5-TTS (E2 TTS, F5-TTS).

What it got wrong, and what came next

• Teacher dependence: a whole autoregressive model must be trained to get durations and targets; durations from attention are noisy. FastSpeech 2 (FastSpeech 2) uses a forced aligner; Glow-TTS (Glow-TTS) learns alignment with MAS; One TTS Alignment makes alignment a module.
• Distilled targets cap quality at the teacher's level and lose variety. FastSpeech 2 (FastSpeech 2) trains on real mels with pitch and energy inputs.
• One-to-many problem: deterministic output gives flat prosody; stochastic duration and generative decoders (VITS, StyleTTS → StyleTTS 2, Matcha-TTS) address it.

Check yourself

When you're back at a laptop

• Implement the length regulator with torch.repeat_interleave(h, durations, dim=0) and a batched version with padding.
• Train a tiny duration predictor on durations from Montreal Forced Aligner for 100 LJSpeech clips and plot predicted vs real durations.