FastSpeech 2

Why this paper exists

FastSpeech (FastSpeech) made synthesis parallel and robust, but at a cost:

• A teacher pipeline. You first had to train an autoregressive Transformer TTS (Transformer TTS), then extract durations from its attention and generate a distilled mel dataset. That is slow and fragile.
• Noisy durations. Attention-derived durations are imprecise, which hurts rhythm.
• Quality ceiling. The student learns the teacher's outputs, so it cannot exceed them, and distillation throws away natural variation.

Underneath sits the one-to-many problem: the same text can be spoken with many pitch contours, loudness patterns and rhythms. A non-autoregressive model trained with L1/L2 loss on real data, given only text, predicts the average of these, which sounds flat and blurry. Distillation hid the problem by making the targets less varied.

The core idea

Give the decoder the missing information explicitly. If the model is told how long, how high and how loud each part is, the mapping from input to mel becomes much closer to one-to-one, so a regression loss on real mels works. During training those values come from the ground-truth audio; at inference, small predictors estimate them from text. Because they are explicit inputs, you can also edit them.

How it works

phonemes -> embedding + PE -> encoder (FFT blocks)
                     | H: N x d
   +------------- variance adaptor ----------------+
   | duration predictor -> d_i  (train: MFA)       |
   | length regulator: repeat h_i d_i times        |
   |                    T x d,  T = Σ d_i          |
   | pitch predictor  -> p_t -> bin -> embed -> +  |
   | energy predictor -> e_t -> bin -> embed -> +  |
   +-----------------------------------------------+
                     | T x d
        + PE -> mel decoder (FFT blocks)
             -> linear -> mel (T x 80) -> vocoder

Encoder and decoder. Stacks of feed-forward Transformer blocks (self-attention + 1-D conv), as in FastSpeech.

Durations from a forced aligner. The Montreal Forced Aligner (MFA), a classic HMM-based aligner, aligns each training transcript to its audio and gives phoneme boundaries, converted to integer frame counts d_i. No teacher network.

Variance adaptor. Three predictors with the same small architecture (two 1-D conv layers with ReLU, layer norm and dropout, then a linear output):

• Duration predictor: predicts log duration per phoneme. Trained with MSE against MFA durations.
• Pitch predictor: F0 is extracted from the audio (PyWorld), unvoiced gaps are interpolated, and it is log-scaled and normalized. In the paper the predictor works on a continuous-wavelet-transform (CWT) "pitch spectrogram" rather than raw F0, to model pitch variation at several time scales; check the paper for the details. The value is quantized into 256 bins (log-spaced), each bin has a learned embedding, and that embedding is added to the hidden sequence.
• Energy predictor: energy is the L2 norm of each STFT frame's magnitude, quantized into 256 uniform bins, embedded, and added.

In the paper, pitch and energy are predicted at the frame level (after the length regulator). Many open implementations and FastPitch average them per phoneme instead, which is smoother and easier to control.

Teacher forcing of variances. During training the ground-truth duration, pitch and energy are fed into the adaptor (and the predictors are trained alongside with MSE). At inference the predicted values are used.

Loss.

L = L1(mel_pred, mel_true) + MSE(log d_pred, log d_true) + MSE(p_pred, p_true) + MSE(e_pred, e_true)

• mel_true: real mel spectrogram (T × 80).
• d: per-phoneme frame counts; p, e: pitch and energy targets.

FastSpeech 2s. A variant that generates waveform directly from text using an adversarial waveform decoder; a step toward VITS (VITS).

Build the intuition

Length regulator on a toy input

Phonemes for a short word, say [n, ə, m], with 2-dim encoder outputs:

h_n = [0.1, 0.9]   h_ə = [0.5, 0.2]   h_m = [0.8, 0.4]
durations d = [2, 3, 1]

Length regulator (repeat_interleave):

frame: 1      2      3      4      5      6
       h_n    h_n    h_ə    h_ə    h_ə    h_m
       .1 .9  .1 .9  .5 .2  .5 .2  .5 .2  .8 .4

Shape goes from 3 × 2 to 6 × 2, with 6 = 2 + 3 + 1. At 12.5 ms hop that is 75 ms of audio.

Now pitch. Suppose the pitch predictor outputs a log-F0 for each frame, which falls into bins [40, 41, 52, 55, 57, 50]. Look up E_pitch[bin] for each frame and add it:

frame 3 input = h_ə + E_pitch[52] + E_energy[...]

Frames 3 to 5 share the same phoneme vector but get different pitch embeddings, so the decoder can make a rising pitch within the vowel. That is information the text alone could not provide.

Controls at inference

• Speed: multiply durations by 0.8 → round([1.6, 2.4, 0.8]) = [2, 2, 1], 5 frames. Keep every duration ≥ 1, or a phoneme vanishes.
• Pitch: add a constant to log-F0 before binning to raise the whole voice, or scale its variance for a more animated reading.
• Emphasis: increase energy and duration on one word.

For a voice agent that must read a loan amount slowly and clearly, these knobs are exactly what you want.

What would happen without the variance adaptor?

Train FastSpeech's architecture on real mels with no pitch or energy inputs. For a phoneme that is sometimes high and sometimes low in the data, the L1-optimal prediction is the median: a flattened contour. Harmonics blur because different pitches put energy in different frequency bins. The adaptor removes most of that ambiguity before the decoder sees it.

Analogy: the decoder is a session singer. FastSpeech handed them a lyric sheet and a recording of another singer to imitate. FastSpeech 2 hands them a lyric sheet with note lengths, notes and dynamics written in, so they can sing it straight, and you can rewrite the markings.

Numbers worth remembering

• Data: LJSpeech (~24 hours, one English speaker).
• Training is about 3x faster than FastSpeech, since there is no teacher stage.
• Quality: MOS above FastSpeech and comparable to or better than autoregressive Tacotron 2 / Transformer TTS in their tests; check Table 1 for exact values.
• Inference speed similar to FastSpeech: all frames in parallel, far faster than autoregressive models.
• 256 bins each for pitch and energy quantization.

Where it shows up today

• FastPitch (NVIDIA NeMo) is the same family: per-symbol pitch predictor, parallel decoder, now usually with the learned aligner from One TTS Alignment.
• AI4Bharat's Indian-language TTS (Indic TTS (AI4Bharat)) uses FastPitch-style models with HiFi-GAN (HiFi-GAN).
• Variance predictors (duration, pitch, energy) reappear in NaturalSpeech 2 (NaturalSpeech 2 → NaturalSpeech 3), StyleTTS 2 / Kokoro (StyleTTS → StyleTTS 2, prosody predictors for F0 and energy) and Voicebox's duration model (Voicebox).
• Montreal Forced Aligner is still widely used for TTS and also for ASR data work (compare WhisperX's alignment, WhisperX).

What it got wrong, and what came next

• External aligner dependency: MFA needs a pronunciation lexicon and acoustic model per language; for Hindi, Tamil or code-mixed Hinglish that can be a real obstacle. Glow-TTS (Glow-TTS) learns alignment with MAS, and One TTS Alignment gives a reusable learned aligner.
• Still deterministic: given text, output is fixed and prosody can be flat. VITS (VITS) adds a stochastic duration predictor; StyleTTS 2 (StyleTTS → StyleTTS 2) samples style; flow-matching decoders (Matcha-TTS, F5-TTS) model the full distribution.
• Two-stage with a separate vocoder (HiFi-GAN): VITS trains end to end.
• Pitch extraction errors (octave jumps, unvoiced frames) leak directly into training targets.

Check yourself

When you're back at a laptop

• Implement the length regulator and check shapes:

h = torch.randn(3, 4)
d = torch.tensor([2, 3, 1])
x = torch.repeat_interleave(h, d, dim=0)
assert x.shape == (6, 4)

• Write a duration predictor (2 conv layers + linear on log durations), feed fake durations, and test speed scaling with clamping to ≥ 1.
• Run a pretrained FastPitch from NeMo on CPU; shift pitch by +2 semitones and scale durations by 0.8, and listen.