Grad-TTS

Why this paper exists

Non-autoregressive acoustic models like FastSpeech 2 (FastSpeech 2) regress the mel with an L1/L2 loss, which predicts the average of all plausible mels and sounds over-smoothed (buzzy, flat). Glow-TTS (Glow-TTS) uses a flow for exact likelihood, but flows need large invertible architectures. Diffusion had just worked for vocoders (WaveGrad + DiffWave); this paper brings it to the acoustic model, where the output (an 80 x T mel) is far smaller than a waveform.

The core idea

Make the diffusion process end at the encoder's prediction instead of at zero-mean noise. The encoder output μ already looks like a blurry mel, so the decoder starts from μ + noise and only has to add the detail, which takes fewer steps than starting from pure noise.

How it works

• Encoder + alignment: a Transformer text encoder outputs μ per phoneme (80 dims, mel-shaped). MAS (Glow-TTS) aligns phonemes to frames; an encoder loss makes the aligned μ close to the target mel (a Gaussian log-likelihood). A duration predictor learns the MAS durations.
• Forward SDE (stochastic differential equation) that turns a mel X_0 into noise around μ:

dX_t = ½ (μ - X_t) β_t dt + sqrt(β_t) dW_t,   t ∈ [0, 1]

β_t is the noise schedule, W_t Brownian motion. As t → 1, X_t approaches N(μ, I).

• Training: a U-Net decoder learns the score (gradient of log-density) of noisy mels at every t, conditioned on μ. The loss is the standard denoising score-matching regression.
• Inference: sample X_1 ~ N(μ, I/τ) (τ is a temperature; higher means less noise), then solve the reverse-time ODE or SDE with N steps to get X_0, the mel. Vocode with HiFi-GAN (HiFi-GAN).

Build the intuition

Analogy: Standard diffusion is restoring a painting from a canvas covered entirely in static. Grad-TTS restores it from a blurry photo of the painting with some static on top. The blurry photo (the encoder's μ) already has the composition right; the steps only add brushwork.

What would happen with 1 step? You get roughly μ plus a single denoising nudge: intelligible but over-smoothed, like FastSpeech 2. With ~10 steps detail appears; beyond some point extra steps give little. The paper shows this trade-off directly.

Numbers worth remembering

• Decoder steps are chosen at inference; the paper compares from a few up to ~1000 (check the MOS-vs-steps figure).
• Around 10 steps already gave quality competitive with Tacotron 2 and Glow-TTS on LJSpeech (check Table 1).
• Model size ~15M parameters (check the paper).

Where it shows up today

• Matcha-TTS (Matcha-TTS) keeps Grad-TTS's encoder, MAS and "decoder refines μ" design, and replaces diffusion with flow matching for fewer steps.
• The pattern "coarse prediction + generative refinement" appears in CosyVoice's flow-matching mel stage (CosyVoice → CosyVoice 2) and in many voice conversion systems from the same Huawei group.
• Diffusion as an acoustic model scaled up in NaturalSpeech 2 (NaturalSpeech 2 → NaturalSpeech 3) and Seed-TTS (Seed-TTS).

What it got wrong, and what came next

• Many solver steps over a U-Net per utterance: slow for real-time and CPU use. Matcha-TTS (Matcha-TTS) uses OT-CFM, which reaches good quality in a few Euler steps, with a smaller, faster decoder.
• Still needs MAS and a duration predictor; E2/F5-TTS (E2 TTS, F5-TTS) drop explicit alignment.
• Mel output still needs a separate vocoder (HiFi-GAN, iSTFTNet + Vocos).

Check yourself

When you're back at a laptop

• Clone the official huawei-noah/Speech-Backbones Grad-TTS code, run the LJSpeech checkpoint with 2, 10 and 50 steps, and compare mels and audio.
• Plot μ against the final mel for one sentence to see what the decoder adds.