TTS · Phase 2: Non-autoregressive TTS and alignment
One TTS Alignment
One TTS Alignment To Rule Them All
In one sentence
NVIDIA's paper packages TTS alignment as a small, drop-in module: a soft text-to-mel attention trained with a CTC-style "forward-sum" loss over all monotonic alignments, sharpened into hard durations with Monotonic Alignment Search and a binarization loss, which removes the need for forced aligners or teacher models in any TTS architecture.
Why this paper exists
By 2021 every TTS family had its own alignment hack. Tacotron 2 (Tacotron 2) relied on fragile soft attention. FastSpeech (FastSpeech) took durations from a teacher; FastSpeech 2 and NVIDIA's FastPitch (FastSpeech 2) from the Montreal Forced Aligner or a pretrained Tacotron 2. Glow-TTS (Glow-TTS) learned durations with MAS, but only because its flow decoder gives an exact likelihood. Each option either needs an extra model or a lexicon per language, or is tied to one architecture. The question: can one alignment learner work for autoregressive and parallel models alike?
The core idea
Learn alignment from a separate, lightweight pair of encoders whose only job is to say how well each mel frame matches each text token. Train it with an objective that sums over all monotonic alignments (so it is differentiable and does not commit early), then extract the single best path with MAS for the parts of the TTS model that need integer durations, and push the soft alignment toward that hard one.
How it works
- Alignment encoders: a small conv encoder over text embeddings and another over mel frames. The score between token
iand framejis a negative squared distance,-||e_i - m_j||². - Soft alignment: for each frame
j, a softmax over tokens givesA_soft[i, j], the probability that framejbelongs to tokeni. - Alignment prior: a static 2-D beta-binomial prior that is high near the diagonal is multiplied in (in log space, added). It encodes "frame j of T is probably near token j·N/T", which speeds up convergence a lot early in training.
- Forward-sum loss: the probability of the text given the mel, summed over every monotonic path through
A_soft, computed with the CTC forward algorithm (CTC). In practice they reuse PyTorch's CTC loss.
L_fs = -log Σ_{monotonic paths φ} Π_j A_soft[φ(j), j]
A_hard = MAS(A_soft) (Viterbi best path)
L_bin = -Σ_{i,j} A_hard[i,j] · log A_soft[i,j]- Hard alignment: MAS, exactly as in Glow-TTS (Glow-TTS), on
log A_soft, gives a 0/1 matrix and therefore integer durations, used by duration predictors and length regulators. - Binarization loss:
L_binis a cross-entropy pulling the soft distribution toward the hard path, so the two agree. It is switched on after the soft alignment has roughly converged. - Integration: for parallel models (FastPitch, RAD-TTS) the hard durations replace the external aligner; for autoregressive models (Tacotron 2, Flowtron) the loss and prior regularize the attention.
Build the intuition
Take the 2-token, 4-frame log-likelihood grid from the Glow-TTS page (Glow-TTS). There are three valid monotonic paths:
| path | score |
|---|---|
| a a a b | -7 |
| a a b b | -4 |
| a b b b | -6 |
- MAS picks
a a b bwith score -4. - Forward-sum uses all three:
log(e^-7 + e^-4 + e^-6) ≈ -4 + log(1 + 0.050 + 0.135) ≈ -3.83. The loss is about 3.83.
Early in training the grid is noisy, many paths score similarly, and forward-sum spreads gradient across all of them, which is stable. Later, one path dominates and the sum approaches the max. The binarization loss accelerates that, so the soft attention ends up looking like the hard durations the decoder is trained with.
The beta-binomial prior is like starting with a fuzzy diagonal band drawn on the grid: the model does not have to discover from scratch that speech reads left to right.
Analogy: forward-sum is a teacher who gives credit for every reasonable reading of a messy answer sheet; MAS is the final grader who picks the single best reading; the binarization loss is the student learning to write answers clearly enough that both agree.
Numbers worth remembering
- Evaluated on LJSpeech (single speaker) and with several architectures: Tacotron 2, Flowtron, RAD-TTS, FastPitch.
- Reported faster convergence and better robustness on long utterances than the baselines, and FastPitch with the learned aligner matched or beat FastPitch with external durations in their listening tests; check the paper's tables for exact numbers.
Where it shows up today
- NeMo's FastPitch and RAD-TTS use this aligner, and so do many FastPitch-based Indian-language models (Indic TTS (AI4Bharat)) where no forced aligner exists for the language.
- The "CTC-style sum for the loss, Viterbi max for the durations" pattern is a good default whenever you build a small TTS model from scratch; compare plain MAS in VITS (VITS).
- Comparable idea in ASR: alignment by CTC forced alignment (as in WhisperX's word timings, WhisperX).
What it got wrong, and what came next
- Still produces hard, deterministic durations; models still need a duration predictor at inference, and rhythm can be flat. Stochastic duration (VITS, VITS) and implicit alignment (E2 / F5-TTS, E2 TTS, F5-TTS) take other routes.
- Large zero-shot models trained on tens of thousands of hours mostly abandoned explicit alignment, relying on attention plus scale; explicit aligners remain the choice for small, fast, robust models.
Check yourself
Q1What does the forward-sum loss compute, and why use it instead of MAS alone for training?
It computes the negative log of the total probability of all monotonic alignments of the text to the mel, using the CTC forward algorithm over the soft alignment matrix. Unlike MAS, it is differentiable and does not commit to one path while the alignment is still poor, so training is stable.
Q2What is the binarization loss for?
It pulls the soft alignment toward the hard MAS path, so the attention the alignment module learns matches the integer durations used by the rest of the model. Without it the soft and hard alignments can disagree, which hurts duration targets.
Q3Why add a beta-binomial prior?
At the start of training the alignment encoders are random, so the soft alignment is near uniform and learning is slow. The prior concentrates probability near the diagonal, encoding that speech progresses through the text roughly linearly, which speeds up and stabilizes convergence.
Q4What would happen if you used this aligner for a language with no pronunciation lexicon, such as a dialect of Bhojpuri?
It still works, since it learns from text-audio pairs directly, on characters or any tokenization, with no lexicon or pretrained aligner. Quality depends on data size and how consistently the script maps to sound, but the main blocker for MFA-based pipelines disappears.
When you're back at a laptop
- Extend your MAS script from Glow-TTS: compute the forward-sum with
logsumexpin place ofmax, and check it against the hand result (about -3.83) on the 2 × 4 grid. - Implement a beta-binomial prior with
scipy.stats.betabinomfor a 20 × 100 grid and plot it. - Find the aligner module in NeMo's TTS collection (used by FastPitch) and map each part to this page.