TTS · Phase 4: End-to-end and expressive TTS
Matcha-TTS
Matcha-TTS: A Fast TTS Architecture with Conditional Flow Matching
In one sentence
Matcha-TTS keeps Grad-TTS's encoder + MAS + "refine a coarse mel" design but trains the decoder with optimal-transport conditional flow matching (OT-CFM), a plain regression onto straight noise-to-data paths, so a small U-Net decoder produces good mels in a handful of ODE steps.
Why this paper exists
Grad-TTS (Grad-TTS) showed that a generative decoder fixes the over-smoothed mels of regression models like FastSpeech 2 (FastSpeech 2), but diffusion needs many solver steps for good quality, each a full U-Net pass. That made it slow and memory-hungry compared with VITS (VITS). Flow matching (Flow Matching, 2022) had just offered a simpler, faster way to train continuous generative models. Matcha is the first clean application to TTS acoustic modelling, and the easiest place to learn flow matching for speech.
The core idea
Instead of learning to reverse a noising SDE, learn a velocity field v_θ(x, t) that moves a noise sample to a data sample along a straight line. Training is regression onto the line's velocity, which is known exactly for each (noise, data) pair. Straight paths can be followed with big Euler steps, so few steps are enough at inference.
How it works
phonemes
│
▼
text encoder (Transformer, rotary pos.)
│ μ per phoneme (80 dims)
├──► duration predictor (learns MAS durations)
▼
MAS align (train) / predicted durations (infer)
│ μ per frame (80 x T) ── prior loss vs mel
▼
x_0 ~ N(0, I) (80 x T) ─┐
▼
decoder v_θ(x_t, t | μ): 1D U-Net
(ResNet blocks + Transformer blocks)
│ Euler ODE, N steps
▼
mel x_1 (80 x T)
▼
vocoder (HiFi-GAN)- Encoder and alignment: from Glow-TTS / Grad-TTS (Glow-TTS, Grad-TTS), with rotary position embeddings. MAS finds the monotonic alignment during training; a duration predictor learns its durations for inference.
- Prior loss: the aligned encoder output
μshould be close to the target mel (a Gaussian negative log-likelihood), as in Grad-TTS. - Decoder: a 1D U-Net mixing ResNet blocks and Transformer blocks, taking the current point
x_t, the timetand the conditionμ(concatenated). Unlike Grad-TTS, the noise starts fromN(0, I), not centered onμ;μis a condition. - Total loss:
L = L_dur + L_prior + L_OT-CFM.
The OT-CFM loss
x_0 ~ N(0, I) (noise, 80 x T)
x_1 = target mel (data, 80 x T)
t ~ Uniform(0, 1)
x_t = (1 - (1 - σ_min)·t)·x_0 + t·x_1
u = x_1 - (1 - σ_min)·x_0
L_OT-CFM = E_{t, x_0, x_1} || v_θ(x_t, t | μ) - u ||²x_t is a point on the (almost) straight line from noise to data at time t; u is that line's constant velocity; v_θ is the decoder's predicted velocity given x_t, t and the encoder condition μ; σ_min is a tiny constant (around 1e-4) that keeps a sliver of noise at t = 1. With σ_min = 0 this is exactly the flow matching loss from Flow Matching: x_t = (1 - t)·x_0 + t·x_1, u = x_1 - x_0.
Inference
x = sample N(0, τ²·I) (τ = temperature)
h = 1 / N
for k in 0 .. N-1:
x = x + h · v_θ(x, k·h | μ)
mel = xN is the number of function evaluations (NFE). The paper compares small N like 2, 4 and 10.
Build the intuition
A 1D example
Noise sample x_0 = -1, data x_1 = 3, ignore σ_min. At t = 0.25: x_t = 0.75·(-1) + 0.25·3 = 0, and the target velocity is u = 3 - (-1) = 4. At every t on this line the target is 4, so if the model learned it perfectly for this pair, one Euler step of size 1 from -1 lands exactly on 3.
Why then do we need more than one step? The model sees many (noise, data) pairs whose lines cross. At a crossing point it can only predict the average velocity of all lines through it, so the learned field curves. A few steps follow the curve well; one step overshoots toward the average, which sounds blurrier. Conditioning on μ narrows the set of plausible mels, so lines cross less and few steps suffice.
Analogy: Diffusion is walking home through fog, taking many small careful steps that each correct the last. Flow matching first draws straight roads from every starting point to every house, then learns a sign at each crossroad pointing roughly the right way. With good signs you get home in a few long strides.
Diffusion vs flow matching in one table
| Grad-TTS (diffusion) | Matcha (OT-CFM) | |
|---|---|---|
| Learns | score of noisy mels | velocity along straight paths |
| Start point | N(μ, I) | N(0, I), μ as condition |
| Paths | curved (SDE) | near-straight |
| Typical steps | tens | 2-10 |
Numbers worth remembering
- Trained and evaluated on LJSpeech (~24 hours, one English speaker), with HiFi-GAN as the vocoder.
- Decoder steps compared: small NFE such as 2, 4 and 10 (check the paper's tables for MOS and RTF at each).
- Reported faster synthesis and lower memory than Grad-TTS, with speed comparable to fast non-autoregressive baselines and strong MOS (check Table 1 for exact values).
- Around ~18M parameters (check the paper); comfortably fits on your 4 GB RTX 3050 for training.
Where it shows up today
- Flow matching is now the dominant acoustic-model objective: Voicebox (Voicebox), E2 TTS (E2 TTS), F5-TTS (F5-TTS) and CosyVoice's token-to-mel stage (CosyVoice → CosyVoice 2), which uses a Matcha-style conditional flow-matching decoder (check the CosyVoice paper).
- Matcha itself is a practical small TTS: open code, a few ODE steps, easy to train on a new single-speaker dataset (for example a Hindi or Tamil voice).
- The few-step idea continues toward one-step generators: consistency in CALM (CALM (Pocket TTS)) and drifting (Drifting Models).
What it got wrong, and what came next
- Still needs MAS, phonemes and a duration predictor; E2 and F5-TTS (E2 TTS, F5-TTS) remove explicit alignment and scale to zero-shot cloning on large data.
- Single speaker (or ID-based multi-speaker) in the paper; no prompt-based cloning. Voicebox (Voicebox) turns flow matching into speech infilling for zero-shot TTS.
- Mel output still needs a vocoder (HiFi-GAN, iSTFTNet + Vocos).
- Offline per sentence; chunk-aware streaming flow matching came with CosyVoice 2 (CosyVoice → CosyVoice 2).
Check yourself
Q1Write the OT-CFM training target and explain each term.
x_t = (1 - (1 - σ_min)t)·x_0 + t·x_1 and u = x_1 - (1 - σ_min)·x_0, with loss || v_θ(x_t, t | μ) - u ||². x_0 is Gaussian noise, x_1 the target mel, t uniform in [0, 1], μ the aligned encoder output, and σ_min a tiny noise floor. The network regresses the velocity of the straight path from noise to data.
Q2Why does flow matching need fewer steps than diffusion?
It is trained toward nearly straight paths with constant velocity, which an Euler solver follows accurately with large steps. Diffusion's reverse process follows curved, noisy trajectories that need many small steps to integrate well.
Q3How does Matcha's starting distribution differ from Grad-TTS?
Matcha starts from standard Gaussian noise and gives μ to the decoder as a condition. Grad-TTS starts from noise centered on μ and its SDE converges to N(μ, I).
Q4What would happen at NFE = 1?
The decoder predicts one velocity from pure noise and jumps all the way. Because the learned field averages over many crossing paths, the result is close to the mean of plausible mels, so it sounds smoother and less detailed, though still intelligible thanks to the conditioning.
Q5What are the three losses in Matcha-TTS?
A duration loss (the predictor regresses MAS-derived log durations), a prior loss (the aligned encoder output should match the mel under a Gaussian), and the OT-CFM loss for the decoder.
Q6What does the temperature τ do at inference?
It scales the initial noise. Lower τ gives more conservative, consistent output; higher τ gives more variation but risks artifacts, similar to VITS's noise scale.
When you're back at a laptop
- Implement OT-CFM on 2D two-moons in ~30 lines: an MLP
v(x, t), the loss above, then sample with 1, 4 and 16 Euler steps and plot.
x1 = sample_moons(B); x0 = torch.randn_like(x1)
t = torch.rand(B, 1); s = 1e-4
xt = (1 - (1 - s) * t) * x0 + t * x1
u = x1 - (1 - s) * x0
loss = ((model(xt, t) - u) ** 2).mean()pip install matcha-tts, run the LJSpeech checkpoint with--steps 2,4,10and compare audio and time on your 3050 and CPU.- Read
matcha/models/components/flow_matching.pyand match it line by line to the loss above.