TTS · Phase 7: Flow matching and diffusion at scale
MaskGCT
MaskGCT: Zero-Shot Text-to-Speech with Masked Generative Codec Transformer
In one sentence
MaskGCT is a two-stage non-autoregressive TTS that generates discrete tokens by iterative parallel unmasking (MaskGIT-style): text to semantic tokens, then semantic tokens to acoustic codec tokens, with no explicit alignment or duration model.
Why this paper exists
By 2024 there were two main zero-shot TTS families, each with a cost:
- Autoregressive codec LMs (VALL-E VALL-E, CosyVoice CosyVoice → CosyVoice 2): natural prosody, but slow, token by token, and prone to skipping or repeating words.
- NAR flow/diffusion over mel (Voicebox Voicebox, E2/F5 E2 TTS, F5-TTS): fast and robust, but working on continuous mel with a separate vocoder.
MaskGCT asks: can you keep discrete tokens (with the semantic/acoustic split that AudioLM AudioLM made standard) but generate them in parallel?
The core idea
Masked generative modelling: train a bidirectional Transformer to predict randomly masked tokens given the visible ones. At inference, start with every target position masked, predict all of them, keep the most confident predictions, re-mask the rest, and repeat for a fixed number of rounds. Each round fills in more tokens, in parallel. This is the MaskGIT recipe from image generation.
How it works
MaskGCT applies it twice:
text + prompt semantic tokens
-> [T2S masked transformer] -> semantic tokens
semantic + prompt acoustic tokens
-> [S2A masked transformer] -> acoustic tokens (RVQ)
-> codec decoder -> waveform- Semantic tokens: vector-quantized w2v-BERT 2.0 features (data2vec / w2v-BERT / BEST-RQ for w2v-BERT), which carry mostly content and some prosody.
- Acoustic tokens: from a residual-VQ codec; the S2A stage generates the codebook layers, coarse to fine (check the paper for layer count and scheduling).
- No alignment: the T2S stage is told only the total length of the target token sequence (like E2/F5) and learns where the text goes.
- The prompt's tokens are prefixed as context, which gives zero-shot cloning.
Build the intuition
Analogy: a crossword solved in passes. First pass, fill only the answers you are sure of. Those letters constrain the rest, so the second pass fills more, and so on. Autoregressive decoding is filling the grid strictly left to right, one letter at a time.
What would happen with just one unmasking round? Every token is predicted independently given only the text and prompt, so neighbouring tokens may disagree (two plausible pronunciations mixed together), much like 1-step flow matching averaging modes (Flow Matching). More rounds let early choices condition later ones.
Numbers worth remembering
- Trained on Emilia (LibriTTS → Emilia), ~100k hours of multilingual in-the-wild speech (check the paper).
- Two stages, each a non-autoregressive masked Transformer; the number of unmasking steps per stage is a quality/speed knob (tens of steps; check the paper).
- Released openly through the Amphion toolkit.
Where it shows up today
- A reference design for "discrete tokens, but parallel", alongside F5-TTS as the continuous NAR design.
- The semantic-then-acoustic split matches AudioLM (AudioLM), SPEAR-TTS (SPEAR-TTS) and CosyVoice (CosyVoice → CosyVoice 2).
- Masked parallel decoding of codebooks also appears in other audio generators (e.g. MaskGIT-style decoders in music models).
What it got wrong, and what came next
- Two stages, two models: more moving parts than F5-TTS's single model.
- Total length must be chosen, as with E2/F5.
- Not streaming: bidirectional attention over the whole utterance. For voice agents see DSM (Delayed Streams Modeling (TTS)) and CALM (CALM (Pocket TTS)).
- Discrete tokens are lossy: quality is capped by the codec's bitrate, the problem CALM attacks with continuous latents (CALM (Pocket TTS)).
Check yourself
Q1How does MaskGIT-style decoding work at inference?
Start with all target tokens masked. Predict every masked position in parallel, keep the most confident ones, re-mask the rest, and repeat for a fixed number of rounds with a schedule that unmasks more each round. It needs far fewer passes than one per token.
Q2What are the two stages and what token type does each produce?
Text-to-semantic produces semantic tokens from quantized w2v-BERT 2.0 features; semantic-to-acoustic produces the residual-VQ codec tokens that the codec decoder turns into a waveform.
Q3How is MaskGCT similar to F5-TTS, and how is it different?
Both are non-autoregressive, avoid explicit alignment by fixing only the total length, and clone from a prompt given as context. MaskGCT generates discrete tokens with masked prediction in two stages; F5-TTS generates a continuous mel with flow matching in one stage plus a vocoder.
Q4Why do several unmasking rounds beat one?
One round predicts all tokens independently, so they can be mutually inconsistent. Later rounds condition on tokens fixed earlier, so the sequence becomes coherent.
When you're back at a laptop
- Run MaskGCT from Amphion with your own prompt; time it against F5-TTS on the same sentence.
- Vary the number of T2S steps (e.g. 10, 25, 50) and note where quality stops improving.