VALL-E

Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers

Wang, Chen, Wu, Zhang, Zhou, Liu, et al. · Microsoft · 2023

Deep read22 minDay 8 of 10#64 of 86

In one sentence

VALL-E treats TTS as language modelling over EnCodec tokens: given the phonemes of the text and the codec tokens of a 3-second recording, an autoregressive Transformer writes the first codebook and a non-autoregressive Transformer fills in codebooks 2-8, so the output continues the recording in the same voice, emotion and acoustic environment.

Why this paper exists

Before VALL-E, zero-shot voice cloning (speaking in a voice never seen in training, from a short sample) was done by conditioning a mel-spectrogram TTS on a speaker embedding from a speaker-verification model: SV2TTS (SV2TTS), YourTTS (YourTTS). Two things limited it:

Data. TTS models were trained on tens to hundreds of hours of clean studio speech (LibriTTS is ~585 hours). That is not enough speakers to generalize to arbitrary new voices.
The speaker embedding is a bottleneck. A single vector captures roughly "who" but not the prompt's emotion, microphone, room or speaking style. Clones sounded like a generic nearby voice.

Meanwhile text LLMs had shown that a big model trained on lots of messy data with next-token prediction learns to continue a prompt in its style (in-context learning). Neural codecs (SoundStream, EnCodec) had just made audio into tokens. VALL-E connects the two.

The core idea

Treat the voice prompt like a text prompt. Turn the 3-second recording into EnCodec tokens, put the phonemes of (prompt transcript + target text) in front, and ask a decoder-only Transformer to continue the token sequence. Because the prompt's tokens encode everything about the recording, the continuation keeps the speaker, emotion and acoustics, with no speaker embedding at all.

Then train on far more data than any TTS before: ~60k hours of LibriLight audiobooks, transcribed automatically by an ASR model. Messy, but with 7,000+ speakers.

How it works

EnCodec at 6 kbps gives 8 codebooks × 75 frames/s. Generating 8 tokens per frame autoregressively would be 600 sequential steps per second, so VALL-E splits the job:

phonemes x (prompt text + target text)
        +  prompt tokens C~ (3 s = 225 frames x 8)

  AR model (codebook 1 only, left to right)
  in:  x, C~[:,1], c[<t,1]
  out: c[t,1]   for t = 1..T, until EOS
        |
        v   c[:,1]  (T tokens)
  NAR model, run 7 times (j = 2..8)
  in:  x, C~[:, 1..8], c[:, 1..j-1] (summed)
       + stage embedding for j
  out: c[:, j] for ALL t in parallel
        |
        v   c[:, 1..8]  (T x 8 tokens)
  EnCodec decoder -> 24 kHz waveform

The factorization:

p(C | x, C~) = p(c_{:,1} | x, C~_{:,1})        (AR)
  × Π_{j=2..8} p(c_{:,j} | c_{:,<j}, x, C~)  (NAR)

AR term = Π_{t=1..T} p(c_{t,1} | c_{<t,1}, C~_{:,1}, x)

x = phoneme sequence (from a grapheme-to-phoneme converter) of prompt transcript plus target text.
C~ = EnCodec tokens of the prompt, shape [T', 8].
c_{t,j} = token at frame t in codebook j of the output; C is the full [T, 8] output.
T = output length in frames, chosen by the AR model when it emits an end-of-sequence token. That is how VALL-E decides duration with no duration model.

The AR model is a decoder-only Transformer over [phonemes ; prompt codebook-1 tokens ; generated codebook-1 tokens], with causal attention. Codebook 1 holds the coarsest, most content-and-prosody-heavy information, so the AR model decides the words, timing and intonation.

The NAR model has the same size but full (non-causal) attention. For stage j it sees the phonemes, the prompt's tokens in all 8 codebooks (for voice detail), and the sum of the embeddings of codebooks 1..j-1 of the output; it predicts codebook j for every frame at once. A learned stage embedding (injected via adaptive layer norm) tells it which j it is on. So the whole output needs T AR steps plus 7 NAR passes.

Model size: both are 12-layer Transformers with 16 heads and 1,024 dimensions (check the paper for feed-forward size). Decoding: sampling for the AR model (beam search tended to loop), greedy for the NAR model.

Build the intuition

Analogy: a band covering a song. The lead singer (AR model) performs the melody line note by note, deciding timing and phrasing. Then the backing musicians (NAR model) each fill in their part for the whole song at once, listening to the melody and to the original recording for the sound they should match.

Count the steps for a 5-second sentence:

Approach	Sequential steps
Flatten all 8 codebooks, all AR	5 × 75 × 8 = 3,000
VALL-E: AR codebook 1 + 7 NAR passes	375 + 7 = 382
Mimi-based model at 12.5 Hz (per-frame depth decoder)	63 backbone steps

Why does it skip or repeat words? Nothing ties phoneme i to frames t1..t2. The AR model must learn alignment implicitly through attention, as Tacotron did (Tacotron 2). With sampling, it can lose its place: jump over a word, loop on a syllable, or never emit end-of-sequence. Explicit durations (FastSpeech, FastSpeech 2) and monotonic alignment (MAS, Glow-TTS) prevented this in earlier TTS; VALL-E traded that robustness for in-context cloning.

What would happen if the prompt is a noisy phone call? VALL-E reproduces the noise and the telephone bandwidth, because those are in the prompt tokens. Great for realism, bad for a voice agent that needs clean audio from a user-supplied sample.

Numbers worth remembering

~60k hours (LibriLight, ASR-transcribed), 7,000+ speakers, English only.
EnCodec 6 kbps: 8 codebooks × 75 Hz; prompt 3 s = 225 frames.
AR and NAR models: 12 layers, 16 heads, 1,024 dims each.
LibriSpeech test-clean zero-shot (check Table 2): WER ~5.9 vs YourTTS ~7.7, speaker similarity ~0.58 vs YourTTS ~0.34.
One AR pass over codebook 1 plus 7 NAR passes.

Where it shows up today

VALL-E X (cross-lingual) and VALL-E 2 (repetition-aware sampling, grouped code modelling) from Microsoft; ELLA-V and RALL-E add alignment to fix skipping.
SpeechTokenizer's USLM (SpeechTokenizer + SNAC) is VALL-E with a semantic first codebook.
The AR-for-coarse, parallel-for-fine pattern became Moshi's and Sesame CSM's backbone + depth decoder (Moshi, Sesame CSM), and MusicGen's delay pattern.
XTTS (XTTS), CosyVoice (CosyVoice → CosyVoice 2), Seed-TTS (Seed-TTS), Llasa / Spark-TTS / Orpheus (Llasa + Spark-TTS) are all "LLM over speech tokens + prompt continuation" descendants.

What it got wrong, and what came next

Skipping, repeating, not stopping (no explicit alignment): VALL-E 2's repetition-aware sampling; CosyVoice (CosyVoice → CosyVoice 2) with supervised semantic tokens; non-autoregressive F5-TTS (F5-TTS) and MaskGCT (MaskGCT) remove AR decoding entirely.
Slow decoding (one AR step per 13 ms of audio): lower frame rates (Mimi 12.5 Hz), and flow-matching NAR models like Voicebox (Voicebox).
Lossy, entangled acoustic tokens: semantic + flow matching designs (CosyVoice) and continuous-latent models (MELLE MELLE, CALM CALM (Pocket TTS)).
Prompt sensitivity and copying noise: better data (Emilia, LibriTTS → Emilia) and stronger acoustic models.
English only, audiobook style: VALL-E X for cross-lingual; Emilia and IndicVoices-R (IndicVoices-R) for spontaneous and Indian-language data.
Misuse risk: the paper did not release the model, citing impersonation concerns.

Check yourself

Q1How does VALL-E clone a voice from 3 seconds without a speaker embedding?

The prompt is converted to EnCodec tokens and placed in the sequence before the tokens to be generated, with the prompt's transcript in the phoneme input. The model is trained to continue codec token sequences, so it continues in whatever voice, emotion and acoustic conditions the prefix tokens encode. Cloning is in-context learning, like a text LLM continuing in the style of its prompt.

Q2Why does VALL-E need both an AR and a NAR model?

Codebook 1 carries content, timing and prosody, which depend strongly on what came before, so it is generated autoregressively; that is also how output length is decided. Codebooks 2-8 refine acoustics given codebook 1 and can be predicted for all frames at once. Generating all 8 codebooks autoregressively would be 8x more sequential steps (600 per second instead of 75).

Q3Why does VALL-E sometimes skip or repeat words, and why did FastSpeech not have this problem?

VALL-E has no explicit text-to-frame alignment: the AR model's attention must implicitly track which phoneme it is on, and sampling can make it lose its place or loop. FastSpeech predicts a duration per phoneme and expands each phoneme exactly that many frames, so every phoneme is spoken exactly once by construction.

Q4What tells the NAR model which codebook it is predicting?

A learned stage embedding for j (2 to 8), injected through adaptive layer normalization. The same weights serve all 7 stages; the input is the sum of embeddings of the codebooks already known.

Q5How long is the AR sequence for a 4-second output with a 3-second prompt and 40 phonemes?

About 40 phonemes + 225 prompt frames + 300 output frames (4 × 75) ≈ 565 positions, of which the model generates the last ~300 one at a time. The NAR passes each handle the 300 output frames in parallel.

Q6What would happen if you used EnCodec at 1.5 kbps (2 codebooks) to make VALL-E faster?

The AR part would not get faster at all, because it still generates 75 codebook-1 tokens per second. You would save only 6 of the 7 NAR passes, which are cheap anyway, and lose a lot of audio quality and speaker similarity. To speed up the AR part you need a lower frame rate.

Q7Why was 60k hours of ASR-transcribed audio better than a few hundred hours of clean TTS data?

Zero-shot cloning needs the model to have seen huge speaker and condition diversity; 7,000+ speakers in varied conditions teach it to generalize to new voices. The transcripts are noisy, but the language-model objective tolerates noise, the same lesson Whisper (Whisper) showed for ASR.

When you're back at a laptop

Run an open VALL-E reimplementation (for example the lifeiteng/vall-e repo or Amphion's VALL-E) on a short prompt of your own voice.
Print the AR input sequence for one example and identify phonemes, prompt tokens and generated tokens.
Try a long, repetitive sentence ("the the cat cat sat") and a very short prompt; listen for skipped or repeated words.
Count time spent in the AR loop versus the 7 NAR passes on your RTX 3050.

arXiv ↗PDF ↗