AudioLM

AudioLM: a Language Modeling Approach to Audio Generation

Borsos, Marinier, Vincent, Kharitonov, Pietquin, Sharifi, Roblek, Teboul, Grangier, Tagliasacchi, Zeghidour · Google · 2022

Deep read20 minDay 8 of 10#63 of 86

In one sentence

AudioLM generates realistic, coherent audio with no text at all by running three language models in a hierarchy: semantic tokens (k-means on w2v-BERT features) for long-term content, then coarse and fine acoustic tokens (SoundStream RVQ codes) for voice and detail.

Why this paper exists

Before AudioLM there were two ways to model raw audio with language models, and each failed at half the job:

Acoustic tokens only (model codec codes directly): the audio sounds good locally, but over several seconds the content wanders into babble, because the tokens spend most of their capacity on acoustic detail and the model cannot learn linguistic structure from them.
Semantic tokens only (Generative Spoken Language Modeling, GSLM: k-means units of HuBERT features, HuBERT): the content is more coherent, but you cannot get high-quality audio, and speaker identity is lost, because those units deliberately discard voice information.

The insight is that you need both, with a clear division of labour.

The core idea

Split audio into two token types and generate them coarse to fine:

Semantic tokens capture phonetics, words and longer structure. They are low-rate and easy to model over long contexts.
Acoustic tokens capture speaker, prosody detail, recording conditions and everything needed to reconstruct the waveform.

Model the semantic tokens first (long-range coherence), then generate acoustic tokens conditioned on the semantic tokens (local fidelity). Each stage is a standard decoder-only Transformer doing next-token prediction.

How it works

audio prompt (e.g. 3 s)
   |                      |
 w2v-BERT -> k-means    SoundStream RVQ
   |                      |
 semantic tokens        acoustic tokens
 (25 Hz, 1024 units)    (50 Hz, 12 levels)
   |                      |
 [Stage 1] semantic LM: continue semantic
   |
 [Stage 2] coarse LM: semantic -> RVQ 1-4
   |
 [Stage 3] fine LM: RVQ 1-4 -> RVQ 5-12
   |
 SoundStream decoder -> waveform

Semantic tokenizer. Take embeddings from an intermediate layer of a pretrained w2v-BERT (data2vec / w2v-BERT / BEST-RQ, a self-supervised speech model trained with contrastive + masked prediction), cluster them with k-means (1,024 clusters), and use the cluster ID per frame as a token: 25 tokens/s.
Acoustic tokenizer. A SoundStream codec (SoundStream) at 16 kHz, 50 frames/s, 12 RVQ levels of 1,024 (6 kbps).
Stage 1, semantic modelling. p(s_t | s_<t): an autoregressive Transformer over semantic tokens alone. This is where long-term structure (what is being said, in which order) is decided.
Stage 2, coarse acoustic modelling. p(c_t | s, c_<t): generate the first 4 RVQ levels, flattened frame by frame, conditioned on all the semantic tokens. This decides speaker, prosody and recording conditions, copied from the prompt.
Stage 3, fine acoustic modelling. p(f_t | c, f_<t): generate RVQ levels 5-12 conditioned on the coarse tokens. Fine levels depend mostly on local acoustics, so this stage uses short context windows.
Decode all 12 levels with the SoundStream decoder.

Here s are semantic tokens, c coarse acoustic tokens, f fine acoustic tokens, t the position, and _<t means "all earlier positions". All three are trained separately on unlabelled audio (LibriLight, ~60k hours) with ordinary cross-entropy.

Inference uses: speech continuation. Give it a 3-second prompt from an unseen speaker; it continues with plausible, mostly grammatical speech in the same voice, prosody and recording conditions. With piano recordings instead of speech, the same recipe continues piano music coherently.

Build the intuition

Analogy: writing a radio play. First the writer drafts the script (semantic tokens: what is said). Then a director decides who says it and how (coarse acoustic tokens: voice, tone, room). Then the sound engineer adds the fine texture (fine acoustic tokens). A writer who also has to think about microphone hiss will write a worse script, which is why acoustic-only models babble.

Why split coarse and fine? Count tokens for 10 s:

Stream	Rate	10 s
Semantic	25/s	250
Coarse (4 levels × 50)	200/s	2,000
Fine (8 levels × 50)	400/s	4,000

Modelling all 6,000 acoustic tokens in one long context would be expensive and pointless, since fine detail depends only on nearby frames. Splitting lets the coarse model use long context and the fine model short context.

What would happen if you gave Stage 2 a prompt from speaker A but semantic tokens produced from speaker B's speech? You would get B's words in A's voice: semantic tokens carry little speaker information, so voice comes from the acoustic prompt. That is the basis of SPEAR-TTS's "speaking" stage (SPEAR-TTS).

Numbers worth remembering

Semantic tokens: w2v-BERT + k-means, 1,024 clusters, 25 Hz.
Acoustic tokens: SoundStream, 16 kHz, 50 Hz, 12 RVQ levels × 1,024 (6 kbps); coarse = levels 1-4, fine = levels 5-12.
Trained on ~60k hours of unlabelled LibriLight speech.
Human raters asked to tell AudioLM continuations from real speech were right only ~51% of the time (chance), while a trained classifier could still detect them with high accuracy (check the paper for both figures).

Where it shows up today

SPEAR-TTS (SPEAR-TTS) adds text by putting a text-to-semantic model in front of AudioLM's later stages.
VALL-E (VALL-E) uses the same "AR coarse, then the rest" idea over EnCodec tokens, with phonemes standing in for semantic tokens.
CosyVoice (CosyVoice → CosyVoice 2), MaskGCT (MaskGCT) and Spark-TTS (Llasa + Spark-TTS) all go text → semantic tokens → acoustics.
Moshi / Mimi (Moshi) and SpeechTokenizer (SpeechTokenizer + SNAC) fold the semantic and acoustic tokens into one codec, with codebook 1 distilled from an SSL model.
MusicLM uses the same hierarchy for music.

What it got wrong, and what came next

No text control. It only continues audio. SPEAR-TTS (SPEAR-TTS) and VALL-E (VALL-E) add text.
Three models, two tokenizers, slow sequential decoding. SoundStorm (Google, 2023) replaces the acoustic stages with parallel masked decoding; single-codec designs (Mimi, SpeechTokenizer) remove the second tokenizer.
Unsupervised semantic tokens still carry some prosody and speaker information and are not perfectly aligned to phonemes. CosyVoice (CosyVoice → CosyVoice 2) uses ASR-supervised tokens instead.
Content is only locally coherent. Continuations are fluent but the meaning wanders over longer spans; it has no real language understanding. Moshi (Moshi) adds a text stream (inner monologue) from a text LLM.

Check yourself

Q1What are semantic and acoustic tokens, and what does each carry?

Semantic tokens are k-means cluster IDs of self-supervised speech features (here w2v-BERT); they carry phonetic and linguistic content and little about the voice. Acoustic tokens are neural codec codes (here SoundStream RVQ); they carry everything needed to reconstruct the waveform: voice, prosody detail, noise, room. Semantic tokens give coherence, acoustic tokens give quality.

Q2Why not just train one big LM directly on SoundStream tokens?

Acoustic tokens are dominated by local detail, so a model trained on them alone spends its capacity on texture and produces locally realistic but meaningless babble over several seconds. They are also many (600/s here), which makes long contexts expensive. Generating semantic tokens first gives the long-range plan cheaply.

Q3Why does the fine acoustic stage only need short context?

RVQ levels 5-12 encode residual detail that depends mainly on the current coarse frame and its neighbours, not on what was said ten seconds ago. So the fine model can work on short chunks, which keeps the most token-heavy stage cheap.

Q4How does AudioLM keep the prompt speaker's voice in a continuation?

The coarse acoustic stage is conditioned on the prompt's acoustic tokens as its prefix and continues them. Speaker identity and recording conditions live in those acoustic tokens, so the model keeps producing tokens consistent with them, just as a text LM keeps the style of its prompt.

Q5What would you need to add to turn AudioLM into a TTS system?

A model that maps text to semantic tokens. Because semantic tokens are content-like, this is a translation-like task that can be trained on a modest amount of transcribed speech, while the semantic-to-acoustic stages train on untranscribed audio. That is exactly SPEAR-TTS.

Q6Why did the authors also train a classifier to detect AudioLM speech?

Continuations fooled human listeners about half the time, which raises misuse risks such as voice impersonation. They showed a simple classifier could still detect the generated audio reliably, as a mitigation.

When you're back at a laptop

Build semantic tokens yourself: run HuBERT-base (or w2v-BERT 2.0) on an hour of speech, take a middle layer, fit k-means with 500-1,000 clusters, and look at token sequences for two speakers saying the same sentence. They should be similar.
Encode the same clips with EnCodec and compare: the acoustic tokens should differ a lot between speakers.
Read an open re-implementation (e.g. lucidrains' audiolm-pytorch) and map its three transformer classes to the three stages.

arXiv ↗PDF ↗