HuBERT

HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units

Hsu, Bolte, Tsai, Lakhotia, Salakhutdinov, Mohamed · Meta AI · 2021

Deep read18 minDay 2 of 10#14 of 86

In one sentence

HuBERT makes BERT-style pretraining work for speech by creating its own targets offline: it clusters audio frames with k-means, trains the model to predict the cluster IDs of masked frames, then re-clusters the model's own features and repeats, and the resulting discrete "hidden units" became the standard semantic speech tokens.

Why this paper exists

wav2vec 2.0 (wav2vec 2.0) showed that self-supervised pretraining gives strong ASR from very little labelled data, but its training is delicate. It learns the target vocabulary (a Gumbel-softmax quantizer) at the same time as the model that predicts it, and keeps the two from collapsing with a diversity loss, a temperature schedule and a contrastive loss that depends on how negatives are sampled.

BERT's recipe for text is far simpler: fixed vocabulary, mask tokens, cross-entropy. Speech has no fixed vocabulary, though. The question HuBERT asks: can you just make up a vocabulary offline, even a crude one, and train with plain masked cross-entropy?

The core idea

Targets do not have to be correct; they have to be consistent. Cluster the frames with k-means on simple features such as MFCCs (mel-frequency cepstral coefficients, a classic 39-dimensional spectral summary per frame). Each frame gets a cluster ID from 0 to 99. These IDs only loosely track phones, but the same sound tends to land in the same cluster.

Then train BERT-style: mask spans of the input and predict the cluster IDs only at masked positions. To do that the model must infer what is hidden from the surrounding context, so it learns acoustic and language-like structure that is better than the targets it was trained on. Then cluster the trained model's hidden features. These clusters are much more phonetic. Retrain on the new targets. Each round, better features give better targets.

How it works

iteration 1: MFCC --k-means(100)--> targets
iteration 2: layer-6 feats --k-means(500)--> targets

waveform --> CNN encoder (20 ms frames)
                |
          mask spans of frames
                |
          Transformer encoder
                |  o_t per frame
          projection A
                |
   softmax over cluster embeddings e_c
                |
   cross-entropy vs cluster ID z_t
   (masked frames only)

Architecture

The network is the same as wav2vec 2.0: a 7-layer CNN on the raw waveform producing 50 frames/s, then a Transformer. Sizes: BASE (12 layers, ~95M parameters), LARGE (24 layers, ~317M) and X-LARGE (48 layers, ~1B).

Targets

Iteration 1: k-means with 100 clusters on MFCC features. BASE is trained on these.
Iteration 2: take the iteration-1 BASE model, extract features from an intermediate Transformer layer (layer 6), and run k-means with 500 clusters. Train a fresh BASE model on these targets.
LARGE and X-LARGE are trained on clusters of features from the iteration-2 BASE model (the paper uses layer 9) rather than starting from MFCCs.

The paper evaluates target quality against forced-aligned phone labels with phone purity and PNMI (phone-normalised mutual information: how much knowing the cluster tells you about the phone). Clusters of learned features score far higher than MFCC clusters, and intermediate layers are better than the last layer, which specialises toward the pretraining task.

Masking and loss

Masking works as in wav2vec 2.0: random span starts, spans of 10 frames, masked frames replaced by a learned embedding. The prediction for frame t:

p(c | x̃, t) = exp(sim(A o_t, e_c) / τ)
              / Σ_{c'=1..C} exp(sim(A o_t, e_c') / τ)

x̃: the masked input sequence.
o_t: Transformer output at frame t.
A: a learned projection matrix.
e_c: a learned embedding for cluster c. C is the number of clusters (100 or 500).
sim: cosine similarity; τ = 0.1.

The loss is cross-entropy against the true cluster ID z_t, weighted between masked frames (set M) and unmasked ones:

L = α L_m + (1 - α) L_u
L_m = -Σ_{t in M} log p(z_t | x̃, t)

The best setting is α = 1: compute the loss only on masked frames. Predicting unmasked frames just teaches the model to copy the clustering, including its mistakes. Predicting masked frames forces the model to infer the content from context.

Fine-tuning

Same as wav2vec 2.0: drop the prediction head, add a linear CTC layer (CTC) over characters, and fine-tune on labelled data with the CNN frozen.

Build the intuition

Analogy: a teacher with a sloppy answer key. A student learns by filling in blanked-out words in passages and checking the answers against the key, which has some errors. Because the key's mistakes are random rather than consistent, and the student must use context to answer, the student ends up understanding the language better than the key. Next term, the student writes a new, better key, and the class starts over with it.

Why does predicting noisy targets work? Suppose MFCC cluster 37 catches mostly /s/, along with some /sh/ and some noise. To predict "37" behind a mask, the model has to work out from context that an /s/-like sound probably belongs there, and to do that it has to model words and phone sequences. The noise in the labels averages out; what is consistent in them is phonetic. The paper also shows that the consistency of targets (the same sound always mapped to the same ID) matters more than their absolute accuracy.

What if you used a single huge codebook from the start? With MFCCs, more clusters mostly means splitting on speaker and channel detail rather than on phones. Better features to cluster, not more clusters, is what improves the targets.

Numbers worth remembering

Iteration 1: 100 clusters on MFCCs. Iteration 2: 500 clusters on layer-6 features.
Sizes: BASE ~95M, LARGE ~317M, X-LARGE ~1B parameters. The latter two are pretrained on Libri-Light 60k hours.
Matches or beats wav2vec 2.0 on every Libri-Light fine-tuning split (10 min, 1 h, 10 h, 100 h, 960 h).
The ~1B X-LARGE model shows up to 19% and 13% relative WER reduction on dev-other and test-other compared with the previous state of the art.
Frame rate 50 Hz, so 1 s of speech is 50 discrete units (often deduplicated to fewer when used as tokens).

Where it shows up today

Discrete speech tokens for generation: Generative Spoken Language Modelling (GSLM, also Meta 2021) trains a language model on HuBERT units, and that is the template for "textless" speech LMs. SpeechTokenizer (SpeechTokenizer + SNAC) distils HuBERT into the first layer of its residual codec so that layer carries content. AudioLM (AudioLM) uses the same idea with w2v-BERT (data2vec / w2v-BERT / BEST-RQ) units as its "semantic" tokens.
Semantic vs acoustic tokens: the split between content-like SSL tokens and codec tokens (SoundStream, EnCodec) comes from this line of work. Moshi's Mimi codec (Moshi) distils WavLM, HuBERT's successor, into its first codebook.
Direct successor: WavLM (WavLM) keeps HuBERT's masked-prediction recipe and adds denoising.
Features: HuBERT layers are common inputs for voice conversion, speech resynthesis and probing studies.

What it got wrong, and what came next

Multiple expensive iterations. Every round means pretraining a model, extracting features over the whole dataset, and running k-means at scale. BEST-RQ (data2vec / w2v-BERT / BEST-RQ) removes all of this with a frozen random quantizer; data2vec (data2vec / w2v-BERT / BEST-RQ) uses an EMA teacher (a moving average of the student's own weights) that improves continuously instead of in rounds.
Clean, single-speaker bias. Trained on audiobooks, it is weaker on noisy, overlapped and speaker-related tasks. WavLM (WavLM) adds simulated noise and overlapping speakers.
Units are not pure phones. HuBERT units still carry some speaker and prosody information, and they differ across k-means runs. Speech-LLM and TTS work since then has moved partly toward supervised ASR-derived tokens (for example CosyVoice, CosyVoice → CosyVoice 2) or codec tokens with distilled semantics.
Offline and English-centric in the original release; multilingual variants (mHuBERT and others) came later.

Check yourself

Q1Where do HuBERT's targets come from in the first iteration, and in the second?

First iteration: k-means with 100 clusters on MFCC features of the training audio. Second: k-means with 500 clusters on intermediate-layer (layer 6) features of the model trained in iteration 1. Learned features cluster far more phonetically than MFCCs, so the second set of targets is better.

Q2Why is the loss computed only on masked frames?

On unmasked frames the model can see the audio and just learns to imitate the k-means assignment, errors included. On masked frames it must infer the target from context, which forces it to learn acoustic and sequential structure. Ablations show that masked-only loss (α = 1) works best, especially with noisy targets.

Q3How can a model trained on noisy labels end up better than the labels?

The noise in k-means labels is not systematic with respect to context, but the phonetic signal in them is. To predict masked labels the model learns the consistent structure, roughly which sounds follow which, and averages out the noise. Its hidden features therefore encode phones better than the MFCC clusters, which is why re-clustering them improves the targets.

Q4Compare HuBERT's objective to wav2vec 2.0's in one line each.

wav2vec 2.0: pick the true learned-quantizer unit among ~100 same-utterance distractors with a contrastive loss, plus a diversity loss to keep the codebook in use. HuBERT: classify each masked frame into one of C fixed, offline k-means clusters with cross-entropy. HuBERT has no negatives to sample and no codebook that can collapse.

Q5Why do people take HuBERT units from a middle layer rather than the last one?

The last layers specialise toward the pretraining task, predicting the specific cluster IDs, while middle layers hold the most phonetic, abstract information. Phone purity and PNMI peak in the middle layers, so middle-layer clusters make the best targets and the best tokens.

Q6You want discrete tokens for a Hindi speech LM. What would happen if you used English-only HuBERT BASE units?

They would partly work, since many phones are shared, but Hindi-specific contrasts such as aspirated vs unaspirated and retroflex vs dental stops might merge into the same clusters, and the units would be noisier on accented or telephone audio. You would get better tokens from a multilingual or Indic SSL model and k-means fitted on Hindi data.

When you're back at a laptop

Load facebook/hubert-base-ls960 with transformers and extract layer-6 hidden states (output_hidden_states=True) for a few minutes of your own speech.
Fit sklearn.cluster.MiniBatchKMeans(n_clusters=100) on those frames, then map a new utterance to unit IDs. Deduplicate repeats (5 5 5 12 12 becomes 5 12) and look at how many units per second you get.
Say the same word twice, in two voices if you can, and compare the two unit sequences. Similar content should give similar sequences.
Stretch: compare clusters from MFCCs against clusters from layer 6 by eye on a short clip.

arXiv ↗PDF ↗