Reading Room

ASR · Phase 5: Scale and multilinguality

Distil-Whisper

Distil-Whisper: Robust Knowledge Distillation via Large-Scale Pseudo Labelling

Gandhi, von Platen, Rush · Hugging Face · 2023

Skim7 minDay 3 of 10#18 of 86

In one sentence

Distil-Whisper keeps Whisper's full encoder, cuts the decoder to 2 layers, and trains it on Whisper-large's own transcripts (pseudo-labels) filtered by a WER threshold, giving an English model several times faster at nearly the same accuracy.

Why this paper exists

Whisper Whisper large is accurate but slow. Its decoder is autoregressive: one token at a time, each step through all 32 decoder layers. The encoder runs only once per 30 s window, so the decoder dominates latency. Training a small model from scratch would need Whisper-scale labelled data, which is not public, and open human-labelled corpora use inconsistent transcript styles.

The core idea

Keep parameters where they run once (the encoder) and remove them where they run per token (the decoder). Let the teacher label lots of open audio, giving consistent targets, and discard clips where the teacher was probably wrong.

How it works

  1. Student. Copy and freeze Whisper-large's encoder. Build a 2-layer decoder initialised from teacher decoder layers (the first and last).
  2. Pseudo-labels. Run Whisper-large over ~22k hours of open English audio; its outputs (model-made transcripts) become targets.
  3. WER filter. Compare each pseudo-label with the human transcript; drop the clip if WER (word error rate) is above a threshold (~10%). This removes teacher hallucinations (text not in the audio).
  4. Loss. L = a * L_PL + b * L_KL: cross-entropy on pseudo-label tokens plus KL divergence between teacher and student next-token distributions, with weights a, b (check the paper for values).
  5. Speculative decoding. The student drafts tokens and the teacher verifies them in one pass: identical output to the teacher, about 2x faster.

Build the intuition

Analogy: a senior engineer reads a spec once; the junior reads it through the senior's eyes (the shared encoder) but writes the summary fast. Keep the expensive reading, shrink the repeated writing.

If you distilled the encoder instead, a 30-token transcript would still need 30 passes through 32 layers. Autoregressive speed scales with decoder depth times output length.

Numbers worth remembering

  • distil-large-v2: ~756M parameters vs ~1.55B for Whisper large-v2.
  • ~5.8x faster, within ~1% WER on out-of-distribution short-form test sets.
  • ~22k hours of pseudo-labelled English audio.
  • 2 decoder layers vs 32; encoder unchanged.

Where it shows up today

  • distil-large-v3 and distil-small.en are common defaults in Transformers pipelines and faster-whisper / WhisperX WhisperX setups. Half the size helps fit a 4 GB RTX 3050.
  • The pseudo-label-and-filter recipe reappears in Canary Canary and Granary Granary.
  • Draft-model speculative decoding is now standard in LLMs and audio LLMs SALMONN / Qwen-Audio / Qwen2-Audio.

What it got wrong, and what came next

  • English only. For Indian languages you would redo pseudo-labelling with a good Indic teacher (see Vistaar).
  • Not streaming: same 30 s encoder-decoder design; see Cache-aware streaming Conformer, Kyutai DSM.
  • Capped by the teacher. Canary Canary shows curated data can beat Whisper outright.

Check yourself

Q1Why shrink the decoder rather than the encoder?

The encoder runs once per window; the decoder runs once per token. Going from 32 to 2 decoder layers removes most latency, while the full encoder keeps Whisper's robust acoustic representation.

Q2What does the WER filter do?

It drops clips whose pseudo-label disagrees strongly with the human transcript. That removes hallucinated or misheard labels, so the student does not learn them, which helps explain its lower hallucination rate.

Q3Why add a KL term to the pseudo-label loss?

KL matches the student to the teacher's full next-token distribution, which carries more information than one hard label per step, such as which alternatives were plausible.

Q4How can the student speed up the teacher without changing its output?

Speculative decoding: the student drafts several tokens and the teacher checks them in one forward pass, keeping the agreed prefix. The teacher approves every token, so output equals the teacher's greedy output.

When you're back at a laptop

  • Transcribe one clip with openai/whisper-large-v3 and distil-whisper/distil-large-v3; time both and diff the text.
  • Pass assistant_model= to generate() and confirm the output matches the teacher.
  • Feed both 30 s of silence and compare hallucinations.
arXiv ↗PDF ↗