Reading Room

ASR · Phase 5: Scale and multilinguality

Whisper

Whisper: Robust Speech Recognition via Large-Scale Weak Supervision

Radford, Kim, Xu, Brockman, McLeavey, Sutskever · OpenAI · 2022

Deep read25 minDay 3 of 10#17 of 86

In one sentence

Whisper skips self-supervision and trains a plain Transformer encoder-decoder on 680,000 hours of imperfectly transcribed internet audio, with special tokens in the decoder prompt choosing language, task (transcribe or translate) and timestamps; the result is a single model that works zero-shot on almost any dataset, at the cost of slow autoregressive decoding and occasional hallucination.

Why this paper exists

By 2022 the best LibriSpeech numbers came from self-supervised models such as wav2vec 2.0 (wav2vec 2.0) fine-tuned on that dataset. They looked superhuman on LibriSpeech test-clean but broke on phone calls, accents, noisy video or a different microphone, because fine-tuning on one dataset taught them its quirks. Each deployment needed its own fine-tune and an external LM.

The paper's argument: SSL pretrains a great encoder, but the mapping to text still comes from narrow labelled data, and that makes the model brittle. The internet has huge amounts of audio paired with transcripts of mixed quality (subtitles, captions, transcribed podcasts). Training on it is weak supervision: labels that are plentiful but imperfect.

The core idea

Scale and diversity of (noisy) labelled data buy robustness. Collect 680k hours of audio-transcript pairs from the web, filter them heavily, and train a standard sequence-to-sequence Transformer (the attention encoder-decoder, or AED, family from LAS, Listen, Attend and Spell) to output the raw transcript text, with punctuation and casing, with no text normalisation or special pipeline. Make it multitask and multilingual by putting control tokens at the start of the decoder's output, so one model does transcription in ~100 languages, translation to English, language identification, voice activity detection and timestamps.

Then evaluate zero-shot, without fine-tuning on any test dataset, to measure robustness rather than fit to one benchmark.

How it works

Architecture

audio 16 kHz, padded or cut to 30 s
          |
log-mel: 80 channels x 3000 frames
(25 ms window, 10 ms hop)
          |
2 x conv1d + GELU (2nd has stride 2)
          |   1500 positions (20 ms each)
+ sinusoidal position embedding
          |
Transformer encoder (N blocks)
          |------------------+
                             | cross-attention
special tokens + text --> Transformer decoder
(learned positions)          | (N blocks)
                             |
             next-token softmax over
             ~51k BPE + special tokens
  • Input is always exactly 30 s. Shorter audio is zero-padded. The model has never seen anything else, so longer audio needs a windowing scheme (below).
  • Encoder and decoder have the same width and depth. Sizes: tiny ~39M, base ~74M, small ~244M, medium ~769M, large ~1.55B parameters. English-only .en variants exist for the smaller sizes.
  • Text uses a byte-level BPE tokenizer (GPT-2's, refit for the multilingual models).
  • Training loss: ordinary next-token cross-entropy on the transcript tokens.

The special-token sequence

The decoder's input starts with a prompt of control tokens, and the model continues from it. A full training target looks like this:

<|startofprev|> ...text of the previous
                   30 s window...
<|startoftranscript|>
<|hi|>                 language
<|transcribe|>         or <|translate|>
<|0.00|> नमस्ते सब लोग <|1.84|>
<|1.84|> आज हम बात करेंगे <|4.10|>
<|endoftext|>
  • <|startofprev|> + text (optional): the transcript of the previous window, given as context so names and style stay consistent. No loss is computed on it.
  • <|startoftranscript|> (SOT): the real start.
  • Language token (<|en|>, <|hi|>, <|ta|>, ... ~99 of them): if you leave it out, the model predicts it, which is how Whisper does language identification.
  • <|nospeech|> (called <|nocaptions|> in older tokenizers) can be predicted in place of the language token when the window has no speech. That is its built-in VAD (voice activity detection), and its probability is used as the no-speech score.
  • Task token: <|transcribe|> outputs the source language; <|translate|> outputs English text whatever the input language.
  • Timestamps: either <|notimestamps|> (text only), or text segments wrapped in timestamp tokens. There are 1,501 timestamp tokens, <|0.00|> to <|30.00|> in 20 ms steps, relative to the window start. Each segment is start, text, end, so a boundary appears as an end timestamp followed by the next start.
  • <|endoftext|>: done.

At inference you write the prompt yourself (for example SOT <|en|> <|transcribe|> <|notimestamps|>) and the model generates the rest. Forcing the language token is the most useful setting in practice: auto-detection on short or code-mixed Hinglish audio often picks the wrong language.

Data

  • 680k hours in total: ~438k hours of English speech with English text, ~126k hours of non-English speech with English text (the translation data), ~117k hours of non-English transcription data in 96 languages.
  • Filtering matters as much as size. They removed transcripts that look machine-generated (for example all uppercase or all lowercase, or no punctuation), because training on another ASR system's output teaches its mistakes. They dropped pairs whose spoken language (detected from the audio) did not match the language of the transcript, deduplicated, and removed overlap with evaluation sets.

Long-form transcription (the heuristics)

Real audio is longer than 30 s. Whisper transcribes it sequentially: decode a 30 s window with timestamps, move the window to the end of the last complete segment, and repeat. Text after the last complete timestamp pair is discarded and re-decoded in the next window. Greedy decoding alone tends to loop or drift, so the paper adds heuristics (ablated in its long-form table):

  1. Beam search with 5 beams, using log-probability as the score.
  2. Temperature fallback: decode at temperature 0. If the result looks bad, re-decode at 0.2, 0.4, ... up to 1.0. "Bad" means either:
    • compression ratio > 2.4: the gzip compression ratio of the text. Repetitive loops compress very well, so a high ratio flags "the the the..." looping.
    • average log-probability < -1: the model was unsure of its own tokens.
  3. Previous-text conditioning: pass the last window's transcript after <|startofprev|>. The paper only does this when the temperature is below 0.5, because conditioning on garbage spreads garbage.
  4. Voice activity detection: if P(<|nospeech|>) > 0.6 and the average log-probability is < -1, treat the window as silence and output nothing.
  5. Initial timestamp constraint: force the first timestamp of a window to be between 0.0 and 1.0 s, so the model cannot skip the start of the window.

Build the intuition

Analogy: a stenographer who has transcribed every podcast, lecture and YouTube video on the internet, in many languages. They handle any accent or microphone with ease. But they also learned the habits of the web, and when the tape goes quiet, out of habit they write down what the end of a YouTube video usually says.

Why it hallucinates

Hallucination means fluent text that is not in the audio. Three things combine:

  • The decoder is a language model. It produces the next likely token given the text so far and the audio. When the audio carries no speech (silence, music, noise), the audio gives no evidence, so the text prior takes over and it writes something plausible.
  • The training data taught it what to say over non-speech. Internet subtitles put "Thank you for watching", "Subscribe to my channel" or translator credits over intros, outros and music. To the model, silence or music followed by those phrases is a learned pattern.
  • Long-form decoding feeds errors back in. One hallucinated window becomes the <|startofprev|> context for the next, and the model can lock into repetition loops. The compression-ratio check catches some loops, but not short, fluent made-up sentences. A 2024 study ("Careless Whisper", Koenecke et al.) found hallucinated content in roughly 1% of real-world transcriptions, some of it harmful.

A CTC model (CTC) classifies each frame from the audio and emits blank on silence; no text prior pushes it to keep writing. Its errors are substitutions, not inventions, which matters for voice agents.

Mitigations used in practice: run a VAD (for example Silero) and only send speech to Whisper (WhisperX (WhisperX) and faster-whisper's vad_filter do this). Disable condition_on_previous_text. Tighten the no-speech and log-prob thresholds. Drop segments with very low word confidence.

Numbers worth remembering

  • 680k hours of weakly supervised data; ~117k hours across 96 non-English languages, plus ~126k hours of X→English translation.
  • Input: 30 s, 80 log-mel bins, 3000 frames, which become 1500 encoder positions (20 ms each); large-v3 switched to 128 mel bins.
  • Sizes: ~39M (tiny) to ~1.55B (large). large-v3-turbo cuts the decoder from 32 layers to 4 (~809M total).
  • Robustness headline: large and wav2vec 2.0 LARGE both get ~2.7% WER on LibriSpeech test-clean, but Whisper makes ~55% fewer errors on average across other English datasets zero-shot.
  • Long-form heuristics: compression ratio 2.4, log-prob -1, no-speech 0.6, temperature steps of 0.2.

Where it shows up today

  • The default open ASR model, via faster-whisper, whisper.cpp (CPU) and WhisperX (WhisperX). On a 4 GB RTX 3050, small fits easily in fp16; for large, use int8 in faster-whisper.
  • Distil-Whisper (Distil-Whisper) keeps the encoder and shrinks the decoder for speed; OWSM (OWSM) reproduces the recipe with public data; Canary (Canary) shows a curated, much smaller dataset can match it.
  • The audio encoder of speech LLMs: SALMONN and Qwen-Audio / Qwen2-Audio (SALMONN / Qwen-Audio / Qwen2-Audio) start from Whisper's encoder.
  • Indian languages: IndicWhisper in Vistaar (Vistaar) fine-tunes Whisper on Indian data with large WER gains, because off-the-shelf Whisper is weak on most Indian languages; Svarah (Svarah) measures it on Indian-accented English.

What it got wrong, and what came next

  • Hallucination on silence and noise. VAD wrappers (WhisperX, WhisperX); distillation with WER-filtered pseudo-labels (Distil-Whisper, Distil-Whisper) reduced it; CTC/TDT models such as Parakeet (TDT, Fast Conformer) avoid it by design.
  • Not streaming. The 30 s window and full attention mean it waits for the audio. Re-decoding growing buffers is a costly workaround. Native streaming designs are cache-aware Conformers (Cache-aware streaming Conformer) and Kyutai's delayed streams (Kyutai DSM).
  • Slow decoding. One decoder pass per token. Distil-Whisper and large-v3-turbo cut decoder depth; non-autoregressive heads (CTC, TDT) avoid the loop.
  • Loose word timestamps. WhisperX re-aligns with a phoneme CTC model.
  • Uneven languages, since hours per language follow the web. MMS (MMS) and USM (Google USM) use SSL to cover more; Canary (Canary) shows curated, balanced data can match it.

Check yourself

Q1What does "weak supervision" mean here, and why does it give robustness?

The labels are real transcripts scraped from the internet, plentiful but noisy and inconsistent, rather than carefully curated ones. The audio covers huge variety in speakers, accents, microphones and noise, so the model never learns one dataset's quirks. Robustness comes from diversity of training conditions, and evaluating zero-shot shows it.

Q2Write the decoder prompt to translate Tamil speech into English text without timestamps.

<|startoftranscript|> <|ta|> <|translate|> <|notimestamps|>. The language token names the source language, <|translate|> makes the output English (with <|transcribe|> you would get Tamil text), and <|notimestamps|> turns off the timestamp tokens.

Q3How does Whisper do language identification with no separate classifier?

Feed only <|startoftranscript|> and run one decoder step. The distribution over the language tokens at that position is the language ID prediction. It was trained to predict that token, so it is a classifier built into the vocabulary.

Q4What is the compression-ratio check, and what failure does it catch?

It gzip-compresses the decoded text and divides the original length by the compressed length. Repetitive output ("thank you thank you thank you") compresses very well, so a ratio above 2.4 indicates a repetition loop, and the window is re-decoded at a higher temperature. It does not catch short, fluent hallucinations.

Q5Why does Whisper hallucinate on silence when a CTC model does not?

Whisper's decoder is autoregressive and acts as a language model trained on internet subtitles, which often put phrases like "Thanks for watching" over non-speech. With no speech evidence the text prior wins. CTC classifies each frame from the audio independently and outputs blank on silence, so nothing pushes it to produce text.

Q6What would happen if you fed Whisper 45 s of audio directly, with no windowing?

The input pipeline pads or trims to exactly 30 s, so the last 15 s would simply be cut off. The encoder has positional embeddings for only 1500 positions. Long audio has to go through the sequential window loop or VAD-based chunking.

Q7Why is previous-text conditioning disabled when decoding at high temperature?

High-temperature output is the fallback for windows that already looked bad, so it is more likely to be wrong. Feeding it as <|startofprev|> context to the next window would spread the error and can start loops. Conditioning helps consistency only when the previous text is trustworthy.

When you're back at a laptop

  • pip install openai-whisper, load tiny on CPU, and transcribe a recording of yourself. Use whisper.decoding with without_timestamps=False and print the token IDs; decode each with tokenizer.decode([id]) to see the special tokens and timestamps.
  • Force language="hi" versus "en" on Hinglish speech, and set task="translate". Compare the outputs.
  • Make 30 s of silence and 30 s of white noise with numpy, transcribe each with condition_on_previous_text=True, and record what it invents. Then pass no_speech_threshold and a VAD, and see the hallucination disappear.
  • Print result["segments"] and look at compression_ratio, avg_logprob and no_speech_prob for each segment.
arXiv ↗PDF ↗