ASR · Phase 5: Scale and multilinguality
WhisperX
WhisperX: Time-Accurate Speech Transcription of Long-Form Audio
In one sentence
WhisperX wraps Whisper in a pipeline (voice activity detection to cut audio, batched transcription, then forced alignment with a phoneme CTC model) to make long-form transcription faster, less hallucination-prone, and accurate to the word timestamp.
Why this paper exists
Whisper Whisper sees 30 s at a time. For long audio, the reference code slides a window forward, conditioning each window on the previous text. That is:
- Sequential, so it cannot batch on a GPU.
- Drift-prone: errors carry forward, and silent windows invite hallucinated or looping text.
- Coarse in time: timestamps are often off by a second or more, too loose for subtitles, diarization or barge-in logic in a voice agent.
The core idea
Stop asking Whisper to segment and align. A cheap VAD (voice activity detector, which marks where speech is) picks the chunks, Whisper only transcribes, and a small phoneme recogniser pins words to the audio.
How it works
- VAD over the whole file finds speech regions.
- Cut and merge: split long regions at quiet points and merge short ones, so chunks are close to but under 30 s and contain mostly speech.
- Batched transcription: chunks are independent (no previous-text conditioning), so many run in one batch.
- Forced alignment: a wav2vec 2.0 wav2vec 2.0 CTC CTC phoneme model gives per-frame posteriors; Viterbi over the known transcript's token path gives each word's start and end.
- (Repo extra) diarization assigns speakers using the word timestamps.
Build the intuition
Analogy: a clerk marks paragraph boundaries, several translators work in parallel, and a proofreader pins every word to its exact line.
Alignment is easier than recognition: the text is known, so you solve only "where", not "what". Without VAD, fixed 30 s cuts slice words in half and send pure silence to Whisper, exactly where it hallucinates.
Numbers worth remembering
- About 12x speed-up from batched inference over sequential long-form Whisper (check the paper's table for the setup).
- Chunks capped at 30 s to match Whisper's input window.
- Timestamp precision bounded by wav2vec 2.0's 20 ms frames (50 frames/s).
Where it shows up today
- The WhisperX repo is a widely used open long-form tool, usually on faster-whisper, with Distil-Whisper Distil-Whisper as an option.
- "VAD, then ASR, then aligner" is the standard production and data-pipeline pattern (Granary Granary, MMS alignment MMS).
- VAD gating before ASR is a voice agent's first defence against hallucination on silence.
What it got wrong, and what came next
- Offline only: no partial results while someone speaks; see Cache-aware streaming Conformer, Kyutai DSM.
- Needs a per-language aligner: many Indian languages lack a good one; MMS CTC models MMS help.
- Loses cross-chunk context such as consistent name spellings.
Check yourself
Q1Why does dropping text conditioning allow batching?
In vanilla Whisper, window n waits for window n-1's output as its prompt. Independent chunks can all go in one GPU batch, and one chunk's error no longer leaks into the next.
Q2Why use a CTC phoneme model for timestamps?
CTC gives per-frame token probabilities at a fixed frame rate, so Viterbi over the known transcript yields frame-level word boundaries. Whisper's attention decoder has no frame-level output, so its timestamps are coarse.
Q3How does VAD reduce hallucinations?
Whisper tends to invent text on silence or noise. Removing non-speech first means most chunks it sees actually contain speech.
When you're back at a laptop
- Run
whisperxwith--model smallon a 10-minute podcast; compare its word timestamps with vanilla Whisper's in an audio editor. - Try a Hindi clip and check whether an alignment model exists.
- Time batch sizes 1, 4 and 8 on your 4 GB GPU (use int8 if memory runs out).