ASR track
Speech recognition
From CTC and RNN-T to Conformer, wav2vec 2.0, Whisper, Parakeet, Canary, streaming models, Moshi and Indian-language ASR.
0 of 35 read · about 7.9 hours of reading
The map: how every model fits together
Almost every modern ASR system is:
waveform → log-mel → subsampling → ENCODER
→ DECODER / LOSS → text| Choice | Options |
|---|---|
| Encoder | RNN → CNN (Jasper, QuartzNet) → Transformer → Conformer → Fast Conformer / Zipformer |
| Decoder / loss | CTC, Transducer (RNN-T, TDT), attention encoder-decoder (AED), hybrid CTC + attention |
| Training | supervised, self-supervised pretraining + fine-tuning, or weakly supervised at huge scale |
| Streaming | offline (sees the whole utterance) vs streaming (limited look-ahead, cache) |
| Newest | speech + LLM, full-duplex models, delayed streams |
If you understand the three decoder families and the Conformer, you can read any new ASR paper.
Phase 1
Foundations
Audio has 50-100 frames per second, text has a few tokens per second, and nobody tells you which frames belong to which token. CTC, RNN-T and attention are the three answers to that alignment problem.
Phase 2
Efficient transducers
NVIDIA's line of convolutional models and the Token-and-Duration Transducer: the path that leads to Parakeet.
Phase 3
The encoder
The Conformer and its descendants. Attention for global context, convolution for local patterns, and ever more aggressive downsampling for speed.
Phase 4
Self-supervised learning
Labelled speech is expensive, especially for Indian languages, and unlabelled audio is nearly free. This is how low-resource ASR works.
Phase 5
Scale and multilinguality
Whisper's weak supervision at 680k hours, the tools built around it, and the models that answered it with better data (Canary) or self-supervision at scale (USM, MMS).
Phase 6
Streaming ASR
Live calls need text while the person is still speaking, with a few hundred milliseconds of delay at most.
Phase 7
Speech meets LLMs
Full-duplex speech models and the audio-encoder + adapter + LLM pattern.
Phase 8
Indian languages and accents
AI4Bharat's models, datasets and benchmarks, and why off-the-shelf ASR struggles on Indian speech.