ASR self-check

Audio arrives at 50-100 frames per second while text has a few tokens per second, and training data does not say which frames belong to which token. CTC CTC adds a blank symbol and sums the probability of every frame-level path that collapses to the transcript, using a forward-backward dynamic program. RNN-T RNN-T sums over all monotonic paths through a T×U lattice of (frame, tokens emitted so far), where each step either emits a token or a blank that advances to the next frame. Attention encoder-decoders (LAS Listen, Attend and Spell, Whisper Whisper) never model alignment explicitly: the decoder generates one token at a time and soft-attends over all encoder frames. CTC assumes the output at each frame is independent of the outputs at other frames given the audio, so it cannot learn spelling or word dependencies itself and benefits from an external LM. RNN-T removes this with a prediction network that conditions each output on previously emitted tokens. AED removes it because each token is conditioned on the full previous token sequence through the autoregressive decoder.

The collapse function B first merges consecutive repeated symbols, then removes all blanks: B(hh_e_ll_lo) = hello. The order matters: merging happens before blank removal. Without a blank, any run of identical symbols merges into one, so "ll" in "hello" could never be produced, since hhellllo collapses to helo. A blank between the two l's (l_l) keeps them as separate tokens after merging. The blank also gives the model a "nothing new here" output for frames between tokens, which is most frames, since tokens are far sparser than frames. See CTC.

A transducer RNN-T is frame-synchronous: at each new encoder frame it decides whether to emit tokens or a blank, so text appears as audio arrives, with latency set only by the encoder's look-ahead. Its alignment is monotonic by construction, so it cannot jump back or loop. An AED decoder attends over the whole encoder output, so it needs the full utterance (or a window, like Whisper's 30 s Whisper) before generating, and making attention streamable requires extra machinery such as monotonic or chunked attention. AEDs can also hallucinate or repeat when attention misfires, which is dangerous in a live product. Transducers also run well with cache-aware streaming encoders Cache-aware streaming Conformer and were proven on-device by Google Streaming E2E ASR for mobile. That is why Parakeet-RNNT/TDT and Zipformer transducers Squeezeformer / E-Branchformer / Zipformer dominate streaming ASR.

The Token-and-Duration Transducer TDT makes the joint network predict two things: the token (or blank) and a duration, meaning how many encoder frames to advance after this output. Standard RNN-T advances exactly one frame per blank, so greedy decoding visits every frame and spends most steps emitting blanks. TDT can predict "skip 4 frames", jumping over stretches with nothing new, so it makes far fewer decoder and joint calls per utterance. The loss sums over all (token, duration) paths in the lattice, extending the RNN-T forward algorithm. Accuracy is equal or better because the model learns where tokens really are rather than being forced to step blindly. This is the TDT in Parakeet-TDT, and with Fast Conformer's 8× subsampling Fast Conformer it gives very fast batch transcription.

The block is x → x + ½·FFN(x) → + MHSA(x) → + Conv(x) → + ½·FFN(x) → LayerNorm, where MHSA uses relative positional encoding and Conv is pointwise conv, GLU, depthwise conv, BatchNorm, Swish, pointwise conv Conformer. Self-attention connects any two frames, so it captures global context such as speaker characteristics and long-range word cues. The depthwise convolution captures local patterns over a few neighbouring frames, such as formant transitions within a phone, which attention learns less efficiently. Combining them beat both pure Transformers and pure CNNs like ContextNet ContextNet on LibriSpeech. The macaron structure, borrowed from Macaron-Net, splits one feed-forward layer into two half-weighted FFNs sandwiching attention and convolution, like the two biscuits of a macaron. The ablations showed this beats a single full FFN. Fast Conformer Fast Conformer and Zipformer Squeezeformer / E-Branchformer / Zipformer keep this basic recipe.

Fast Conformer Fast Conformer replaces Conformer's 4× convolutional subsampling with 8× subsampling using cheaper depthwise-separable convolutions, so 10 ms mel frames become 80 ms encoder frames (12.5 per second instead of 25). It also slims down the convolution kernel and supports limited-context (local) attention with a global token for very long audio. Self-attention cost grows with the square of the sequence length, so halving the number of frames cuts attention compute and memory by about 4×, while the convolution and FFN layers get about 2× cheaper. The result is a much faster encoder with little or no accuracy loss, because 80 ms is still shorter than most subword tokens. A side benefit is that transducer and CTC decoding have fewer frames to step through. It is the encoder of Parakeet, Canary Canary and Nemotron ASR.

wav2vec 2.0 wav2vec 2.0 encodes raw audio with a CNN into 20 ms latents, masks spans of them, and runs a Transformer over the masked sequence. The unmasked latents are also quantized with Gumbel-softmax codebooks. For each masked position the model must pick the true quantized latent among distractors sampled from other positions (a contrastive InfoNCE loss), plus a diversity loss that keeps codebook entries in use. HuBERT HuBERT instead creates fixed targets offline: k-means cluster IDs, first on MFCCs and later on its own hidden features, and trains with a BERT-style cross-entropy to predict the cluster ID of masked frames, iterating the process. This separates target creation from learning and is more stable than learning targets and representations together. BEST-RQ data2vec / w2v-BERT / BEST-RQ simplifies further: targets come from a frozen random projection plus a random codebook applied to the input features, with no learning and no clustering, and it still works at scale in Google USM Google USM.

Weak supervision Whisper means training on labels that are noisy and not made for the task: about 680k hours of internet audio paired with whatever transcripts existed, filtered with heuristics (for example removing machine-generated transcripts and language mismatches). It is supervised learning, but at a scale and messiness no curated dataset matches. The diversity of speakers, accents, microphones and noise makes Whisper robust zero-shot across many test sets, rather than overfitting to one clean benchmark like LibriSpeech. It hallucinates because its decoder is effectively a conditional language model trained to always output fluent text for 30 s windows. When the audio contains silence, music or noise, the LM prior takes over and produces plausible text, often phrases common in its training data such as "Thanks for watching". Long-form heuristics, VAD pre-segmentation (WhisperX WhisperX) and distillation with filtered pseudo-labels (Distil-Whisper Distil-Whisper) reduce it.

The decoder prompt is a sequence of special tokens before any text Whisper. It starts with <|startoftranscript|>, then a language token such as <|hi|> or <|en|>, which the model can predict itself (that is how language identification works) or you can force it. Next a task token: <|transcribe|> outputs text in the spoken language, <|translate|> outputs English. Then either <|notimestamps|> for plain text, or the model interleaves timestamp tokens like <|0.00|> ... <|2.40|> around segments, quantized to 20 ms. Previous-window text can be passed as context with <|startofprev|> for long-form transcription. Because one model handles all combinations, changing a few prompt tokens switches it between tasks with no extra heads.

Whisper Whisper bets on volume: 680k hours of weakly labelled web audio, a plain Transformer, and minimal curation beyond filtering. Canary Canary uses an order of magnitude less data but curated and balanced across languages and tasks, with pseudo-labelled and synthetic data, a stronger Fast Conformer encoder Fast Conformer, and careful training techniques. It matched or beat Whisper on ASR and translation in its supported languages. The lesson is that label quality, balance and recipe can substitute for raw scale, which matters when you cannot collect hundreds of thousands of hours. The trade-off is coverage: Canary supports far fewer languages than Whisper. NVIDIA's data work continues in Granary Granary, and the same idea drives Indian ASR, where curated benchmarks and fine-tuning beat larger generic models Vistaar.

In a naive approach you train a Conformer with full context and then run it on chunks at inference, so the model sees much less context than it was trained with and accuracy drops. Cache-aware streaming Cache-aware streaming Conformer trains with the same limited left and right context it will have at inference, using attention masks and causal (or limited look-ahead) convolutions. At inference it caches the keys and values for attention and the convolution states from previous chunks, so each new chunk is processed once with no recomputation and the outputs match training exactly. Chunk size is how much new audio is processed per step; look-ahead is how many future frames each frame may see. Larger chunks and more look-ahead improve WER because the model sees more future context, but add latency because you must wait for that audio. Smaller chunks reduce latency but raise compute overhead per second and WER. Products typically pick a point like 80-500 ms total latency on this curve.

Delayed streams modelling Kyutai DSM represents audio (as Mimi codec tokens Moshi) and text as parallel streams on the same time axis, both at the codec frame rate, with text padded so each word sits near when it is spoken. One stream is shifted by a fixed delay relative to the other, and a decoder-only Transformer predicts the next step of all streams. For ASR, text is delayed behind audio, so when the model emits a word it has already heard a little audio past it, which acts as look-ahead. For TTS, audio is delayed behind text, so the model reads a few words ahead before speaking them Delayed Streams Modeling (TTS). The same architecture and training code do both; only which stream is the input and the direction of the delay change. Because generation is step-by-step and the delay is fixed, latency is constant and arbitrarily long inputs stream naturally. This powers Kyutai STT and Kyutai TTS.

Mimi is a neural audio codec at 12.5 Hz with residual vector quantization (8 codebooks in Moshi), where the first codebook is distilled from WavLM features so it carries semantic content and later codebooks carry acoustic detail Moshi. The RQ-Transformer splits generation in two: a large temporal Transformer (initialized from the Helium text LLM) runs once per 80 ms frame, and a small depth Transformer predicts the codebooks within that frame one after another. Inner monologue means the model also predicts time-aligned text tokens of its own speech before the audio tokens, which improves linguistic quality and gives free streaming ASR/TTS-like behaviour. Full duplex means the user's audio and Moshi's audio are two parallel streams modelled at every step, so Moshi can listen while speaking, handle interruptions and backchannels, with no explicit turn-taking. Its theoretical latency is about 160 ms (80 ms frame plus 80 ms acoustic delay), around 200 ms in practice. The same codec and multi-stream design underlie delayed streams modelling Kyutai DSM and Sesame CSM Sesame CSM.

Training data for big models is dominated by English and a few high-resource languages; Indian languages get a tiny share, and much of it is read or broadcast speech. Real Indian speech adds code-mixing (Hindi words and English words in one sentence, in either script), regional accents of English, telephone-band audio, background noise, and agglutinative morphology in Dravidian languages that inflates word error rates. Tokenizers and text normalization are also tuned for English, so numbers and scripts get scored inconsistently. To measure it, use Vistaar Vistaar, which aggregates Indian-language benchmarks and shows the gains from IndicWhisper; Kathbath from IndicSUPERB IndicSUPERB for read speech in 12 languages; Svarah Svarah for Indian-accented English; and IndicVoices for conversational speech. Report both WER and CER, since CER is fairer for agglutinative languages, and add your own in-domain test set of real call audio. Baselines should include IndicWav2Vec IndicWav2Vec, IndicConformer, MMS MMS and Whisper.

Choose CTC CTC when you need the simplest, fastest decoder: frame-parallel greedy decoding, easy batching, natural fit for streaming and for adding an n-gram LM or hotword biasing, as in Parakeet-CTC or wav2vec 2.0 fine-tunes wav2vec 2.0. Its weakness is the conditional-independence assumption, which costs some accuracy without an LM. Choose TDT TDT (or RNN-T RNN-T) when you want transducer accuracy, with an internal LM, monotonic alignment and true streaming, at speed close to CTC; with a cache-aware Fast Conformer Cache-aware streaming Conformer this is usually the best choice for a live voice agent. Choose an AED like Whisper Whisper or Canary Canary for offline or turn-level transcription where accuracy, punctuation, multilingual handling or translation matter more than latency, such as call analytics after the call. AEDs are slower, harder to stream and can hallucinate on silence, so in a live agent they need VAD gating and segment-level use. In practice many stacks use a streaming transducer for the live turn and an AED for post-call accuracy.