Fast Conformer
Fast Conformer with Linearly Scalable Attention for Efficient Speech Recognition
In one sentence
Fast Conformer is a Conformer redesigned for speed: 8x convolutional subsampling with depthwise separable layers (instead of 4x), a smaller convolution kernel, and an optional limited-context attention with a global token, making it roughly 2.8x faster with similar accuracy and able to transcribe hours of audio in one pass; it is the encoder of NVIDIA's Parakeet and Canary models.
Why this paper exists
The Conformer (Conformer) runs its blocks at 40 ms per frame (4x subsampling of 10 ms features). That causes two problems:
- Compute and memory scale with frame count, and self-attention scales with its square. A 1.1B-parameter encoder at 25 frames per second is expensive to train and serve.
- Long audio does not fit. Full attention over an hour of audio needs a T×T score matrix with T in the tens of thousands, which no GPU holds. In practice you chop audio into chunks and stitch the results, which causes errors at chunk boundaries.
There was also a hidden cost: the Conformer's subsampling block (two full 2D convolutions with many channels at high time and frequency resolution) was a noticeable share of total compute on its own.
The core idea
Most of the information a subword ASR model needs survives at a coarser time resolution. Downsample harder and cheaper at the input, so every layer after it does half the work and attention does a quarter. Then, for very long audio, replace full attention with local windowed attention plus one global token, which makes attention cost linear in length.
How it works
The changes relative to Conformer
Conformer Fast Conformer
--------- --------------
2 x Conv2d, stride 2 3 x depthwise-separable
(4x, 40 ms frames) Conv2d, stride 2
(8x, 80 ms frames)
fewer channels (256)
N x Conformer block N x Conformer block
conv kernel 31/32 conv kernel 9
full attention full attention, or
local window + global
token (linear cost)- 8x subsampling. Three stride-2 stages instead of two. Each stage uses a depthwise separable convolution (per-channel spatial filter followed by a 1x1 channel mix, the trick from QuartzNet Jasper → QuartzNet → Citrinet) and the subsampling uses 256 channels, which makes this front end much cheaper than the original.
- Smaller depthwise kernel in the conv module: 9 instead of 31. At 80 ms frames a kernel of 9 still spans 720 ms, comparable in audio time to what the original kernel covered at 40 ms, and it is cheaper.
- Limited-context attention with a global token (Longformer-style). Each frame attends only to a window of nearby frames, and a single global token attends to and is attended by every frame, carrying utterance-level context. Attention cost goes from
O(T^2)toO(T * W)for window size W.
The blocks themselves (macaron FFNs, MHSA, conv module) are otherwise Conformer blocks, and the encoder works with CTC, RNN-T, TDT (TDT) or attention decoders.
Frame rates: 4x vs 8x
Input features are 10 ms per frame, so 100 frames per second.
| 4x (Conformer) | 8x (Fast Conformer) | |
|---|---|---|
| frame duration | 40 ms | 80 ms |
| frames per second | 25 | 12.5 |
| 10 s clip | 250 frames | 125 frames |
| 1 minute | 1,500 | 750 |
| 1 hour | 90,000 | 45,000 |
What it does to attention cost
Full self-attention builds a T x T score matrix per head per layer. Compute is O(T^2 * d); memory for scores is O(T^2).
- Halving T cuts attention compute and score memory by 4x.
- FFN and conv modules are linear in T, so they get 2x cheaper.
- For a 1-minute clip: 1,500² = 2.25M scores versus 750² ≈ 0.56M per head per layer.
- For 1 hour: 90,000² = 8.1 billion scores versus 45,000² ≈ 2.0 billion per head per layer. In fp16 that is ~16 GB versus ~4 GB, per head, per layer. Even at 8x, full attention over an hour is out of reach.
- With local attention of, say, 128 frames each side plus one global token: 45,000 × ~258 ≈ 11.6M scores. Linear in length, about 170x fewer than full attention at the same frame rate.
Build the intuition
Analogy: reading a transcript at 25 lines per second versus 12.5. Each line now carries twice as much speech, but a subword is still several lines long, so nothing is lost, and anyone comparing every line to every other line (attention) has a quarter of the comparisons to make.
Why 8x is safe for subwords, and where it stops. CTC needs at least one frame per output token, plus blanks between repeats. At 12.5 frames per second that allows up to ~12 tokens per second. Fast English speech is roughly 3-5 words per second, and a SentencePiece vocabulary of ~1,000 pieces gives a bit more than one token per word on average, so there is headroom. Characters (~15 per second in fast speech) would not fit, which is why the 8x models use subwords. A language with long words split into many pieces, or a very small vocabulary, eats into this headroom; a transducer with duration-0 emissions (TDT, TDT) can put several tokens on one frame and relaxes the limit.
Why a global token helps. With only local windows, a frame at minute 40 cannot know the recording conditions or speaker from minute 1. One extra token that sees everything acts like a shared notice board, which is the attention-based version of ContextNet's squeeze-and-excitation (ContextNet).
For your laptop: halving the frames is the difference between a 0.6B encoder fitting in 4 GB for a long clip or not. On CPU, the 2x reduction in linear-cost modules matters most, because attention is a smaller share at short lengths.
Numbers worth remembering
- 8x subsampling: 80 ms per frame, 12.5 frames per second (vs 40 ms, 25 per second for Conformer).
- Roughly 2.8x faster than Conformer at similar WER (check the paper's tables for per-dataset WERs).
- Conv module depthwise kernel: 9 (vs 31-32).
- With limited-context attention plus a global token, the paper reports transcribing up to ~11 hours of audio in a single pass on one GPU.
- Production sizes built on it: Parakeet ~0.6B and ~1.1B, Canary ~1B (check model cards for exact layer counts).
Where it shows up today
- NVIDIA Parakeet (CTC, RNN-T and TDT TDT heads) and Canary (Canary) use Fast Conformer encoders.
- The cache-aware streaming models (Cache-aware streaming Conformer) are Fast Conformers trained with limited left and right context, which is what NVIDIA's streaming Nemotron ASR models build on (check the model card).
- AI4Bharat's IndicConformer models use NeMo Conformer-family encoders, a practical route for Indian-language ASR.
- 80 ms frames also match the frame rates of modern codecs: Mimi in Moshi (Moshi) runs at 12.5 Hz, the same rate.
What it got wrong, and what came next
- One fixed resolution after subsampling. Zipformer (Squeezeformer / E-Branchformer / Zipformer) varies frame rate across the stack and goes even lower in the middle.
- Limited context trades some accuracy on long-range dependencies for linear cost; for normal-length utterances full attention is still used.
- Offline by default. Streaming needs causal or chunked attention and caches, added by the cache-aware streaming work (Cache-aware streaming Conformer).
- 8x is near the limit for CTC with small vocabularies; transducer heads (TDT) handle dense token rates more gracefully.
Check yourself
Q1What are the three main changes Fast Conformer makes to the Conformer?
8x subsampling with three stride-2 depthwise separable convolutions and fewer channels, instead of 4x with two full convolutions. A smaller depthwise kernel (9 instead of 31) in the conv module. And optional limited-context (windowed) attention plus a global token for long audio.
Q2Compute the frame rate after 4x and 8x subsampling of 10 ms features.
10 ms × 4 = 40 ms per frame, which is 25 frames per second. 10 ms × 8 = 80 ms per frame, which is 12.5 frames per second. A 10 s clip goes from 1,000 input frames to 250 or 125.
Q3Halving the number of frames makes attention how much cheaper, and why?
About 4x. Self-attention computes a score for every pair of frames, so cost grows with T². Halving T quarters the pairs. Linear-cost parts like the FFNs and conv modules become 2x cheaper.
Q4Why is 8x downsampling acceptable for a subword model but risky for a character model?
CTC needs at least one frame per output token. At 12.5 frames per second a model can emit roughly 12 tokens per second, enough for subwords at normal speech rates but not for characters, which arrive faster in quick speech. Running out of frames forces deletions.
Q5How does limited-context attention with a global token make long-form transcription possible?
Each frame attends only to a fixed window, so cost grows linearly with length rather than quadratically. The global token attends to all frames and all frames attend to it, so utterance-level information still flows. Memory then fits hours of audio on one GPU instead of minutes.
Q6Why reduce the conv module's kernel from 31 to 9?
After 8x subsampling each frame covers twice as much time, so a smaller kernel still spans a similar stretch of audio (9 × 80 ms = 720 ms). The smaller kernel is cheaper and, per the paper, did not hurt accuracy.
Q7What would happen if you pushed to 16x subsampling?
Frames would be 160 ms, about 6 per second. Attention would be another 4x cheaper, but CTC could not emit more than ~6 tokens per second, so fast speech would cause deletions, and fine timing for word timestamps would get coarser. You would likely need a transducer with multi-token emission or a larger vocabulary.
When you're back at a laptop
- Write a function
frames(seconds, stride)and tabulate encoder frames and attention score counts for 10 s, 1 min and 1 h at 4x and 8x. - Load a Parakeet or Fast Conformer model in NeMo, run a 1-minute clip, and print the encoder output shape to confirm 12.5 frames per second.
- In NeMo, try switching the encoder to limited-context attention (
change_attention_modelwith a local attention mode) and transcribe a long recording; compare memory use with full attention. Check the NeMo docs for the exact argument names.