Reading Room

ASR · Phase 5: Scale and multilinguality

Canary

Less is More: Accurate Speech Recognition & Translation without Web-Scale Data

Puvvada, Żelasko, Huang, Hrinchuk, Koluguri, et al., Ginsburg · NVIDIA · 2024

Deep read18 minDay 3 of 10#21 of 86

In one sentence

Canary is a Fast Conformer encoder with a Transformer decoder (an attention encoder-decoder like Whisper) that does ASR and speech translation in English, German, Spanish and French, and matches or beats much larger-data models such as Whisper because its training data is carefully balanced, augmented with synthetic translations, and trained with an efficient recipe.

Why this paper exists

By 2023 the lesson of Whisper Whisper seemed to be "more web audio wins": 680k hours of weakly labelled internet data gave robust zero-shot ASR. That lesson is expensive. Few teams can collect, store and train on that much audio, and web transcripts are noisy, which shows up as hallucination.

Open attempts with public data, such as OWSM OWSM, reproduced Whisper's recipe with less data but got lower accuracy. Was scale the only path, or was the recipe leaving accuracy on the table?

The core idea

Spend effort on data quality and training recipe instead of raw hours. Concretely:

  • Use the faster, stronger Fast Conformer Fast Conformer encoder instead of a plain Transformer.
  • Balance languages and datasets so no single corpus dominates.
  • Create speech-translation training data by machine-translating ASR transcripts, since real speech-translation corpora are small.
  • Teach the model that non-speech should produce empty output, which reduces hallucination.

How it works

Architecture

audio (16 kHz)
   |
log-mel (10 ms hop, 100 frames/s)
   |
Fast Conformer encoder
 (8x conv subsampling -> 80 ms frames, 12.5/s)
   |  encoder states  H: [T/8 x d]
   v
Transformer decoder  <--- prompt tokens:
 (cross-attends H)        <src lang> <task>
   |                      <tgt lang> <pnc>
   v
text tokens, one at a time (autoregressive)
  • Encoder. Fast Conformer Fast Conformer is a Conformer Conformer with 8x downsampling up front (depthwise-separable convolutions) instead of 4x. A 10 s clip becomes ~125 encoder frames instead of ~250, which cuts attention cost by about 4x since attention is quadratic in length.
  • Decoder. A standard Transformer decoder with cross-attention to the encoder output, as in Whisper. This is AED (attention encoder-decoder): no CTC or transducer loss.
  • Prompt (task tokens). The decoder is conditioned on special tokens for source language, task (transcribe or translate), target language, and whether to output punctuation and capitalisation (PnC). So one model does English ASR, German ASR, English-to-German translation, and so on.
  • Tokenizer. A concatenated tokenizer: a separate SentencePiece vocabulary per language joined into one output space, so each language gets enough vocabulary.
  • Size. About 1B parameters (Canary-1B).

Training objective

Plain token-level cross-entropy, as in any AED:

L = - Σ_{u=1..U} log p(y_u | y_<u, prompt, x)

  • x is the audio (through the encoder), prompt the task tokens, y_u the u-th output token, y_<u the tokens before it (teacher forcing during training), and U the output length.

The recipe (where the gains come from)

  1. Data balancing. Sample languages and corpora with weights so a large English corpus does not drown out smaller German or Spanish sets.
  2. Synthetic translation data. Take ASR data (audio + transcript), translate the transcript with a text machine-translation model, and use the result as a speech-translation target. This gives far more speech-translation pairs than human-made corpora.
  3. Noise-robust fine-tuning. Add non-speech audio with an empty target, so the model learns to output nothing on silence or noise, a direct fix for Whisper-style hallucination.
  4. Efficient batching. Dynamic bucketing groups utterances of similar length so batches waste little padding.
  5. Initialisation. The encoder can start from a pretrained ASR checkpoint, which speeds convergence.

Build the intuition

Analogy: Whisper is a student who read every website in the library, typos and all. Canary is a student with a tenth of the reading, but a well-chosen syllabus, practice exams in every subject (balancing), worked translation exercises made by a tutor (synthetic data), and explicit practice on "this page is blank, write nothing".

Worked example of the subsampling cost. A 20 s clip at 100 log-mel frames/s is 2,000 frames.

  • With 4x subsampling (classic Conformer): 500 frames. Self-attention compares every pair: 500 x 500 = 250,000 scores per head per layer.
  • With 8x (Fast Conformer): 250 frames, 62,500 scores. Four times fewer, and the decoder's cross-attention also halves its memory.

Thought experiment: train without balancing on 80% English data, and the model is excellent on English and worse elsewhere.

Thought experiment 2: why does an empty-target example on noise help? An AED decoder is a language model conditioned on audio. With no grounding signal, a language model keeps generating fluent text. Showing it many "noise in, end-of-sequence out" examples gives it a strong prior to stop.

Numbers worth remembering

  • Canary-1B: ~1B parameters, 4 languages (en, de, es, fr), ASR plus translation between English and each of the other three.
  • Trained on about 86k hours of speech in the paper, versus Whisper's 680k: roughly an order of magnitude less (check the data table for the per-language split).
  • Fast Conformer: 8x subsampling, 80 ms per encoder frame.
  • At release Canary-1B ranked at or near the top of the Hugging Face Open ASR Leaderboard for English (check the leaderboard history for the exact average WER).

Where it shows up today

  • The Canary family: later NVIDIA releases (for example canary-1b-flash and canary-1b-v2, check the model cards for languages and data) keep the Fast Conformer + Transformer decoder design; multilingual releases use Granary Granary data.
  • Parakeet shares the encoder but uses CTC CTC, RNN-T RNN-T or TDT TDT decoders instead of AED; streaming versions use cache-aware training Cache-aware streaming Conformer.
  • Speech + LLM: NVIDIA has paired Canary-style encoders with LLM decoders, the same pattern as SALMONN / Qwen-Audio SALMONN / Qwen-Audio / Qwen2-Audio.
  • The "curated, balanced, pseudo-labelled data" lesson is the same one behind Distil-Whisper Distil-Whisper and Granary Granary, and it is how you should approach Indian-language ASR, where web-scale data does not exist (IndicWav2Vec, Vistaar).

What it got wrong, and what came next

  • Few languages. Four languages is a narrow scope compared with Whisper (~100), USM Google USM or MMS MMS. Later Canary releases extended to 25 European languages using Granary Granary; Indian languages are still not its focus.
  • Not streaming. It is offline AED with autoregressive decoding. For live voice agents you want a streaming transducer Cache-aware streaming Conformer or delayed-streams model Kyutai DSM.
  • Synthetic translation targets inherit MT errors. The quality of the text translation model caps the speech-translation quality on that data.

Check yourself

Q1What is the main claim of the "Less is More" paper?

That data curation and training recipe can substitute for raw scale. Canary is trained on roughly a tenth of Whisper's audio yet matches or beats Whisper on ASR and speech translation in its supported languages. The gains come from a better encoder, balanced sampling, synthetic translation data, and noise-robust fine-tuning.

Q2How does one Canary model know whether to transcribe or translate?

The decoder is given prompt tokens before generating: source language, task, target language, and punctuation on or off. These condition every output step, so the same weights produce a German transcript or an English translation depending on the prompt. This is the same idea as Whisper's special tokens.

Q3Why use a Fast Conformer encoder rather than Whisper's plain Transformer?

Fast Conformer adds convolution modules that model local acoustic patterns, and its 8x subsampling halves the sequence length compared with 4x. Since self-attention is quadratic in length, that is about 4x less attention compute, which makes both training and inference cheaper for the same accuracy.

Q4Where does Canary's speech-translation training data come from?

Mostly from ASR data whose transcripts were translated by a text machine-translation model. Real speech-translation corpora are small, so this synthetic pairing multiplies the available data. The trade-off is that MT mistakes become training targets.

Q5What would happen if you removed the noise-robust fine-tuning step?

The decoder, which behaves like a conditional language model, would be more likely to produce fluent but invented text on silence or background noise. The fine-tuning teaches it that non-speech input should produce an empty output.

Q6Canary and Parakeet share an encoder. When would you pick each?

Pick Canary when you need translation, multiple tasks from one model, or the best offline accuracy. Pick Parakeet (CTC or TDT) for fast English ASR, especially at low latency or in streaming variants, since it avoids slow autoregressive decoding.

When you're back at a laptop

  • Load nvidia/canary-1b (or a newer Canary) in NeMo and transcribe an English clip; then switch the prompt to translate it into German.
  • Compare WER and speed with Whisper-small on the same 10 clips; try int8 or fp16 to fit your 4 GB GPU, or run on CPU.
  • Feed 30 s of noise to both and count how often each outputs text.
  • Open the NeMo model config and find the subsampling factor, number of encoder/decoder layers, and the tokenizer type.
arXiv ↗PDF ↗