ASR · Phase 5: Scale and multilinguality
SeamlessM4T
SeamlessM4T: Massively Multilingual & Multimodal Machine Translation
In one sentence
SeamlessM4T is one model family for speech-to-speech, speech-to-text, text-to-speech and text-to-text translation plus ASR across about 100 languages, built from a w2v-BERT 2.0 speech encoder, NLLB-based text components, and a unit-based speech generator.
Why this paper exists
Speech translation used to be a cascade: ASR, then machine translation, then TTS. Each stage adds latency and passes its errors on, and each needs per-language models. End-to-end models existed but covered few languages, because paired speech-translation data is scarce.
The core idea
Share components across tasks: speech and text encoders feed one text decoder; a second stage turns text into discrete speech units, and a vocoder turns units into audio. Training data for rare language pairings is mined automatically.
How it works
- Speech encoder: w2v-BERT 2.0, a Conformer Conformer pretrained with combined contrastive and masked-prediction objectives (w2v-BERT data2vec / w2v-BERT / BEST-RQ) on ~1M hours of unlabelled audio.
- Text encoder and decoder from NLLB (Meta's "No Language Left Behind" translation model). The decoder writes text in the target language; for ASR, target equals source.
- Speech output (UnitY): a text-to-unit model predicts discrete acoustic units (cluster IDs of self-supervised features, like HuBERT HuBERT units), and a unit HiFi-GAN vocoder HiFi-GAN produces the waveform. Text first, then speech.
- Data (SeamlessAlign): speech and text are embedded in a shared multilingual sentence-embedding space (SONAR) and matched across languages to mine aligned pairs.
Build the intuition
Analogy: an interpreter booth with one shared brain for meaning (the text decoder), separate ears (speech or text encoders) and separate mouths (text, or units then voice).
Why units? Generating waveforms straight from decoder states is hard. Units are a compact, mostly linguistic code, roughly clustering-discovered phonemes, so predicting them is just another sequence-to-sequence task, and the vocoder handles acoustics.
As a pure English ASR model it works, but a dedicated model like Canary Canary or Parakeet is usually more accurate and faster.
Numbers worth remembering
- ~100 languages for speech input and text; speech output for ~35 (check the paper per task).
- w2v-BERT 2.0 pretrained on ~1M hours of unlabelled audio.
- SeamlessM4T-Large ~2.3B parameters; a Medium variant also released.
- Reports a sizeable BLEU gain over the previous best direct speech-to-text translation on FLEURS (see the results tables).
Where it shows up today
- SeamlessM4T v2 improved speech generation (UnitY2); the family added SeamlessExpressive (keeps vocal style) and SeamlessStreaming (simultaneous translation).
- w2v-BERT 2.0 is widely reused as a multilingual encoder, including for Indian languages.
- A baseline in Canary Canary and OWSM OWSM. Its unit-based output links to codec-token TTS (EnCodec, AudioLM).
What it got wrong, and what came next
- Not streaming in v1; see SeamlessStreaming, and Cache-aware streaming Conformer, Kyutai DSM for streaming ASR.
- Loses the speaker's voice and emotion; SeamlessExpressive targets that.
- Heavy: Large will struggle on a 4 GB GPU.
- Text-then-speech adds latency; full-duplex models like Moshi Moshi avoid that pipeline.
Check yourself
Q1Which tasks does SeamlessM4T cover?
Speech-to-speech, speech-to-text, text-to-speech and text-to-text translation, plus ASR. They share a speech encoder, text encoder and text decoder, with a unit generator and vocoder for speech output.
Q2Why generate speech through discrete units?
Units make speech generation a sequence-to-sequence problem over a small vocabulary, which is easier than predicting raw audio. A separate vocoder then adds acoustic detail.
Q3How did Meta get translation data for so many pairs?
By mining: speech and text segments were embedded in a shared multilingual space, and close segments across languages were paired, giving far more data than human-made corpora.
When you're back at a laptop
- Load
facebook/seamless-m4t-v2-large(or Medium) on CPU; transcribe a Hindi clip, then translate it to English text and English speech. - Compare its Hindi ASR with MMS MMS and Whisper.
- Time each stage to see where latency goes.