Reading Room

ASR · Phase 5: Scale and multilinguality

MMS

Scaling Speech Technology to 1,000+ Languages

Pratap, Tjandra, Shi, et al., Auli · Meta AI · 2023

Skim7 minDay 3 of 10#24 of 86

In one sentence

MMS (Massively Multilingual Speech) builds speech technology for 1,000+ languages by using recordings of religious texts (mainly New Testament readings) as labelled data, pretraining wav2vec 2.0 on 1,400+ languages, and fine-tuning CTC ASR for 1,100+ languages with small per-language adapters.

Why this paper exists

There are about 7,000 languages; ASR covered around 100 (Whisper Whisper, USM Google USM) because only those have labelled speech. Many Indian languages and dialects had almost none. Is there any source of paired speech and text for a thousand languages?

The core idea

The New Testament has been recorded in well over a thousand languages, each with its text. Automatic alignment turns that into tens of hours of labelled speech per language. Add self-supervised pretraining on unlabelled audio, and each language needs little labelled data.

How it works

  1. Data: New Testament audio and text for ~1,100 languages (labelled), plus recordings without usable text for 3,800+ languages (unlabelled).
  2. Forced alignment: CTC CTC alignment cuts chapters into verse-sized segments, with an efficient GPU implementation for hours-long audio; text is romanised (uroman) so one aligner covers many scripts. Poorly aligned segments are dropped.
  3. Pretraining: wav2vec 2.0 wav2vec 2.0 on 1,400+ languages, 300M and 1B parameter sizes.
  4. ASR: one shared model with a CTC head, plus a small adapter per language (a few million parameters in each Transformer layer) and per-language vocabulary. Switching language means loading a different adapter.
  5. Also: language ID for ~4,000 languages and VITS VITS TTS for ~1,100.

Build the intuition

Analogy: the recordings are a spoken Rosetta Stone: one text, read aloud in a thousand languages.

Why adapters? One shared head for 1,100 languages lets high-resource languages dominate. Adapters give each language its own small capacity on top of shared acoustic knowledge, and train cheaply on ~30 hours.

On Hindi call-centre audio, expect trouble: the training domain is formal read scripture, so code-mixed Hinglish, numbers and product names suffer.

Numbers worth remembering

  • Pretraining 1,406 languages; ASR 1,107; language ID ~4,017; TTS ~1,100.
  • Labelled data averages a few tens of hours per language (check the paper for the exact figure).
  • Models: 300M and 1B parameters.
  • Reports roughly halving Whisper's WER on the FLEURS languages both cover, with far less labelled data (check the table for the language count).

Where it shows up today

  • facebook/mms-1b-all, mms-lid-* and mms-tts-* in Transformers: the baseline for languages Whisper misses.
  • Its CTC forced aligner is in torchaudio (compare WhisperX WhisperX).
  • An Indic baseline next to IndicWav2Vec IndicWav2Vec and IndicWhisper Vistaar, evaluated on Kathbath IndicSUPERB; compare its TTS with AI4Bharat's Indic TTS (AI4Bharat).

What it got wrong, and what came next

  • Domain bias: read religious speech does not match conversation; in-domain fine-tuning (Vistaar) is usually needed.
  • CTC without a strong LM misspells; the paper adds n-gram LMs for some languages.
  • Data licensing limits some reuse; check before commercial use.
  • Meta's next multilingual encoder was w2v-BERT 2.0 in SeamlessM4T SeamlessM4T.

Check yourself

Q1Where does MMS get labelled data for 1,000+ languages?

From New Testament recordings with their text. CTC forced alignment cuts these into short segments paired with the matching verses.

Q2Why adapters instead of separate fine-tuned models?

Each adapter is a few million parameters, so one 1B backbone plus 1,100 adapters is far smaller than 1,100 models, and low-resource languages share the pretrained representation while keeping their own vocabulary.

Q3What would limit MMS in a Tamil voice agent?

Its labelled data is formal read text, so conversation, English code-mixing, numbers and brand names are weaker. Fine-tune on in-domain data or use an Indic-specific model.

When you're back at a laptop

  • Load facebook/mms-1b-all, run processor.tokenizer.set_target_lang("hin") and model.load_adapter("hin"), transcribe Hindi; repeat for Tamil.
  • Run mms-lid-126 on Hinglish clips.
  • Try torchaudio's MMS aligner on a clip with a known transcript.
arXiv ↗PDF ↗