Reading Room

ASR · Phase 7: Speech meets LLMs

Moshi

Moshi: a speech-text foundation model for real-time dialogue

Défossez, Mazaré, Orsini, Royer, Pérez, Jégou, Grave, Zeghidour · Kyutai · 2024

Deep read25 minDay 4 of 10#29 of 86

In one sentence

Moshi is a speech-to-speech dialogue model that listens and speaks at the same time: it models the user's audio and its own audio as parallel token streams from the Mimi codec, predicts a time-aligned text "inner monologue" before its own audio, and runs at about 200 ms latency in practice.

Why this paper exists

A classic voice agent is a cascade: VAD (voice activity detection) decides the user stopped, ASR transcribes, a text LLM answers, TTS speaks. Problems:

  • Latency adds up: each stage waits for the previous one.
  • Information is lost: text drops emotion, emphasis and laughter.
  • Turn-taking is rigid: overlaps, interruptions and back-channels ("mm-hm") are handled badly.

Moshi replaces the whole cascade with one model that treats dialogue as speech-to-speech generation.

The core idea

Turn audio into a low-frame-rate stream of discrete tokens, and let a text-LLM-sized Transformer model three streams in lockstep, one step every 80 ms: the user's audio, Moshi's own audio, and Moshi's text. Because the user's stream is always being read while Moshi's stream is being written, the model is full duplex: listening never stops, even while speaking. Predicting text tokens just before audio tokens lets it keep an LLM's language ability.

How it works

1. Mimi: the audio tokenizer

  • A neural audio codec (encoder, quantizer, decoder) on 24 kHz audio, producing 12.5 frames per second (80 ms each).
  • Residual vector quantization (RVQ): each frame is quantized by a stack of codebooks; codebook 1 encodes the frame coarsely, codebook 2 encodes the leftover error, and so on. Moshi uses 8 codebooks per frame (~1.1 kbps; check the paper).
  • Semantic distillation: the first codebook is trained to match features from a self-supervised speech model (WavLM, WavLM), so it carries mostly what was said, while later codebooks carry how it sounds. This is the split AudioLM (AudioLM) used two separate tokenizers for; Mimi does it in one.
  • Fully causal (streaming) encoder and decoder, so it can tokenize and decode live.

2. Helium: the text backbone

A 7B-parameter text LLM pretrained by Kyutai (~2T tokens; check the paper), used to initialise the big Transformer so Moshi starts with language knowledge.

3. RQ-Transformer: tokens over time and depth

At each 80 ms step there are many tokens (text + 8 Moshi codebooks + 8 user codebooks). Flattening them into one sequence would make the sequence 17× longer. Instead:

time step s (80 ms)
inputs: text_{s-1}, moshi_{s-1}[1..8], user_{s-1}[1..8]
                      |  sum embeddings
                      v
  +-------------------------------------------+
  | Temporal Transformer (7B, from Helium)    |
  | causal over time steps s = 1, 2, 3, ...   |
  +-------------------------------------------+
                      | context vector z_s
                      v
  +-------------------------------------------+
  | Depth Transformer (small)                 |
  | predicts within step s, in order:         |
  |  text_s -> moshi_s[1] -> ... -> moshi_s[8]|
  +-------------------------------------------+
         user_s[1..8] come from the mic (Mimi)
  • Temporal Transformer: one step per frame, so the expensive model runs only 12.5 times per second.
  • Depth Transformer: a small model that, given z_s, predicts the frame's tokens one after another, each conditioned on the ones before it in the frame.
p(frame_s) = Π_k p( tok_{s,k} | z_s, tok_{s,<k} )
z_s = Temporal( all tokens at steps < s )
  • s: time step, k: position within the frame (text, then codebook 1..8).
  • Acoustic delay: the acoustic codebooks are shifted 1-2 steps behind the semantic one, which the paper found improves quality (check the exact delay).

4. Inner monologue

Moshi also predicts its own text, aligned in time to its speech, and placed before its audio tokens in each frame. Words are spread over frames using timestamps, with PAD tokens between words (and an end-of-padding marker before a word). The text acts like a scaffold: the model decides what to say in text, then how it sounds in audio. The paper reports this markedly improves the linguistic quality of the speech.

Side effect: by changing the delay between text and audio, the same machinery gives streaming ASR (text after audio) or streaming TTS (audio after text). That is the seed of DSM (Kyutai DSM, Delayed Streams Modeling (TTS)).

5. Full duplex

time ->   s1   s2   s3   s4   s5   s6
user:    hi   how  are  you  ...  [silence]
moshi:  [sil] [sil] mm  [sil] I'm  fine
text:    PAD  PAD  PAD  PAD  I'm  fine

There are no turns: both audio streams exist at every step. Silence is just tokens for silence. Moshi can back-channel, be interrupted, or start talking while the user is finishing. It is trained on large amounts of unlabelled speech, then on multi-stream conversation data (the Fisher telephone corpus, with each speaker on a separate channel) and synthetic dialogues; check the paper for hours.

Latency

Theoretical latency 160 ms: one 80 ms frame for Mimi plus 80 ms of acoustic delay. About 200 ms in practice on an L4-class GPU.

Build the intuition

Analogy: a phone call on a two-track tape recorder. One track records the other person, one track records you. A cascade only plays back one track at a time; Moshi writes its own track while the other one keeps rolling. The inner monologue is like a speaker reading their own teleprompter one word ahead of their mouth.

Why a depth Transformer instead of predicting the 8 codebooks independently from z_s? Codebook k encodes the residual of codebooks 1..k-1, so it depends on their exact values; independent picks can clash and garble audio.

What would happen without inner monologue? The model must learn language only from audio tokens, which carry few bits of linguistic content per token. The paper's ablation shows much worse factuality and coherence; text is a far denser carrier of meaning.

Numbers worth remembering

  • Mimi: 24 kHz audio to 12.5 Hz tokens, 8 RVQ codebooks used by Moshi (~1.1 kbps).
  • Temporal Transformer ~7B parameters (Helium init); depth Transformer is much smaller.
  • 17 tokens per 80 ms step: 1 text + 8 Moshi audio + 8 user audio.
  • Latency: 160 ms theoretical, ~200 ms practical.
  • Pretraining on ~7M hours of audio (check the paper for the exact figure).

Where it shows up today

  • Kyutai's STT and TTS (DSM, Kyutai DSM, Delayed Streams Modeling (TTS)) reuse Mimi and the multi-stream design; Unmute and other Kyutai demos build on these.
  • Sesame CSM (Sesame CSM) uses Mimi and a similar backbone-plus-small-decoder split.
  • The RQ-Transformer idea (big model over time, small model over codebooks) is now common in codec language models after VALL-E (VALL-E).
  • Codec lineage: SoundStream (SoundStream), EnCodec (EnCodec).

What it got wrong, and what came next

  • Knowledge and reasoning lag a text LLM of the same size; speech-to-speech training dilutes the text skills. Many production agents still use a cascade with a strong text LLM.
  • Hard to control: tools, retrieval and system prompts are harder to plug into a speech-to-speech model than into a text LLM.
  • English-centric, and a 7B model will not run on a 4 GB laptop GPU without heavy quantization.
  • Discrete RVQ tokens cap audio quality; CALM (CALM (Pocket TTS)) explores continuous audio modelling.
  • Encoder-plus-LLM models (SALMONN / Qwen-Audio / Qwen2-Audio) took the other route: keep the LLM in text and only add a speech encoder.

Check yourself

Q1Why does Mimi distil a self-supervised model into its first codebook?

So that the first codebook carries semantic, phonetic content and later ones carry acoustic detail. The language model then gets a token stream where the most important information is in a predictable place, which makes generation more coherent.

Q2What problem does the RQ-Transformer solve?

Each 80 ms step has many tokens. Flattening them would make the sequence many times longer and the big model many times more expensive. The temporal Transformer runs once per step, and a small depth Transformer handles the dependencies among tokens inside the step.

Q3What is the inner monologue and why put text before audio?

Time-aligned text of Moshi's own speech, predicted in each frame before the audio tokens. Text carries meaning densely, so deciding the words first and then their sound leverages the LLM's language skill and improves the content of what Moshi says.

Q4What makes Moshi full duplex?

The user's audio and Moshi's audio are two separate streams present at every time step. The model always reads the user stream and always writes its own, so listening and speaking overlap freely and there is no explicit turn switch.

Q5Where does the 160 ms theoretical latency come from?

One 80 ms Mimi frame, plus the 80 ms acoustic delay between the semantic and acoustic tokens. Real systems add compute time, giving about 200 ms.

Q6How does changing the text/audio delay turn Moshi's machinery into ASR or TTS?

If text lags audio, the model hears speech before writing its text: streaming ASR. If audio lags text, it reads text before speaking: streaming TTS. That observation is DSM.

When you're back at a laptop

  • Install moshi from Kyutai's repo and run Mimi alone: encode a WAV to tokens, print the shape (should be 8 codebooks × ~12.5 tokens/s), decode it back and listen with 1, 4 and 8 codebooks.
  • Run the Moshi web demo (or the MLX build on a Mac) and test interruptions and back-channels.
  • Draw the 17-tokens-per-step layout from memory.
arXiv ↗PDF ↗