Reading Room

TTS · Phase 6: Codec language-model TTS

Sesame CSM

Crossing the uncanny valley of conversational voice (Sesame research blog)

Sesame research team · Sesame · 2025

Skim10 minDay 9 of 10#70 of 86

In one sentence

Sesame's Conversational Speech Model (CSM) is a Llama-style backbone over interleaved text and Mimi audio tokens from the whole conversation that predicts each frame's first codebook, plus a small audio decoder that fills in the rest, so the voice's prosody fits the dialogue rather than each sentence in isolation.

Source note: this is a blog post, not a peer-reviewed paper, and there is no arXiv version. Details below are from the blog; treat anything not stated there as unknown.

Why this paper exists

Most TTS, even good zero-shot TTS, reads one sentence at a time. In a conversation that sounds wrong: the same words ("oh, really?") should sound surprised, sarcastic or sympathetic depending on what was just said. Sentence-level systems pick a plausible prosody, not the right one. Sesame calls this the gap between naturalness (solved, by standard metrics) and contextual appropriateness (not solved). Their product is a voice companion, so this gap matters more than reading quality.

The core idea

Condition generation on the conversation history, both text and audio, so the model hears how both speakers sounded before and continues appropriately. Make it a single-stage model over codec tokens, using the backbone + depth decoder split from Moshi (Moshi) so it can run with low latency.

How it works

history: [text_1][audio_1][text_2][audio_2]...[text_n]
            (Mimi frames, 12.5 Hz)
                      |
   backbone (Llama-style, 1B / 3B / 8B)
   -> codebook 0 for the next frame
                      |
   audio decoder (100M-300M, own head per codebook)
   -> codebooks 1..N-1 for that frame
                      |
                Mimi decoder -> audio
  • Tokens: text with a Llama tokenizer; audio with Mimi, Kyutai's split-RVQ codec (Moshi), at 12.5 frames/s, with one semantic codebook (codebook 0, distilled from an SSL model) and N-1 acoustic codebooks. The blog does not state N.
  • Backbone: a large autoregressive transformer over the interleaved sequence that predicts codebook 0 of each frame.
  • Audio decoder: a much smaller autoregressive transformer that, for each frame, predicts the remaining codebooks one by one, with a separate linear head per codebook. This is the RQ-Transformer / depth transformer idea.
  • Sizes (backbone / decoder): Tiny 1B / 100M, Small 3B / 250M, Medium 8B / 300M.
  • Training: about 1 million hours of mostly English audio (transcribed, diarized, segmented), sequence length 2,048 tokens (~2 minutes), five epochs.
  • Compute amortization: the decoder is trained on a random 1/16 of frames only (codebook 0 on all frames), which the authors report did not noticeably change decoder loss. This cuts the memory cost of training the depth decoder over long sequences.

Build the intuition

Analogy: a sentence-level TTS is an actor reading lines from cards handed to them one at a time. CSM is an actor who has been in the scene all along and heard every line before theirs.

Why the backbone / decoder split? At 12.5 Hz with several codebooks per frame, flattening everything into the backbone would multiply its sequence length and per-second steps by the number of codebooks. Instead, the big model runs once per frame (12.5 times per second, over long context), and the small decoder handles within-frame detail with no long context. Compare VALL-E's AR + NAR split (VALL-E): same motivation, but here both parts are autoregressive and run per frame, which suits streaming.

What would happen if you trained the decoder on every frame of a 2-minute context? Memory would scale with frames × codebooks across the whole sequence; training on 1/16 of frames keeps it manageable while the backbone still learns from every frame.

Numbers worth remembering

  • Mimi tokens at 12.5 Hz; split at codebook 0 between backbone and decoder.
  • Backbones 1B / 3B / 8B, decoders 100M / 250M / 300M.
  • ~1M hours of mostly English audio; 2,048-token context (~2 minutes).
  • Decoder trained on 1/16 of frames.
  • Human CMOS on Expresso: no clear preference vs real recordings without context, but raters preferred the real recordings when given ~90 s of conversation context.

Where it shows up today

  • Sesame's voice companion demo, which drew wide attention in early 2025; Sesame stated an Apache 2.0 open release (a 1B CSM was published on GitHub as SesameAILabs/csm; check the repo for what was released).
  • Same family as Moshi (Moshi) and Kyutai TTS (Delayed Streams Modeling (TTS)): Mimi tokens, a temporal transformer plus a depth transformer.
  • Context-aware prosody is a target for every voice-agent TTS; compare CosyVoice 2's streaming design (CosyVoice → CosyVoice 2).

What it got wrong, and what came next

  • Not a paper: no ablations, few tables, results partly shown only as charts. Treat claims as reported, not verified.
  • Still a prosody gap with context: human recordings won when listeners had the conversation history.
  • Mostly English; weak multilingual ability. Indian languages would need data like IndicVoices-R (IndicVoices-R).
  • No conversational structure: it models what is said but not turn-taking, interruptions or backchannels; the blog points to full-duplex models like Moshi (Moshi) as the next step.
  • No pretrained text LLM weights in the backbone, unlike CosyVoice 2 or Llasa (Llasa + Spark-TTS).

Check yourself

Q1What does CSM condition on that a normal TTS does not?

The conversation history: earlier turns' text and audio tokens from both speakers, up to about 2 minutes. That lets it pick prosody that fits what was just said, not just a plausible reading of the sentence.

Q2Why split the model at codebook 0?

Codebook 0 is Mimi's semantic codebook, carrying content and coarse structure, which needs long context; the remaining acoustic codebooks depend mostly on the current frame. So a big backbone with long context predicts codebook 0 once per frame and a small decoder fills in the rest cheaply.

Q3What did CSM's listening test show?

Without context, listeners had no clear preference between CSM and real recordings, so isolated naturalness looks saturated. With 90 seconds of conversation context, they preferred real recordings, so contextual prosody is still unsolved.

When you're back at a laptop

  • Clone the SesameAILabs/csm repo and generate a two-turn exchange, once with and once without the previous turn as context; listen for prosody differences.
  • Compare its per-frame structure with Moshi's code: find the backbone and the depth decoder in both.