CALM (Pocket TTS)

Why this paper exists

The abstract states the problem directly. Audio language models (ALMs: models that generate audio as a sequence, like a text LLM) represent audio as discrete tokens. Text tokens are invertible: you lose nothing by tokenizing text. Audio tokens are not: they come from lossy neural codecs with a limited bitrate. So:

• To get higher quality you need more tokens (more codebooks per frame or a higher frame rate).
• More tokens mean more compute per second of audio.
• So discrete ALMs face a fidelity vs compute trade-off.

For a voice agent on a CPU, that trade-off is the whole problem: every extra token per frame is extra work in the real-time loop.

Background from earlier pages: VALL-E (VALL-E) and Moshi (Moshi) use residual vector quantization (RVQ), several stacked codebooks per frame, and need extra machinery (a second NAR model, or Moshi's small "depth Transformer") to generate the codebooks within each frame. MELLE (MELLE) already showed AR models can predict continuous frames, but with a simple per-step Gaussian.

The core idea

Do not quantize. Generate continuous latent frames from an audio VAE (variational autoencoder: an encoder compresses waveform to a continuous latent sequence, a decoder reconstructs it), one frame at a time, autoregressively.

Split the work in two, as the abstract describes:

1. A large Transformer backbone runs over the sequence and outputs a contextual embedding at every timestep: "what should come next, given everything so far".
2. A small MLP head takes that embedding and generates the next continuous VAE frame through consistency modelling.

Because the head generates a sample (not a single regression average), it can represent the many plausible next frames without blurring them, and because it is small, running it per frame is cheap.

How it works

text / prior audio context
        |
  [large causal Transformer]   one pass per frame
        |
  context embedding h_t
        |
  [small MLP head]  <- noise
   consistency model, few steps
        |
  continuous VAE frame z_t
        |  (fed back as input for t+1)
        v
  [VAE decoder] -> waveform

Consistency modelling (background, not from the abstract)

A consistency model (Song et al., 2023) learns a function f(x_s, s) that maps a noisy point at any noise level s on a diffusion/flow trajectory directly to the trajectory's clean endpoint. If f is consistent (every point on one trajectory maps to the same endpoint), then one call from pure noise gives a sample, and a few calls (denoise, re-noise a bit, denoise again) improve it. Compare flow matching (Flow Matching), where you integrate a velocity with many small steps.

In CALM, as described in the abstract, this generator is conditioned on the backbone's embedding:

z_t = f_head( noise, s_max | h_t )        1 step
(optionally refine with a few more steps)

z_t is the next VAE frame, h_t the backbone embedding at step t, s_max the maximum noise level.

Why this split is efficient (interpretation)

The expensive Transformer runs once per frame, as in any AR model. The few generation steps happen only in the tiny MLP head, so even several head steps add little cost. This mirrors the "big temporal model + small per-frame model" split in Moshi's RQ-Transformer (Moshi), but the small model now outputs one continuous vector instead of a stack of RVQ tokens.

Pocket TTS

Per the abstract, the release includes Pocket TTS, an open-source 100M-parameter TTS model that runs faster than real time on a laptop CPU (code at github.com/kyutai-labs/pocket-tts). Per the guide (bookmark #2), Kyutai later retrained Pocket TTS's generation head with the drifting objective (Drifting Models) to get a one-step head. Check the Kyutai blog and model card for which head the current release uses.

Build the intuition

Analogy: a discrete ALM is a pianist who may only play notes from a fixed scale, and to sound more nuanced must play more notes per beat. CALM's pianist can bend any note continuously, so one note per beat carries the nuance.

A worked comparison (illustrative numbers, not from the paper): suppose a discrete codec runs at 12.5 frames/s with 8 RVQ codebooks. That is 100 tokens per second of audio, and the model must produce all 8 per frame. A continuous VAE at the same 12.5 frames/s needs 12.5 backbone steps per second, each followed by a cheap head call. Raising quality in the discrete case means adding codebooks (more tokens); in the continuous case it means a better VAE and head, with the backbone step count unchanged.

What would happen if the head were a plain regressor (predict z_t directly with MSE)? It would output the average of plausible next frames: muffled, monotone audio, the same failure as 1-step flow matching from noise in the 1D example in Flow Matching. The head must be a sampler, which is why it is a consistency model.

What would happen if you removed the backbone and let the MLP see only the previous frame? It would lose long-range context (what words have been said, the speaker's style), so it could not follow text or keep a voice. The backbone carries the context; the head carries the local randomness.

Numbers worth remembering

• Pocket TTS: 100M parameters, faster than real time on a laptop CPU (abstract).
• Evaluated on speech and music; the abstract claims better efficiency and fidelity than state-of-the-art discrete ALMs (check the paper's tables for metrics).
• Generation unit: one continuous VAE frame per AR step, produced by an MLP head (frame rate and latent size: check the paper).
• Samples: hf.co/spaces/kyutai/calm-samples (abstract).

Where it shows up today

• Pocket TTS (Kyutai), the model behind bookmarks #2 and #74 in the guide: CALM backbone + one-step head + VAE decoder, streaming on CPU.
• Continues the continuous-AR line from MELLE (MELLE) and the continuous-latent idea of NaturalSpeech 2 (NaturalSpeech 2 → NaturalSpeech 3).
• Compared to F5-TTS (F5-TTS): both use continuous representations, but F5 is NAR over the whole utterance, while CALM is AR, so it can stream.
• Kyutai's streaming designs: delayed streams (Delayed Streams Modeling (TTS), Kyutai DSM) and Moshi's codec and backbone ideas (Moshi).

What it got wrong, and what came next

Interpretation, not from the abstract:

• Few-step is still more than one step. A one-step head is cheaper still; drifting (Drifting Models) trains a generator that needs one step, and Kyutai applied it to Pocket TTS's head (per the guide).
• AR error accumulation. Continuous AR models feed their own outputs back in, so small errors can drift over long utterances; check how the paper handles exposure to its own samples (e.g. noise augmentation) before assuming.
• Quality ceiling set by the VAE. No quantization loss, but the VAE's reconstruction still limits fidelity.
• Text streaming (starting before the full sentence is known) is a separate design question; DSM (Delayed Streams Modeling (TTS)) addresses it for Kyutai TTS.

Check yourself

When you're back at a laptop

• Install Pocket TTS from github.com/kyutai-labs/pocket-tts and run it on CPU.
• Measure TTFA (time to first audio) and RTF for a short and a long sentence; compare with F5-TTS on your RTX 3050.
• Read the paper's method section and fill in what this page leaves open: VAE frame rate, latent size, backbone size, head steps.
• Toy version: train a small causal Transformer + MLP head on sequences of 2D points, first with MSE (watch it average), then with a few-step generative head.