TTS · Phase 8: Continuous and streaming TTS
Drifting Models
Generative Modeling via Drifting
In one sentence
Drifting Models move the generator's output distribution toward the data distribution during training, using a drifting field that reaches equilibrium when the two match, so at inference the generator needs one step; it set a state-of-the-art one-step ImageNet 256×256 result, and Kyutai used it to retrain Pocket TTS's generation head.
Source note: this page is based on the arXiv abstract and the reading guide. The abstract does not give the formula for the drifting field or the loss, so the mechanism below is described only at the level the abstract states; anything more is marked as interpretation.
Why this paper exists
The abstract frames generative modelling as learning a mapping f whose pushforward distribution (the distribution of f(noise)) matches the data distribution. Diffusion and flow-matching models (Flow Matching) carry out that pushforward iteratively at inference: many network calls, each moving samples a little. That costs latency, which matters a lot inside a per-frame audio loop: in an autoregressive TTS like CALM (CALM (Pocket TTS)), every extra generation step runs once per audio frame.
One-step alternatives existed (GANs, consistency models, distillation), each with trade-offs. Drifting is a new way to get one step.
The core idea
Move the iteration from inference time to training time.
- Define a drifting field that says how generated samples should move to get closer to the data distribution.
- The field is zero, i.e. at equilibrium, when the generated and data distributions match.
- This yields a training objective under which the ordinary neural-network optimizer, step by step, evolves the generator's pushforward distribution along the drift (abstract).
- After training, generation is a single forward pass: noise in, sample out.
How it works
What the abstract specifies, as a recipe:
- A generator network
fmaps noiseεto a samplex = f(ε); this is the whole inference procedure. - A drifting field is defined over samples: for each generated sample, a direction it should move to bring the generated distribution closer to the data distribution. It is zero when the distributions match.
- The training objective turns that field into gradients for
f, so each optimizer step moves the generator's outputs a little along the drift. Over training, the pushforward distribution offevolves toward the data.
Interpretation: you can picture it as "regress f(ε) toward f(ε) + drift, with the target held fixed for that step", the way many one-step methods are trained. The exact field and loss are in the paper's method section; this page does not state them.
Compare:
| Where the "steps" happen | Inference cost | |
|---|---|---|
| Flow matching | ODE solver at inference | N network calls |
| Consistency model | trained to jump to endpoint | 1 to a few calls |
| Drifting | optimizer updates during training | 1 call |
Build the intuition
Analogy: flow matching gives every traveller a map and makes them walk the route step by step each time they travel. Drifting moves the city: during training, the whole set of generated points is nudged, update after update, until it sits exactly on the data. Afterwards each traveller is teleported in one hop.
Interpretation (not from the abstract): think of the 1D two-point example from Flow Matching, with data at −2 and +2. A one-step regressor collapses to 0. A drifting-style objective instead pushes each generated sample toward nearby data and away from regions where generated samples are too dense, so the population of outputs splits into two clusters at ±2. Equilibrium means the clusters match the data's proportions, and the drift is zero. Check the paper for the actual field.
Numbers worth remembering
- ImageNet 256×256, one-step generation: FID 1.54 in latent space, 1.61 in pixel space (abstract). FID: Fréchet Inception Distance, lower is better.
- Inference: 1 network evaluation per sample.
- Per the guide (bookmark #2): Kyutai retrained Pocket TTS's generation head with this objective, described as the first speech model and first autoregressive model trained this way.
Where it shows up today
- Pocket TTS: the CALM-style backbone (CALM (Pocket TTS)) with a one-step head trained by drifting, so each audio frame costs one backbone pass plus one cheap head pass on CPU (per the guide; check Kyutai's blog for details).
- Part of a broader push to one-step generation: consistency models (CALM's original head), rectified-flow distillation, and GAN-style vocoders such as HiFi-GAN (HiFi-GAN) that were always one step.
What it got wrong, and what came next
Too recent for follow-ups to be settled. Open questions worth checking in the paper (interpretation):
- How the drifting field scales to high-dimensional data and how batch size affects it.
- Whether conditional generation (on a text or backbone embedding, as in TTS) needs changes to the objective.
- How stable training is compared to flow matching's simple regression.
Check yourself
Q1What is the pushforward view of generative modelling?
Learn a map f so that applying it to random noise produces samples whose distribution matches the data distribution. Diffusion and flow models apply such a map iteratively at inference; drifting learns a map applied once.
Q2Where does the iteration happen in drifting vs flow matching?
In flow matching, at inference, through an ODE solver with many network calls. In drifting, during training, as optimizer updates evolve the generator's output distribution along the drifting field; inference is one call.
Q3What does "equilibrium" mean for the drifting field?
The field stops moving samples, which by design happens when the generated distribution matches the data distribution. That is the training target.
Q4Why does a one-step head matter so much for Pocket TTS?
In an autoregressive TTS the head runs once per audio frame, so its step count multiplies directly into latency and CPU cost. Going from a few steps to one cuts that part of the per-frame cost proportionally.
When you're back at a laptop
- Read the method section and write down the drifting field and loss (this page leaves them out).
- Implement it on 2D toy data (two moons) or MNIST, as the guide suggests (
feasibility.json). - Compare 1-step drifting against 1-, 4- and 16-step flow matching from Flow Matching on the same toy data; plot samples side by side.