Flow Matching

Why this paper exists

By 2022, diffusion models were the best image generators, and audio was adopting them too (WaveGrad, DiffWave, Grad-TTS: WaveGrad + DiffWave, Grad-TTS). They had three practical costs:

• Many sampling steps. A diffusion model reverses a noising process gradually, and good samples typically needed tens to hundreds of network evaluations. For TTS that means a slow acoustic model or vocoder.
• Curved trajectories. The path from noise to data implied by the usual diffusion noise schedules bends, so a coarse ODE or SDE solver drifts off it unless you take many small steps.
• Lots of design knobs. Noise schedules, loss weightings, and the choice between predicting noise, data or score all mattered and interacted.

Continuous normalizing flows (CNFs: a network defines a velocity field, and an ODE moves noise to data) offered a cleaner deterministic picture, but training them meant simulating the ODE during training, which was expensive and unstable. Flow matching gives a simulation-free way to train a CNF, and lets you choose the path shape directly.

The core idea

Pick a simple path between one noise sample x_0 and one data sample x_1: the straight line between them. A point moving along that line at constant speed has velocity x_1 − x_0. Train a network v_θ(x, t) to predict that velocity from the current position x_t and time t, averaged over all random pairs. That per-pair target is trivial to compute, and the paper proves that regressing onto it gives the same gradients as regressing onto the true (intractable) velocity field of the whole distribution. At inference, start from fresh noise and follow v_θ from t = 0 to t = 1.

How it works

The path and the target velocity

Convention in this paper: t = 0 is noise, t = 1 is data.

x_0 ~ N(0, I)          noise
x_1 ~ p_data           a real sample (e.g. a mel clip)
t   ~ Uniform(0, 1)

x_t = (1 − t)·x_0 + t·x_1      straight line

Derive the velocity by differentiating the path with respect to time:

dx_t/dt = d/dt [ (1 − t)·x_0 + t·x_1 ]
        = −x_0 + x_1
        = x_1 − x_0

So the target does not depend on t: each pair moves at constant speed in a straight line. Using x_0 = (x_t − t·x_1)/(1 − t), the same target can be written (x_1 − x_t)/(1 − t): "the remaining distance to the data point, divided by the remaining time". That form is useful below.

The loss

L(θ) = E_{t, x_0, x_1} || v_θ(x_t, t) − (x_1 − x_0) ||²

• v_θ: the network (a U-Net for images; a Transformer for speech in Voicebox and F5-TTS: Voicebox, F5-TTS).
• t: time, fed in through an embedding.
• ||·||²: squared error summed over all dimensions (for mel: frames × mel bins).

The paper's actual optimal-transport path keeps a tiny noise floor: x_t = (1 − (1 − σ_min)·t)·x_0 + t·x_1, target x_1 − (1 − σ_min)·x_0. With σ_min = 0 it reduces to the version above, which is what most code uses.

Why regressing on per-pair targets works

The quantity you really want is the marginal velocity u(x, t): for all pairs whose line passes through x at time t, their average velocity. You cannot compute it, because it requires knowing which data points could have produced x. But for a squared-error loss, the minimizer of E || v(x_t, t) − (x_1 − x_0) ||² is exactly the conditional mean E[x_1 − x_0 | x_t = x], which is u(x, t). The paper calls this conditional flow matching (CFM) and proves the CFM and full flow-matching losses have the same gradient. This is the same trick that makes denoising score matching work.

Training and sampling

TRAIN (one step)                SAMPLE (N Euler steps)
x_1 <- batch of data            x <- N(0, I)
x_0 <- randn_like(x_1)          for i in 0..N-1:
t   <- rand()                     t = i / N
x_t <- (1-t) x_0 + t x_1          x = x + (1/N) * v(x, t)
loss = |v(x_t,t) - (x_1-x_0)|^2 return x   # approx data

Each Euler step costs one network evaluation; the step count is called NFE (number of function evaluations). Higher-order solvers (midpoint, adaptive Runge-Kutta) also work. The paper also shows that diffusion is a special case: diffusion paths, written as flows, are a different (curved) choice of x_t.

Build the intuition

Work it in 1D. Noise x_0 ~ N(0, 1). Data is two points, x_1 = −2 or +2, each with probability 1/2.

The per-pair velocity is x_1 − x_0. The network learns its average given the current position. Because x_t | x_1 ~ N(t·x_1, (1 − t)²), Bayes' rule gives

E[x_1 | x_t = x] = 2·tanh( 2·t·x / (1 − t)² )
v(x, t) = ( E[x_1 | x_t = x] − x ) / (1 − t)

At t = 0, x_t is pure noise and says nothing about which data point it is heading to, so E[x_1 | x] = 0 (the data mean) and v(x, 0) = −x.

Start from x_0 = 0.5 and integrate to t = 1:

What happened:

• 1 step lands on 0, the average of the two modes, which is not a valid sample. A single Euler step from t = 0 always returns the data mean: the blurry, averaged output you get when you ask a regression model for a multimodal answer.
• 2 steps: the first step still heads to the mean. At t = 0.5 the point has moved to the positive side, the field "commits" to +2, and the last step jumps to the posterior mean 1.52. Closer, but still pulled toward 0.
• 4 steps is enough here, and more steps change nothing visible.

Two lessons carry over to speech. First, the individual training paths are straight but the learned marginal field is not, because paths from different pairs cross; that is why one step fails and a few steps work. Second, the hard decisions happen early (near noise), which is exactly why F5-TTS's sway sampling puts more steps near t = 0 (F5-TTS).

Analogy: each training pair is a taxi with a known destination driving in a straight line. The network only sees where a taxi is right now and must guess its velocity. Near the depot every taxi looks the same, so the best guess is "towards the middle of town". Further out, the position reveals the destination and the guess sharpens.

Numbers worth remembering

• Straight-path target: x_1 − x_0, constant in time; the loss is a single MSE.
• σ_min in the paper's OT path is a small constant (on the order of 1e-4 to 1e-5 in common code; check the paper for its setting).
• The paper reports better likelihood and sample quality than diffusion baselines on ImageNet with fewer function evaluations (check the paper's tables for exact FID and NFE).
• Speech models built on it typically use ~10-32 NFE (Matcha-TTS: a handful; F5-TTS: 32 by default, fewer with sway sampling).

Where it shows up today

• Matcha-TTS (Matcha-TTS): the smallest clean speech example, OT-CFM decoder.
• Voicebox (Voicebox), E2 TTS (E2 TTS), F5-TTS (F5-TTS): flow matching over mel spectrograms, at scale.
• CosyVoice / CosyVoice 2 (CosyVoice → CosyVoice 2): LLM for tokens, flow matching for mel; the same pattern sits in other open models.
• Concurrent work reached the same idea: rectified flow (Liu et al.) and stochastic interpolants (Albergo and Vanden-Eijnden). Image models such as Stable Diffusion 3 train with rectified-flow objectives.
• CALM (CALM (Pocket TTS)) and drifting (Drifting Models) push the other way: fewer, ideally one, generation steps.

What it got wrong, and what came next

• Still multi-step. Straight per-pair paths do not make the marginal field straight, as the 1D example shows. Reflow / rectified flow, consistency models (used in CALM, CALM (Pocket TTS)) and drifting (Drifting Models) all aim at one or few steps.
• Says nothing about alignment. For TTS, you still need to tell the model how long the speech is and where each phone goes; Voicebox uses durations, E2/F5 learn alignment implicitly (E2 TTS, F5-TTS).
• Not streaming by itself. A non-autoregressive flow model generates the whole utterance at once. CosyVoice 2 adds chunk-aware flow matching (CosyVoice → CosyVoice 2); Kyutai goes autoregressive (Delayed Streams Modeling (TTS)).

Check yourself

When you're back at a laptop

• Train a 3-layer MLP (input x, t; output velocity) on 2D two-moons with the loss above. CPU is fine; your 3050 is plenty.
• Sample with 1, 4 and 16 Euler steps and scatter-plot each. Expect the 1-step plot to collapse towards the middle.
• Reproduce the 1D table: code v(x, t) from the formula and check that 1 step gives the mean.

x1 = data_batch(); x0 = torch.randn_like(x1)
t = torch.rand(len(x1), 1)
xt = (1 - t) * x0 + t * x1
loss = ((net(xt, t) - (x1 - x0)) ** 2).mean()