TTS · Phase 4: End-to-end and expressive TTS

VITS

Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech

Kim, Kong, Son · Kakao Enterprise / KAIST · 2021

Deep read25 minDay 7 of 10#54 of 86

In one sentence

VITS trains one model from text straight to waveform by making the waveform the output of a VAE (variational autoencoder) whose decoder is a HiFi-GAN generator, with a text-conditioned prior made expressive by a normalizing flow, alignment learned by MAS, and a stochastic duration predictor so the same sentence can be read with different rhythms.

Why this paper exists

Two-stage TTS (acoustic model → mel → vocoder, e.g. Glow-TTS Glow-TTS + HiFi-GAN HiFi-GAN) has a seam: the vocoder is trained on real mels but at inference sees predicted mels, which are smoother and slightly off, so quality drops (people fine-tuned vocoders on predicted mels to patch this). The mel itself is a hand-designed bottleneck that may not be the best intermediate. Earlier end-to-end attempts (like FastSpeech 2s) were worse than the best two-stage systems. Also, deterministic duration predictors (FastSpeech 2) always read a sentence with the same rhythm, which sounds flat.

The core idea

Replace the mel with a learned latent z, and train everything jointly:

A posterior encoder sees the real audio's linear spectrogram and produces z. The HiFi-GAN-style decoder must turn z back into the waveform. That is a VAE, trained with a reconstruction loss plus a KL term.
The prior says what z should look like given only the text. A simple Gaussian per phoneme would be too blunt, so a flow transforms z before comparing it with the text's Gaussians.
At inference, sample from the prior (text → Gaussians → inverse flow → z) and decode. No mel anywhere.

How it works

Training path

linear spec (513 x T)        phonemes (L)
   │                            │
   ▼                            ▼
posterior encoder          text encoder
   │ z ~ q(z|x) (192 x T)       │ μ, σ per phoneme
   ├──────────┐                 │ (192 x L)
   │          ▼                 │
   │       flow f               │
   │          │ f(z)            │
   │          ▼                 ▼
   │       MAS: best monotonic alignment A
   │       KL( f(z) vs N(μ[A], σ[A]) )
   │                            │
   │                 stochastic duration
   │                 predictor (learns A's
   │                 durations, sg on input)
   ▼ random 32-frame slice of z
decoder (HiFi-GAN generator)
   ▼
waveform slice → mel L1 + MPD/MSD + feat. match

Inference path

phonemes → text encoder → μ, σ per phoneme
        → stochastic duration predictor
          (noise in) → durations d
        → repeat μ, σ by d       (192 x T)
        → z_p = μ + σ·ε·τ        (τ = noise scale)
        → inverse flow f⁻¹ → z
        → decoder → waveform

The posterior encoder and the discriminators are used only in training.

The parts

Posterior encoder: non-causal WaveNet-style residual blocks over the linear spectrogram (it sees full-resolution detail, more than a mel), outputting a mean and std per frame for z (192 dims).
Text encoder: a Transformer encoder over phonemes, plus a linear layer giving μ and σ per phoneme.
Flow: a stack of affine coupling layers (like Glow-TTS's decoder, Glow-TTS), conditioned on the speaker for multi-speaker models. It lets a simple per-phoneme Gaussian describe a complex frame-level distribution.
MAS (monotonic alignment search): dynamic programming that finds the monotonic phoneme-to-frame alignment A maximizing the likelihood of f(z) under the text Gaussians. No external aligner, no attention.
Stochastic duration predictor: a small flow over (log) durations trained with a variational bound, so at inference you sample durations from noise. Its input has a stop-gradient so it does not disturb the text encoder.
Decoder: a HiFi-GAN V1 generator. To save memory, it decodes only a random 32-frame window of z per training example (8,192 samples at hop 256).

Loss

L = L_recon + L_kl + L_dur + L_adv(G) + L_fm(G)
L_recon = 45 · || mel(y) - mel(ŷ) ||_1
L_kl    = log q(z | x_lin) - log p(z | text, A)
log p(z | text, A) = log N(f(z); μ[A], σ[A])
                   + log |det ∂f(z)/∂z|

y is the real waveform slice, ŷ the decoded one, mel(·) a mel transform (used only for the loss), x_lin the linear spectrogram, A the MAS alignment, μ[A] the text means expanded to frames. L_dur is the duration predictor's negative variational bound. L_adv and L_fm are HiFi-GAN's adversarial and feature-matching losses (HiFi-GAN).

Build the intuition

Analogy: The posterior encoder is a student who listens to the real recording and writes rich notes (z). The decoder learns to perform from those notes. The text side learns to write notes that look like the student's, using only the script. At show time the student is gone, and the text side's notes go straight to the performer.

What does the flow buy? Without it, the prior says "every frame of phoneme /a/ is a sample from one Gaussian", and the KL forces the posterior to be that simple too, losing detail like pitch movement within a vowel. The flow warps the space so that a simple Gaussian in the warped space corresponds to a complex, frame-varying distribution in z space.

Noise scale τ: Piper exposes noise_scale (for z) and noise_w (for durations). Set both to 0 and every reading is identical and flat; around 0.667 and 0.8 (common defaults) gives natural variation; too high gives slurring.

Numbers worth remembering

LJSpeech MOS: VITS ~4.43 vs ground truth ~4.46, above Tacotron 2 + HiFi-GAN and Glow-TTS + HiFi-GAN (check Table 1).
Latent z: 192 channels at the spectrogram frame rate (hop 256 at 22.05 kHz, ~86 frames/s).
Decoder trained on 32-frame windows (8,192 samples).
Recon loss weight 45 (same as HiFi-GAN's mel weight).
A single-speaker VITS is ~30-40M parameters (check the official repo); it runs much faster than real time on GPU and, in Piper's smaller configs, on CPU.

Where it shows up today

Piper: the popular fast on-device TTS (Raspberry Pi to laptop CPU), VITS models exported to ONNX.
MMS-TTS (Meta, MMS): one VITS model per language for 1,100+ languages, many Indian languages among them.
Coqui TTS's VITS models, and YourTTS (YourTTS) which adds zero-shot cloning on top.
Many Indian-language community models and AI4Bharat comparisons (Indic TTS (AI4Bharat)).
The "VAE + adversarial decoder" pattern continues in StyleTTS 2 (StyleTTS → StyleTTS 2) and in neural codecs (SoundStream, EnCodec).

What it got wrong, and what came next

Limited zero-shot ability: speakers come from an ID embedding. YourTTS (YourTTS) adds speaker-encoder embeddings; codec LMs (VALL-E) and flow matching (Voicebox, F5-TTS) do far better cloning.
Prosody is still somewhat flat on expressive text; StyleTTS 2 (StyleTTS → StyleTTS 2) adds style diffusion and an SLM discriminator.
Offline per sentence: the text encoder and duration predictor see the whole sentence. Fine for Piper (fast enough per sentence), but adds a sentence of delay in a voice agent; see streaming TTS (Delayed Streams Modeling (TTS)).
Training is complex (many losses, GAN instability) and needs MAS, which assumes monotonic alignment and clean transcripts.

Check yourself

Q1Which parts of VITS are used only during training?

The posterior encoder (it needs the real spectrogram), MAS (it needs the posterior z), and the discriminators. At inference you use the text encoder, the stochastic duration predictor, the inverse flow and the decoder.

Q2Why does VITS need a flow in the prior?

The text encoder gives one Gaussian per phoneme, repeated over its frames. That is too simple to describe real frame-level variation, and the KL term would force the posterior to be that simple. The flow transforms z so a simple Gaussian in the flow's output space corresponds to a rich distribution over z.

Q3Why is the posterior encoder fed a linear spectrogram rather than a mel?

It only exists in training, so there is no need for a compact or predictable input. A linear spectrogram keeps more high-frequency detail, which helps the latent carry what the decoder needs. The mel is used only for the reconstruction loss, because it matches perception.

Q4What would happen if you replaced the stochastic duration predictor with a deterministic one?

The model would always read a sentence with the same rhythm. The paper reports the stochastic version sounds more natural, and it lets you sample varied readings; Piper's noise_w controls that variation.

Q5Why decode only a random 32-frame window of z in training?

The decoder runs at the sample rate and its activations are huge; decoding whole utterances would not fit in GPU memory with the rest of the model. Random windows still train the decoder on all of z over time, and at inference it decodes the whole utterance.

Q6How does MAS here differ from Glow-TTS?

It is the same dynamic program, but it aligns the text Gaussians with f(z), the flow-transformed posterior latent, instead of with the mel through Glow-TTS's decoder flow.

Q7What fixes the train/inference mismatch of two-stage systems?

Joint training. The decoder learns from latents produced by the same model, and the KL term pulls the posterior and the text prior together, so what the decoder sees at inference is close to what it saw in training.

When you're back at a laptop

pip install piper-tts, download an en_US voice (or an Indian-language voice if the Piper voice list has one), synthesize on CPU and measure RTF. Vary noise_scale, noise_w and length_scale.
Open the voice's .onnx.json config and the VITS repo's models.py; map TextEncoder, PosteriorEncoder, ResidualCouplingBlock, StochasticDurationPredictor and Generator to the diagram.
Try MMS-TTS for a language you speak (facebook/mms-tts-hin via transformers) and compare with Piper.

arXiv ↗PDF ↗