StyleTTS → StyleTTS 2

Why this paper exists

By 2022, VITS (VITS) and FastSpeech 2 (FastSpeech 2) produced clean speech, but prosody (rhythm, pitch movement, emphasis) was often flat or "averaged", and there was no clean handle to control speaking style. Global Style Tokens (Global Style Tokens) gave a style embedding but in an autoregressive Tacotron with attention problems. Diffusion acoustic models like Grad-TTS (Grad-TTS) gave diversity but were slow because they diffused the whole mel. And evaluation kept showing a gap to real recordings.

The core idea

Separate what is said from how it is said. Compress "how" into one fixed-size style vector per utterance, let it modulate every layer of the decoder and the prosody predictors, and make the rest of the model deterministic given that vector.

• StyleTTS: the style vector comes from a reference clip, via a style encoder.
• StyleTTS 2: the style vector is sampled with diffusion conditioned on the text, so you get natural, varied prosody with no reference. Diffusion over a 256-number vector is very cheap compared with diffusing a whole mel.

How it works

StyleTTS (2022)

• Style encoder: a conv network over the reference mel → style vector s.
• AdaIN (adaptive instance normalization): in each decoder layer, normalize the features per channel, then scale and shift with values predicted from s: AdaIN(h, s) = γ(s) · (h - mean(h)) / std(h) + β(s). Style controls the statistics of every layer.
• Transferable Monotonic Aligner (TMA): an attention aligner pretrained as an ASR model, then fine-tuned during TTS training with monotonic constraints, giving phoneme-to-frame alignments.
• Predictors: a duration predictor and a prosody predictor (pitch F0 and energy), both conditioned on s.
• Output: a mel, vocoded by a separate HiFi-GAN (HiFi-GAN).

StyleTTS 2 (2023)

phonemes ──► PL-BERT ──► h_bert
   │                       │
   │        noise ─► style diffusion (cond. h_bert)
   │                       │
   │               s = [s_a ; s_p]  (128 + 128)
   │                       │
   ▼                       ▼ s_p
text encoder      duration predictor ─► durations
   │                       │
   │   differentiable upsampling to frames
   │                       │
   │              prosody predictor ─► F0, energy
   ▼                       ▼
decoder (AdaIN with s_a) ───────► waveform
 (HiFi-GAN- or iSTFTNet-style)

• Style split: s_a (acoustic: timbre, recording) drives the decoder; s_p (prosodic) drives duration, pitch and energy.
• Style diffusion: a small Transformer denoiser, trained in the EDM diffusion framework, samples s conditioned on PL-BERT text embeddings. Training targets are the style vectors the style encoders extract from the real audio. For long-form reading, the new sentence's style can be blended with the previous one for consistency.
• PL-BERT: a phoneme-level BERT pretrained on a large text corpus, giving the prosody predictors knowledge of sentence meaning and structure.
• SLM adversarial training: a frozen WavLM (WavLM) with a small trainable head acts as a discriminator on generated speech, alongside HiFi-GAN-style MPD and multi-resolution discriminators. WavLM's features capture high-level properties (naturalness, phonetic content), so the discriminator judges more than local texture.
• Differentiable duration modelling: a differentiable upsampling from predicted durations to frames, so gradients from the waveform-level discriminators reach the duration predictor and the whole system trains end to end.
• End to end to waveform: the decoder outputs audio directly. The paper's code provides both a HiFi-GAN-style and an iSTFTNet-style (iSTFTNet + Vocos) decoder (check the paper for which dataset used which).
• Training is multi-stage: first the acoustic modules (with reference-derived styles), then the predictors, style diffusion and SLM adversarial training jointly.

Kokoro

Kokoro (82M parameters, Apache-2.0 weights) is a popular small TTS model whose model card says it is built on the StyleTTS 2 architecture with an iSTFTNet decoder. It is widely described (including in this guide) as dropping the diffusion sampler and shipping fixed voice packs: precomputed style vectors per voice, so each voice is a stored s rather than a sampled one. Treat the exact list of kept and removed parts as something to confirm on the model card; there is no Kokoro paper.

Build the intuition

Analogy: The text is a script; the style vector is a director's note ("calm, low, slightly fast"). StyleTTS reads the note off a recording of another actor. StyleTTS 2 has an assistant who reads the script and writes a fitting note, different every take. Kokoro keeps a binder of favourite notes, one per voice, and always uses the same one.

Why diffuse a vector and not a mel? Diversity in speech is mostly in prosody and style, which a 256-number vector captures. Sampling it with diffusion costs a few passes of a tiny network; the large decoder then runs once, deterministically. You get diffusion's variety at near-GAN speed.

Voice-pack interpolation: since a voice is just s, mixing 0.5 · s_voiceA + 0.5 · s_voiceB gives a voice in between. That is the Kokoro exercise in the guide.

Numbers worth remembering

• StyleTTS 2 on LJSpeech: CMOS slightly above the ground-truth recordings in the authors' tests; on multi-speaker VCTK, roughly on par with ground truth (check Tables 1-2).
• Style vector: 256 dims = 128 acoustic + 128 prosodic (check the config).
• Zero-shot on LibriTTS: strong naturalness from far less training data than VALL-E's ~60k hours (check the paper's comparison and its similarity numbers).
• Kokoro: 82M parameters, per its model card.

Where it shows up today

• Kokoro: one of the most-used small open TTS models, fast on CPU, a sensible choice for an English voice agent on modest hardware.
• Style vectors with AdaIN are common in voice conversion and expressive TTS; GST (Global Style Tokens) is the ancestor.
• Using a pretrained speech model (WavLM, WavLM) as a judge or feature extractor for training is now common, and speaker similarity metrics use the same models (UTMOS).
• StyleTTS 2 is a frequent baseline against which zero-shot systems (VALL-E, F5-TTS) are compared.

What it got wrong, and what came next

• Zero-shot cloning is weaker than in codec LMs and flow-matching models trained on far more data (VALL-E, CosyVoice → CosyVoice 2, F5-TTS); a 256-dim style vector cannot hold every detail of a voice.
• Relies on phonemizers (espeak) and English PL-BERT; other languages need new G2P and a new PL-BERT, which is why Indian-language support is limited.
• Offline per sentence (whole-sentence durations and style); not built for text streaming like Kyutai TTS (Delayed Streams Modeling (TTS)).
• Complex, multi-stage training with several discriminators; reproducing it is harder than training F5-TTS (F5-TTS).

Check yourself

When you're back at a laptop

• pip install kokoro (and espeak-ng), synthesize a sentence on CPU and measure RTF.
• Load two voice packs (they are tensors), interpolate α·A + (1-α)·B for α = 0, 0.25, 0.5, 0.75, 1, and listen.
• Read the StyleTTS 2 repo's models.py and find the style encoder, the diffusion sampler, the prosody predictor and the decoder; compare with Kokoro's code to see what it omits.