TTS self-check

Raw text is first normalized into speakable words ("₹3.5L" → "three point five lakh rupees"), then converted to phonemes by a G2P (grapheme-to-phoneme) model or kept as characters. A text encoder turns symbols into hidden vectors, an alignment or duration stage decides how many audio frames each symbol lasts, and an acoustic model produces an intermediate representation: a mel spectrogram (Tacotron 2, FastSpeech 2), codec tokens (VALL-E) or continuous latents (CALM (Pocket TTS)). A vocoder or codec decoder (HiFi-GAN, iSTFTNet + Vocos) turns that into a waveform. Normalization is hard because the same string reads differently by context ("1/2" as a date or a fraction, "Dr." as doctor or drive), and neural normalizers make rare but unacceptable errors like reading the wrong number Text normalization challenge. Indian languages add the lakh/crore digit grouping (1,00,000), ₹, mixed scripts, and code-mixing like Hinglish where English words appear in Devanagari or Latin script and must be pronounced the English way. LLM output adds markdown, lists, URLs, emoji and abbreviations that were never meant to be spoken. That is why production systems still combine rules (for numbers, currency, dates) with models, and why you cannot pipe raw LLM output straight into TTS.

WaveNet WaveNet is autoregressive at the sample level: each of 16,000-24,000 samples per second requires a full pass through a deep dilated-convolution stack conditioned on all previous samples, so generation is strictly sequential. Flows make sampling parallel by mapping Gaussian noise to audio in one pass: Parallel WaveNet distils a WaveNet teacher into an inverse-autoregressive-flow student, and WaveGlow trains a flow directly by maximum likelihood Parallel WaveNet + WaveGlow. GAN vocoders (MelGAN, Parallel WaveGAN MelGAN + Parallel WaveGAN, HiFi-GAN HiFi-GAN) use a convolutional generator that upsamples the mel to a waveform in one pass, trained against discriminators, which gives small, fast models. LPCNet LPCNet keeps autoregression but makes each step cheap: classic linear prediction estimates most of each sample, and a tiny RNN predicts only the residual, which runs in real time on a phone CPU. iSTFT-based vocoders (iSTFTNet, Vocos iSTFTNet + Vocos) skip time-domain upsampling and predict STFT magnitude and phase per frame, letting the fast inverse FFT produce samples. Each step trades some modelling generality for orders of magnitude less compute.

Tacotron 2 Tacotron 2 learns alignment through location-sensitive attention, and at inference it decodes autoregressively from its own predictions. On long, unusual or out-of-domain text, the attention can jump ahead (skipping words), get stuck (repeating), or wander (babbling), and the stop token may fire too early or never. Errors compound because each frame depends on previous outputs. FastSpeech FastSpeech and FastSpeech 2 FastSpeech 2 predict an explicit integer duration per phoneme and expand the hidden states with a length regulator, so every phoneme is used exactly once, in order, and all frames are generated in parallel. Glow-TTS Glow-TTS removes the need for an external aligner or teacher by finding the most likely monotonic alignment with Monotonic Alignment Search during training, then training a duration predictor on it. Both make skipping and repetition structurally impossible, and durations give direct speed control.

Let L[i, j] be the log-likelihood of mel frame j under text token i's predicted distribution, and Q[i, j] the best total log-likelihood of an alignment ending with frame j assigned to token i. Then Q[i, j] = L[i, j] + max(Q[i, j-1], Q[i-1, j-1]): frame j either stays on the same token or advances by one Glow-TTS. You fill the table over all i and j, take Q[N, T], and backtrack to get a hard monotonic alignment, which is then used to train the model and the duration predictor. Like the CTC forward algorithm CTC, it is a dynamic program over a (tokens × frames) lattice with monotonic, no-skip-backwards transitions, costing O(N·T). The difference is that MAS takes a max (Viterbi, one best path) while CTC takes a log-sum-exp over all paths, giving a soft marginal. MAS also has no blank token: every frame belongs to some text token, and every token must get at least one frame. And MAS is not differentiated through; it produces a target alignment, while CTC's sum is the loss itself.

The multi-period discriminator HiFi-GAN reshapes the 1D waveform into 2D with periods 2, 3, 5, 7 and 11 (each row is one period), then applies 2D convolutions along the period axis, so each sub-discriminator sees samples that are p apart and can judge periodic structure such as pitch harmonics. The multi-scale discriminator looks at the raw waveform and 2× and 4× average-pooled versions, judging structure and smoothness at different time scales. Together they catch both periodic artifacts and broader spectral or envelope problems. The mel L1 loss compares mel spectrograms of generated and real audio, which gives a stable, dense gradient focused on perceptually relevant frequencies. Adversarial losses alone are noisy early in training and can produce audio that fools the discriminator without matching the input; the mel loss anchors the output to the conditioning and speeds convergence. HiFi-GAN also uses a feature-matching loss on discriminator activations, and the full recipe is reused in VITS VITS, EnCodec EnCodec and other codecs.

HiFi-GAN HiFi-GAN upsamples from the mel frame rate to the sample rate with a stack of transposed convolutions, so its later layers run at thousands to tens of thousands of positions per second with many channels. Vocos iSTFTNet + Vocos keeps all its computation at the frame rate (about 94 frames per second for its 24 kHz model with a hop of 256 samples), using a ConvNeXt backbone, and predicts, per frame, the STFT magnitude and phase (as a log-magnitude and a phase angle via cos/sin). A single inverse STFT, a fast FFT-based operation with no learned parameters, then produces the waveform. Because the network never runs at sample rate, it uses far fewer operations per second of audio and is several times faster than HiFi-GAN at similar quality, especially on CPU. The trick is that the iSTFT does the expensive "upsampling" analytically, which is also why iSTFTNet replaced only HiFi-GAN's last upsampling layers and why Kokoro uses it StyleTTS → StyleTTS 2.

VITS VITS is a conditional VAE: text → text encoder → prior distribution per phoneme; linear spectrogram → posterior encoder → latent z; z → HiFi-GAN-style decoder → waveform. The posterior encoder (training only) encodes the real linear spectrogram into a latent z, sampled with the reparameterization trick. A normalizing flow maps z into a space where it is compared with the text-conditioned prior, making the prior far more expressive than a plain Gaussian, and it is run in reverse at inference to turn prior samples into z. MAS Glow-TTS aligns text tokens to latent frames during training by maximizing likelihood under the prior, so no external aligner is needed. The stochastic duration predictor is a flow-based model of durations that can sample different rhythms for the same text, giving natural variation. The decoder is a HiFi-GAN generator turning z straight into waveform, trained on random latent segments to save memory. HiFi-GAN's multi-period discriminator, plus feature matching and mel losses, make the output sharp; together this yields one end-to-end model, used today by Piper and MMS-TTS.

StyleTTS StyleTTS → StyleTTS 2 conditions its decoder and prosody predictors on a style vector, a fixed-size embedding of speaking style taken from a reference clip and applied through adaptive instance normalization. StyleTTS 2 makes the style a latent variable sampled by a small diffusion model conditioned on the text, so you get natural, varied prosody that fits the sentence without needing a reference recording. It uses a large pretrained speech language model (WavLM WavLM) as the backbone of a discriminator, so the generator is pushed to produce speech whose deep SSL features match real speech, a much richer signal than waveform discriminators alone. It also adds differentiable duration modelling so the whole system trains end to end. The result matched or beat human recordings on LJSpeech in their MOS tests. Kokoro (82M parameters) is built on the StyleTTS 2 architecture with an iSTFTNet decoder iSTFTNet + Vocos; it is widely described as dropping the diffusion style sampler and using fixed precomputed voice-style vectors (voice packs), which is why voices can be blended by interpolating vectors. Check the Kokoro model card for the exact components.

Residual vector quantization SoundStream quantizes an encoder vector with codebook 1 (nearest entry), subtracts that entry to get a residual, quantizes the residual with codebook 2, and repeats for N codebooks; the decoded vector is the sum of the chosen entries. Each extra codebook refines what the previous ones missed, so the first codebook carries coarse information and later ones add detail. Quantizer dropout trains with a random number of codebooks per batch, so one model works at many bitrates: decode with 2 codebooks for low bitrate or 8 for high quality. A codec LM must generate every token, so tokens per second = frame rate × codebooks used. EnCodec at 75 Hz with 8 codebooks EnCodec is 600 tokens per second, which makes autoregressive generation costly and is why VALL-E splits generation into an AR model for codebook 1 and a NAR model for the rest VALL-E. Mimi at 12.5 Hz with 8 codebooks is 100 tokens per second Moshi, and single-codebook codecs (Llasa + Spark-TTS) or continuous latents (CALM (Pocket TTS)) cut the budget further. Bits per second = frame rate × codebooks × log2(codebook size).

Semantic tokens are discrete units that mainly capture what is said, the phonetic and linguistic content, typically made by k-means clustering of self-supervised features (w2v-BERT, HuBERT HuBERT) or by quantizing an ASR encoder. Acoustic tokens are codec codes (RVQ from SoundStream, EnCodec) that capture everything needed to reconstruct the waveform: voice timbre, recording conditions and fine detail. AudioLM AudioLM showed that modelling acoustic tokens alone gives good audio but poor long-term structure (babble), while semantic tokens give coherent content, so it generates semantic first and acoustic second. SPEAR-TTS SPEAR-TTS exploits the split for data efficiency: text → semantic needs transcribed data, but semantic → acoustic can be trained on untranscribed audio. CosyVoice CosyVoice → CosyVoice 2 uses supervised semantic tokens from an ASR encoder for its LLM stage and flow matching for acoustics. Mimi Moshi and SpeechTokenizer SpeechTokenizer + SNAC put semantic information into the first RVQ codebook by distilling from WavLM or HuBERT, so one codec provides both. Separating them lets each stage solve an easier problem: content and prosody planning vs faithful rendering.

VALL-E VALL-E treats TTS as language modelling over EnCodec tokens: the prompt is the phonemes of the prompt transcript plus the target text, followed by the EnCodec tokens of a 3-second recording, and the model continues the token sequence. Because it was trained on ~60k hours of many speakers, continuing a sequence that starts with someone's voice naturally keeps that voice, emotion and even the room acoustics. The autoregressive Transformer generates codebook 1 frame by frame, which carries the content, timing and prosody where sequential dependencies matter. A non-autoregressive Transformer then predicts codebooks 2-8, one codebook at a time but all frames in parallel, conditioned on the earlier codebooks and the prompt; doing these autoregressively would multiply decoding steps by 8. It skips or repeats words because nothing enforces a monotonic text-to-audio alignment: the AR model's implicit attention can lose its place, just as Tacotron's did Tacotron 2, and sampling randomness makes it worse. Later work adds alignment constraints, more robust tokens, or non-autoregressive flow matching (CosyVoice → CosyVoice 2, F5-TTS).

Sample data x_1, noise x_0 ~ N(0, I) and a time t ~ U(0, 1), form x_t = (1 − t)·x_0 + t·x_1, and train L = E || v_θ(x_t, t, c) − (x_1 − x_0) ||², where v_θ is the network's predicted velocity and c the conditioning (text, prompt) Flow Matching. The target velocity x_1 − x_0 is constant along the straight path, so the loss is plain regression with no noise schedule, no score function, and no variance terms. Diffusion needs a forward noising schedule, a choice of what to predict (noise, data or score) and weighting, and its sampling paths are curved, so it needs more steps. At inference you integrate dx/dt = v_θ(x, t, c) from noise at t=0 to data at t=1 with an ODE solver such as Euler. More steps (NFE, number of function evaluations) follow the learned field more accurately and improve quality, but cost proportionally more compute and latency; straight paths mean 10-32 steps are often enough for speech (Matcha-TTS, F5-TTS). One-step methods like consistency models and drifting Drifting Models push this trade-off to its limit.

E2 TTS E2 TTS drops phonemes, durations and aligners: it pads the character sequence with filler tokens to the same length as the mel spectrogram and feeds both to a flow-matching model trained Voicebox-style Voicebox to infill masked audio. The model learns the alignment between characters and frames implicitly inside its attention, because it must reconstruct the speech given the whole padded text. F5-TTS F5-TTS keeps this but uses a DiT backbone and a ConvNeXt text encoder that refines the padded characters first, which makes alignment learning faster and more robust, plus sway sampling at inference. What you still choose is the total duration of the output, because the length of the padded sequence fixes how many frames are generated. In practice it is estimated from the prompt's speaking rate, for example the ratio of prompt audio length to prompt text length applied to the target text. A bad estimate makes speech too fast or too slow, and you also choose NFE and the classifier-free guidance strength.

CosyVoice 2 CosyVoice → CosyVoice 2 is a two-stage system: an autoregressive text-to-token LLM followed by a flow-matching token-to-mel model. Its streaming comes from chunk-aware causal flow matching, which can turn speech tokens into mel in chunks with limited look-ahead, and its LLM can be fed text and speech tokens interleaved in a fixed ratio, so audio flows out in chunks once enough tokens exist. The flow-matching stage still needs a chunk of tokens plus its look-ahead before audio comes out, which adds some latency. Kyutai TTS Delayed Streams Modeling (TTS) uses delayed streams modelling Kyutai DSM: text and audio are parallel streams on one time axis, with audio delayed a fixed amount behind text, and a single decoder-only model emits Mimi tokens step by step as text tokens arrive. Kyutai's design is built to consume text word by word as the LLM produces it, start speaking after a short fixed text look-ahead, and continue for arbitrarily long input. CosyVoice 2's interleaved mode can also accept streaming text, but its typical deployments chunk at the sentence or phrase level, so Kyutai's model is the more natural fit for starting before the LLM has finished the sentence. Either way, a sentence-level offline TTS adds a full sentence of delay.

Discrete codec LMs quantize audio, which loses information, so better quality requires more codebooks and more tokens per frame, raising compute (VALL-E, Llasa + Spark-TTS). CALM CALM (Pocket TTS) removes quantization: a Transformer backbone produces a contextual embedding per step, and a small MLP head generates the next continuous frame of an audio VAE latent with consistency modelling, a few-step generator. Per the abstract it gets better quality at lower compute than discrete audio LMs. Multi-step flow matching or diffusion would need many evaluations of the head per frame. Drifting Drifting Models instead trains the generator so its output distribution is moved toward the data during training, yielding a one-step generator at inference, and Kyutai retrained Pocket TTS's head with it. On a CPU, every network call per frame counts: the backbone runs once per frame, and if the head needs one step instead of several, the per-frame cost drops sharply. That is what lets a ~100M-parameter model like Pocket TTS run faster than real time on a laptop CPU while streaming.

Build a test set from real agent scripts: Hindi sentences, Hinglish with English words in both Latin and Devanagari script, Indian English, plus numbers, ₹ amounts in lakh/crore format, dates, account numbers and names, since text normalization errors are the most common failures Text normalization challenge. Measure intelligibility with WER and CER using a strong Indian-language ASR model, such as IndicWhisper or IndicConformer Vistaar, and Whisper-large-v3 for Indian English, normalizing text consistently; CER is fairer for Hindi spelling variants. Measure speaker similarity with a WavLM speaker-verification model if you clone voices, UTMOS as a rough naturalness proxy while remembering it is not validated on Indian languages UTMOS, and RTF and TTFA on your actual serving hardware under realistic concurrency. For human evaluation, use native Hindi and Indian-English speakers: MOS for naturalness, CMOS or MUSHRA Rasa against your current system, and targeted checks on whether numbers and names were read correctly and whether the tone fits a support conversation. Include an end-to-end test in the agent, measuring perceived latency and barge-in behaviour. Benchmarks like IndicVoices-R IndicVoices-R and the IndicTTS test sets Indic TTS (AI4Bharat) help for comparability, but your in-domain set should drive the decision.

Kokoro StyleTTS → StyleTTS 2 and Piper (VITS VITS) are small, fast, run on CPU, and have fixed voices; choose them for low cost, on-device or edge deployment, and when a few stock voices are enough, synthesizing sentence by sentence. Their limits are no zero-shot cloning and limited language coverage (check each model's Indian-language support). F5-TTS F5-TTS (or IndicF5 for Indian languages) gives high-quality zero-shot cloning from a short prompt, but it is non-autoregressive over the whole utterance, needs a GPU for good speed and is not streaming, so it suits offline content, voice design or sentence-level synthesis where TTFA is less critical. CosyVoice 2 CosyVoice → CosyVoice 2 combines cloning, multilingual support and chunk-aware streaming, a good middle ground on a GPU server when you need custom voices and moderate latency. Kyutai TTS Delayed Streams Modeling (TTS) and Pocket TTS CALM (Pocket TTS) are designed for streaming text in from an LLM and starting to speak within a short fixed delay; choose them when TTFA and barge-in responsiveness dominate the user experience, with Pocket TTS especially when you must run on CPU. In practice you weigh language support (Hindi and Hinglish quality first), TTFA, cost per hour of audio, licence, and whether you need cloning.