Both tracks

Glossary

112 terms you meet across the ASR and TTS papers.

Acoustic model: The part of a TTS system that turns text representations into an intermediate audio representation such as a mel spectrogram, codec tokens or latents. In ASR the term traditionally means the model from audio features to phone or character probabilities. See FastSpeech 2.
Acoustic token: A discrete code from a neural audio codec (for example an RVQ code from SoundStream or EnCodec) that carries the detail needed to reconstruct the waveform: voice timbre, recording conditions and fine structure. Contrast with semantic token. See AudioLM.
AED (attention encoder-decoder): An ASR or TTS architecture where an encoder processes the input and an autoregressive decoder generates outputs one at a time while attending over all encoder states. Whisper, LAS and Canary are AEDs. See Listen, Attend and Spell.
Alignment: The mapping between text tokens and audio frames: which frames belong to which token. Both ASR and TTS must learn it because training data does not provide it. See CTC.
Arena evaluation: Human evaluation where listeners hear two systems' outputs for the same input and pick the better one; votes are aggregated into a ranking with Elo or Bradley-Terry scores. TTS Arena and Artificial Analysis use it. See UTMOS.
Autoregressive (AR): Generating a sequence one element at a time, each conditioned on the previously generated ones. Accurate and flexible, but sequential, so slow for long outputs. See VALL-E.
Barge-in: The user starting to speak while the voice agent is still talking; the agent must detect it quickly, stop its audio and listen. Needs fast VAD or a full-duplex model. See Moshi.
Beam search: A decoding method that keeps the k most probable partial hypotheses at each step instead of only the best one. Improves accuracy over greedy decoding at extra compute cost, and allows adding an external language model. See CTC.
BEST-RQ: A self-supervised speech method where the targets for masked frames come from a frozen random projection and a random codebook, with nothing learned or clustered. Used to pretrain Google USM. See data2vec / w2v-BERT / BEST-RQ.
Blank token: A special CTC and transducer output meaning "no new token at this frame". In CTC it also separates repeated letters so "ll" can be produced. See CTC.
Cache (streaming): Stored activations from earlier audio chunks, such as attention keys and values and convolution states, reused when processing the next chunk so nothing is recomputed. See Cache-aware streaming Conformer.
CER (character error rate): Like WER but computed over characters: edits needed to turn the hypothesis into the reference, divided by the reference length. Fairer than WER for agglutinative languages and scripts with spelling variation. See Vistaar.
Chunk size: How much new audio a streaming model processes per step, for example 160 ms. Larger chunks usually lower WER but add latency. See Cache-aware streaming Conformer.
Classifier-free guidance (CFG): At inference, run the model with and without its conditioning and push the output further in the direction the conditioning suggests. A guidance strength above 1 trades diversity for adherence to text or prompt. See F5-TTS.
CMOS (comparative MOS): A listening test where raters hear two systems' outputs and score one relative to the other, typically from -3 to +3. More sensitive than MOS for small differences. See UTMOS.
Code-mixing: Mixing two languages within one utterance, such as Hindi and English (Hinglish), sometimes also mixing scripts. Hard for both ASR and TTS front-ends. See Svarah.
Codebook: A learned table of vectors used by a vector quantizer; each input vector is replaced by the index of its nearest entry. Codebook size sets the bits per code. See SoundStream.
Conditional independence assumption: CTC's assumption that the output at each frame is independent of outputs at other frames given the audio. It makes CTC fast but means it has no built-in language model. See CTC.
Conformer: An ASR encoder block combining self-attention for global context and depthwise convolution for local patterns, sandwiched between two half-step feed-forward layers. The basis of most modern ASR encoders. See Conformer.
Consistency model: A generative model trained so that any point on a noise-to-data path maps directly to the same clean output, allowing generation in one or a few steps instead of many. Used in CALM's generation head. See CALM (Pocket TTS).
Contrastive loss: A loss that pushes a prediction to be similar to the correct target and dissimilar to distractors. wav2vec 2.0 uses it to identify the true quantized latent of masked frames. See wav2vec 2.0.
CTC (Connectionist Temporal Classification): A loss that trains a frame-level classifier without alignments by summing the probability of all frame paths that collapse to the transcript. Fast, non-autoregressive decoding. See CTC.
Delay pattern: A way to predict several codebooks with one decoder by shifting codebook k by k time steps, so finer codebooks are predicted after coarser ones for the same frame. Used by MusicGen and Parler-TTS. See Parler-TTS.
Delayed streams modelling (DSM): Kyutai's approach of modelling text and audio as parallel time-aligned streams with one stream delayed behind the other. Text delayed gives streaming ASR; audio delayed gives streaming TTS. See Kyutai DSM.
Depthwise separable convolution: A convolution split into a depthwise step (one filter per channel, over time) and a pointwise 1x1 step (mixing channels). Far fewer parameters and operations than a full convolution. See Jasper → QuartzNet → Citrinet.
Diarization: Labelling who spoke when in a recording, splitting audio by speaker. Used in transcription and in data pipelines that need single-speaker clips. See LibriTTS → Emilia.
Diffusion model: A generative model trained to reverse a gradual noising process; at inference it starts from noise and denoises over many steps. Used in WaveGrad, Grad-TTS and NaturalSpeech 2. See WaveGrad + DiffWave.
Dilated convolution: A convolution whose filter taps are spaced apart by a dilation factor, so stacking layers with doubling dilation grows the receptive field exponentially. WaveNet's core building block. See WaveNet.
Distillation: Training a smaller or faster student model to reproduce the outputs of a larger teacher. Distil-Whisper and Parallel WaveNet are examples. See Distil-Whisper.
Drifting: A generative training method that moves the generator's output distribution toward the data during training, giving a one-step generator at inference. Kyutai used it for Pocket TTS's head. See Drifting Models.
Duration predictor: A TTS module that predicts how many audio frames each phoneme or character should last. Enables parallel generation and speed control. See FastSpeech 2.
Encoder: The network that turns input features (audio frames or text) into a sequence of hidden representations used by a decoder or loss. See Conformer.
Endpointing: Deciding that the user has finished their turn, so the system can respond. Too eager interrupts people; too slow adds latency. See Streaming E2E ASR for mobile.
F0 (fundamental frequency): The rate at which the vocal folds vibrate, perceived as pitch. TTS models such as FastSpeech 2 predict it to control intonation. See FastSpeech 2.
Feature matching loss: A GAN loss that makes the generator's output produce discriminator intermediate features similar to those of real audio. Stabilizes GAN vocoder training. See MelGAN + Parallel WaveGAN.
Flow matching: A generative method that trains a network to predict the velocity carrying noise to data along simple paths, using plain regression. Sampling integrates an ODE in a few steps. See Flow Matching.
Forced alignment: Finding the time boundaries of each known word or phoneme in a recording, given the transcript. Used for word timestamps (WhisperX) and for TTS duration targets (Montreal Forced Aligner). See WhisperX.
Forward-backward algorithm: A dynamic program that sums probabilities over all alignment paths in a lattice efficiently, in time proportional to tokens × frames. Used to compute the CTC and RNN-T losses. See CTC.
Frame: One time step of an audio feature sequence, such as one mel spectrogram column every 10 ms or one codec step every 80 ms. See Conformer.
Frame rate: How many frames per second a representation has: log-mel features are typically 100 Hz, wav2vec 2.0 outputs 50 Hz, Fast Conformer 12.5 Hz, EnCodec 75 Hz, Mimi 12.5 Hz. Sets sequence length and compute. See Fast Conformer.
FSQ (finite scalar quantization): A quantizer that rounds each dimension of a low-dimensional vector to a few fixed levels, giving an implicit codebook with no learned entries and no unused codes. Used in CosyVoice 2. See CosyVoice → CosyVoice 2.
Full duplex: A dialogue system that listens and speaks at the same time, modelling both the user's and its own audio continuously. Moshi is full duplex. See Moshi.
G2P (grapheme-to-phoneme): Converting written text into phonemes, using a pronunciation dictionary, rules or a model. Essential for English; Indic scripts are closer to phonetic, so characters often suffice. See Indic TTS (AI4Bharat).
GAN (generative adversarial network): A generator trained against a discriminator that tries to tell real from generated samples. The basis of fast vocoders such as HiFi-GAN. See HiFi-GAN.
Greedy decoding: Taking the most likely output at each step without search. For CTC, take the argmax per frame then collapse repeats and blanks. See CTC.
Griffin-Lim: A classic iterative algorithm that estimates phase from a magnitude spectrogram to recover a waveform. Fast and training-free, but sounds metallic. See Tacotron.
Hallucination: A model producing fluent output not supported by the input, such as Whisper writing sentences during silence or a codec LM adding words. Typical of autoregressive decoders with strong language priors. See Whisper.
Hop length: The step between successive STFT or mel frames, in samples; for example 160 samples at 16 kHz is 10 ms. Sets the frame rate. See iSTFTNet + Vocos.
Inner monologue: Moshi's technique of predicting time-aligned text tokens of its own speech before the corresponding audio tokens, improving linguistic quality. See Moshi.
iSTFT (inverse short-time Fourier transform): Reconstructs a waveform from per-frame magnitude and phase spectra by inverse FFT and overlap-add. Vocos and iSTFTNet use it as a fast, parameter-free output layer. See iSTFTNet + Vocos.
k-means units: Discrete speech units made by clustering self-supervised features (for example HuBERT layers) with k-means. HuBERT trains on them and many speech generation models use them as semantic tokens. See HuBERT.
Language model fusion: Combining an ASR model's scores with an external text language model during decoding (shallow fusion), to fix spelling and word choice. Most useful for CTC. See IndicWav2Vec.
Latency: The delay between an input event and the system's response, such as from end of speech to transcript, or from text to first audio. The key constraint for voice agents. See Cache-aware streaming Conformer.
Length regulator: The FastSpeech component that repeats each phoneme's hidden vector by its predicted duration, expanding the text sequence to the length of the audio frames. See FastSpeech.
Log-mel spectrogram: A time-frequency representation made by taking the STFT, pooling the power into mel-spaced frequency bands (usually 80 or 128) and taking the log. The standard input to ASR models and the standard output of mel-based TTS. See Whisper.
Look-ahead: How much future audio (or text) a streaming model may see before producing output for the current position. More look-ahead improves accuracy but adds latency. See Cache-aware streaming Conformer.
Macaron FFN: The Conformer's split of one feed-forward layer into two half-weighted feed-forward layers placed before and after the attention and convolution modules. See Conformer.
MAS (Monotonic Alignment Search): A Viterbi-style dynamic program that finds the most likely monotonic text-to-frame alignment under the current model, used to train TTS without an external aligner. See Glow-TTS.
Masked prediction: Self-supervised training where parts of the input are hidden and the model must predict targets for the hidden parts from context, as in BERT. Used by wav2vec 2.0, HuBERT and BEST-RQ. See HuBERT.
Mel scale: A frequency scale that is roughly linear below about 1 kHz and logarithmic above, approximating human pitch perception. Mel filterbanks pool spectra onto it. See SpecAugment.
MFCC (mel-frequency cepstral coefficients): Compact features from a discrete cosine transform of the log-mel spectrum. Used in classic ASR, and as HuBERT's first clustering features. See HuBERT.
Mimi: Kyutai's neural audio codec at 12.5 Hz with residual vector quantization, whose first codebook is distilled from WavLM to carry semantic content. Used in Moshi, Kyutai STT/TTS and Sesame CSM. See Moshi.
MOS (mean opinion score): The average of listener ratings of quality or naturalness on a 1 to 5 scale. The gold standard for TTS, but costly and not comparable across tests. See UTMOS.
Mu-law (μ-law): A companding transform that compresses audio amplitudes logarithmically, letting 8 bits (256 levels) represent speech well. WaveNet predicts a softmax over μ-law values. See WaveNet.
Multi-period discriminator (MPD): HiFi-GAN's discriminator that reshapes the waveform into 2D with several prime periods to judge periodic structure such as pitch harmonics. See HiFi-GAN.
Multi-resolution STFT loss: A loss comparing STFT magnitudes of generated and real audio at several window sizes, giving stable gradients for vocoders and codecs. See MelGAN + Parallel WaveGAN.
MUSHRA: A listening test where raters score several systems for the same input side by side on a 0-100 scale, with a hidden reference and a low-quality anchor included to check raters. See Rasa.
Neural audio codec: An encoder-quantizer-decoder network that compresses audio into a low-rate sequence of discrete tokens (or latents) and reconstructs it. Lets language models generate audio. See EnCodec.
NFE (number of function evaluations): How many network calls a diffusion or flow model uses to generate one sample. Fewer means faster but usually lower quality. See F5-TTS.
Non-autoregressive (NAR): Generating all output positions in parallel rather than one after another. Fast, but needs another way to fix output length and handle dependencies. See FastSpeech.
Normalizing flow: An invertible network that maps a simple distribution to data, with exact likelihood via the change-of-variables formula. Used in WaveGlow, Glow-TTS and VITS. See Parallel WaveNet + WaveGlow.
Phoneme: The smallest unit of sound that distinguishes words in a language, such as /p/ vs /b/. TTS often uses phonemes as input; some ASR predicts them. See FastSpeech 2.
Prediction network: The RNN-T component that reads previously emitted tokens and acts like a small language model, removing CTC's independence assumption. See RNN-T.
Prosody: The rhythm, stress, intonation and timing of speech, carried by pitch, duration and energy. What makes speech sound natural or robotic. See StyleTTS → StyleTTS 2.
Pseudo-labelling: Transcribing unlabelled audio with an existing model and training on those transcripts as if they were labels, usually after filtering low-quality ones. See Distil-Whisper.
Quantizer dropout: Training a residual vector quantizer with a random number of codebooks per batch, so one codec works at several bitrates. See SoundStream.
Real-time factor (RTF): Processing time divided by audio duration. RTF 0.1 means 10 s of audio is processed in 1 s; below 1 is faster than real time. See UTMOS.
Receptive field: How much of the input a single output position can depend on. Dilated and stacked convolutions grow it; attention makes it the whole sequence. See WaveNet.
Relative positional encoding: Position information based on the distance between two positions rather than their absolute index, so attention generalizes to different sequence lengths. Used in the Conformer. See Conformer.
Residual vector quantization (RVQ): Quantizing a vector with a first codebook, then quantizing the leftover error with a second, and so on; the code is the sum of chosen entries. Each extra codebook adds detail. See SoundStream.
RNN-T (recurrent neural network transducer): An ASR model with an encoder, a prediction network over past tokens and a joint network, trained by summing over all alignments in a frames × tokens lattice. The standard streaming ASR design. See RNN-T.
RQ-Transformer: A two-level Transformer: a large temporal model runs once per audio frame and a small depth model predicts that frame's codebooks one after another. Used in Moshi. See Moshi.
Sample rate: The number of audio samples per second, such as 16 kHz for most ASR and 22.05-48 kHz for TTS. Sets the highest representable frequency (half the sample rate). See LibriTTS → Emilia.
Self-supervised learning (SSL): Pretraining on unlabelled data with targets derived from the data itself, such as predicting masked frames. wav2vec 2.0, HuBERT and WavLM are SSL speech models. See wav2vec 2.0.
Semantic token: A discrete unit capturing mainly what is said rather than how it sounds, usually from clustered SSL features or a quantized ASR encoder. Contrast with acoustic token. See AudioLM.
SIM (speaker similarity): Cosine similarity between speaker embeddings of a reference clip and generated speech, usually from a WavLM-based verification model. The standard automatic metric for voice cloning. See UTMOS.
SMOS (similarity MOS): A listening test where raters score from 1 to 5 how similar a generated voice is to a reference speaker. See UTMOS.
Snake activation: A periodic activation, x + sin²(αx)/α, that helps networks model periodic signals like audio. Used in BigVGAN and DAC. See BigVGAN.
Speaker embedding: A fixed-length vector summarizing a speaker's voice, produced by a speaker-verification model. Used to condition multi-speaker TTS and to measure similarity. See SV2TTS.
Speaker verification: Deciding whether two recordings come from the same speaker, typically by comparing their embeddings. Its models are reused for TTS conditioning and evaluation. See WavLM.
SpecAugment: Data augmentation applied directly to spectrograms: masking bands of frequencies, masking blocks of time and warping time. Standard in ASR training. See SpecAugment.
Speech restoration: Repairing degraded recordings, removing noise and reverberation and restoring missing high frequencies, often with a generative model. Used to turn ASR data into TTS data. See IndicVoices-R.
Squeeze-and-excitation: A block that averages a feature map over time, passes it through a small network and rescales each channel, injecting global context into convolutions. Used in ContextNet and Citrinet. See ContextNet.
STFT (short-time Fourier transform): Splitting audio into short overlapping windows and taking the Fourier transform of each, giving magnitude and phase over time and frequency. The basis of spectrograms. See iSTFTNet + Vocos.
Stop token: A per-frame prediction in autoregressive TTS such as Tacotron 2 that signals when to stop generating. If it misfires, output is cut off or never ends. See Tacotron 2.
Straight-through estimator: A trick for training through non-differentiable quantization: use the quantized value forward, but pass the gradient back as if quantization were the identity. See SoundStream.
Streaming: Producing output while input is still arriving, with bounded delay, rather than waiting for the whole utterance or sentence. See Cache-aware streaming Conformer.
Style vector: An embedding capturing speaking style (prosody, emotion, voice character), used to condition a TTS decoder. Introduced via style tokens and central in StyleTTS and Kokoro. See StyleTTS → StyleTTS 2.
Subsampling: Reducing the frame rate at the encoder input with strided convolutions, for example 4x (10 ms to 40 ms frames) or 8x (to 80 ms). Cuts compute, especially attention's quadratic cost. See Fast Conformer.
Subword tokens: Text units between characters and words, learned with BPE or SentencePiece, so common words are one token and rare words are split. Most ASR models output them. See Jasper → QuartzNet → Citrinet.
TDT (Token-and-Duration Transducer): A transducer that predicts a token and how many frames to skip, so decoding jumps over frames. Faster than RNN-T with equal or better accuracy. See TDT.
Teacher forcing: Training an autoregressive model by feeding it the ground-truth previous outputs rather than its own predictions. Fast and stable, but creates a mismatch with inference. See Tacotron 2.
Text normalization: Converting written text into speakable words: numbers, currency, dates, abbreviations and symbols. A major source of TTS errors, especially for Indian formats and LLM output. See Text normalization challenge.
TTFA (time to first audio): Time from when text is sent to a TTS system until the first audio is playable. The latency a voice-agent user feels, distinct from RTF. See Delayed Streams Modeling (TTS).
UTMOS: A learned predictor of MOS built on fine-tuned self-supervised speech models, from the VoiceMOS Challenge 2022. The most common automatic naturalness score in TTS papers. See UTMOS.
VAD (voice activity detection): Detecting which parts of audio contain speech. Used to segment audio, gate ASR, detect barge-in and avoid hallucinations on silence. See WhisperX.
VAE (variational autoencoder): An encoder-decoder trained so that latents follow a simple prior distribution, allowing sampling. VITS is a conditional VAE; CALM generates VAE latents. See VITS.
Vector quantization (VQ): Replacing a continuous vector with the nearest entry of a codebook, turning continuous features into discrete tokens. See SoundStream.
Vocoder: The model that turns an intermediate representation, usually a mel spectrogram, into a waveform. Examples are WaveNet, HiFi-GAN and Vocos. See HiFi-GAN.
Weak supervision: Training on large amounts of noisy, imperfect labels collected for other purposes, such as internet transcripts. Whisper's training approach. See Whisper.
WER (word error rate): Substitutions plus deletions plus insertions needed to turn the hypothesis into the reference, divided by the number of reference words. The main ASR metric and TTS's intelligibility metric. See Whisper.
Zero-shot voice cloning: Synthesizing speech in the voice of a speaker never seen in training, from a short audio prompt. See VALL-E.

Glossary

A

B

C

D

E

F

G

H

I

K

L

M

N

P

Q

R

S

T

U

V

W

Z