WaveNet
WaveNet: A Generative Model for Raw Audio
In one sentence
WaveNet generates raw audio one sample at a time with a deep stack of dilated causal convolutions that outputs a 256-way softmax over μ-law-quantized amplitudes, and it was the first neural model whose speech sounded close to a human recording.
Why this paper exists
Before 2016 there were two ways to build TTS:
- Concatenative: record one speaker for many hours, cut the audio into small units, and glue units together at synthesis time. Natural in the best case, but audible joins, no flexibility (a new voice or style means a new recording), and huge databases.
- Statistical parametric: an HMM or LSTM predicts vocoder parameters (spectral envelope, F0, aperiodicity), and a hand-designed signal-processing vocoder turns them into a waveform. Flexible and small, but it sounds buzzy and muffled, because the DSP vocoder and the averaging in the model wash out detail.
Nobody had modelled the waveform itself with a neural network, because audio is extremely long: 16,000 samples per second, with structure from sub-millisecond (the shape of a glottal pulse) up to hundreds of milliseconds (a syllable). WaveNet showed that a convolutional autoregressive model could do it.
The core idea
Model the waveform x = (x_1, ..., x_T) as a product of conditionals, exactly like a language model over characters:
p(x) = Π_{t=1..T} p(x_t | x_1, ..., x_{t-1})
Make each conditional a classification over 256 amplitude levels (after μ-law companding) instead of a regression, so the network can express multi-modal uncertainty. Compute the conditionals with dilated causal convolutions, which see far into the past with few layers, and train all time steps in parallel because the targets are known.
How it works
x_1..x_{t-1} (μ-law, one-hot or embedded)
|
causal conv
|
+--> residual block, dilation 1 ---+--> skip
| residual block, dilation 2 ---+--> skip
| ... |
| residual block, dilation 512 --+--> skip
| (stack repeated a few times) |
| v
| sum of skips -> ReLU -> 1x1
| -> ReLU -> 1x1
| -> softmax(256)
+-- conditioning h (speaker id) or
y (linguistic features, upsampled)1. μ-law quantization. Raw 16-bit audio has 65,536 levels; a softmax over that is wasteful. μ-law compresses the amplitude logarithmically, then quantizes to 256 levels:
f(x) = sign(x) · ln(1 + μ|x|) / ln(1 + μ), with x in [-1, 1] and μ = 255.
Small amplitudes get fine resolution, loud ones coarse resolution, which matches hearing. Example: x = 0.5 gives ln(128.5)/ln(256) ≈ 0.876, bin ≈ 239; x = 0.01 gives ≈ 0.229, bin ≈ 157. A quiet sample of 0.01 still lands far from silence (bin ~128), so quiet detail survives.
2. Causal convolutions. The output at time t may only depend on inputs at ≤ t. In practice you shift the conv so it only looks left. During training every position is computed in one pass (teacher forcing); at generation time you must feed each new sample back in.
3. Dilated convolutions. A dilation d conv with kernel size 2 combines x_t and x_{t-d}. Stacking dilations 1, 2, 4, ..., 512 makes the receptive field double every layer.
4. Gated activation units (borrowed from PixelCNN):
z = tanh(W_f * x) ⊙ σ(W_g * x)
* is convolution, ⊙ is element-wise product, W_f is the "filter" weights, W_g the "gate" weights, σ the sigmoid. The gate decides how much of each channel passes.
5. Residual and skip connections. Each block adds its output back to its input (residual, for depth) and also sends a copy to a skip path; the skips from all layers are summed and passed through two ReLU + 1x1 conv layers and a softmax.
6. Conditioning.
- Global (a speaker ID embedding
h):z = tanh(W_f * x + V_f^T h) ⊙ σ(W_g * x + V_g^T h). - Local (linguistic features
y, at a much lower rate): upsampleyto the audio rate with transposed convolutions, then addV_f * yandV_g * yinside the gates.
For TTS, y came from an existing parametric front-end: linguistic features (phonemes, stress, position in syllable/word), durations and log F0 predicted by separate models. WaveNet replaced only the final "features to waveform" step. Later, Tacotron 2 (Tacotron 2) conditioned it on mel spectrograms instead, which is what made it a vocoder in today's sense.
Build the intuition
Receptive field by hand
Receptive field (RF) is how many past samples can influence one output. For a stack of causal convs with kernel size k and dilations d_1, ..., d_L:
RF = 1 + Σ_l d_l · (k - 1)
Small example, k = 2, dilations 1, 2, 4:
- After dilation 1: each output sees
x_t, x_{t-1}→ RF 2. - After dilation 2: combines two of those, at
tandt-2→ coverst-3..t→ RF 4. - After dilation 4: combines outputs at
tandt-4→ coverst-7..t→ RF 8.
Formula: 1 + (1 + 2 + 4) · 1 = 8. Three layers, eight samples. Without dilation, three layers of k = 2 see only 4 samples.
One full block, dilations 1, 2, ..., 512 (10 layers): 1 + 1023 = 1024 samples = 64 ms at 16 kHz.
Three such blocks: 1 + 3 · 1023 = 3070 samples ≈ 192 ms. That covers a syllable. Compare an undilated stack: to reach 3,070 samples with k = 2 you would need 3,069 layers.
(The initial causal conv adds a little more; the paper's exact stacks vary by experiment, so check the paper for each setting.)
Why a softmax and not a regression
If two continuations are equally likely (the waveform could go up or down), an L2 regression predicts the average, which is a flat, muffled signal. A softmax can put mass on both and sample one. This is the same reason later vocoders need GANs, flows or diffusion: regression to the mean destroys audio detail.
Why it is slow
At 16 kHz, one second of audio needs 16,000 sequential forward passes through a ~30-layer network, each depending on the previous sample. On a GPU each step is tiny and latency-bound, so the hardware sits mostly idle. Even with caching of past activations (the "Fast WaveNet" trick), early implementations ran far slower than real time. On your RTX 3050 a naive PyTorch WaveNet would take minutes per second of audio.
Analogy: WaveNet is a very good typist who can only press one key at a time and must look at everything already typed before each press. Training is fast because you can grade every keystroke of a finished document at once; writing a new document is slow because you cannot skip ahead.
Numbers worth remembering
- 16 kHz audio, 8-bit μ-law (256 classes,
μ = 255). - Dilations double from 1 to 512 and the pattern repeats; one block of kernel-2 layers has a 1,024-sample (64 ms) receptive field.
- US English MOS (5-point naturalness): WaveNet 4.21 vs concatenative 3.86 vs LSTM parametric 3.67, natural speech ~4.5. It closed over half the gap between the best previous system and human speech.
- Single-speaker TTS data: ~25 hours of US English and ~35 hours of Mandarin (internal Google data; check the paper for exact figures).
- Multi-speaker model on VCTK (109 speakers) with a speaker-ID embedding.
Where it shows up today
- As the vocoder in Tacotron 2 (Tacotron 2) and SV2TTS (SV2TTS).
- Parallel WaveNet and WaveGlow (Parallel WaveNet + WaveGlow) made it parallel; Google deployed Parallel WaveNet in the Assistant.
- LPCNet (LPCNet) kept the autoregressive sampling but moved most of the work into DSP so it runs on a phone CPU.
- GAN vocoders (MelGAN + Parallel WaveGAN, HiFi-GAN) reuse its dilated-conv residual blocks without the autoregression.
- The idea "classify the next discrete audio symbol" returns at a coarser level in codec language models: VALL-E (VALL-E) predicts codec tokens at 75 Hz instead of samples at 16 kHz. That 200x drop in sequence rate is the whole reason codec LMs are feasible.
What it got wrong, and what came next
- Speed: strictly sequential sample generation. Fixed by distillation into flows (Parallel WaveNet, Parallel WaveNet + WaveGlow), direct flows (WaveGlow, Parallel WaveNet + WaveGlow), GANs (MelGAN + Parallel WaveGAN, HiFi-GAN), DSP + small RNN (LPCNet), and predicting STFT frames with an inverse STFT (iSTFTNet, Vocos, iSTFTNet + Vocos).
- Not end to end: it still needed a classical front-end for linguistic features, durations and F0. Tacotron (Tacotron) and Tacotron 2 (Tacotron 2) learned those from characters.
- 8-bit output: μ-law at 256 levels adds a little quantization noise; Tacotron 2's WaveNet switched to a mixture of logistics over 16-bit values.
Check yourself
Q1Compute the receptive field of a kernel-2 causal stack with dilations 1, 2, 4, 8, 1, 2, 4, 8.
1 + (1+2+4+8) + (1+2+4+8) = 1 + 15 + 15 = 31 samples. Each block of four layers adds 15 samples of history, so repeating blocks grows the receptive field linearly while each block grows it exponentially.
Q2Why μ-law and a 256-way softmax instead of predicting a real number?
A softmax can represent multi-modal distributions over the next sample, while L2 regression collapses to the mean and sounds muffled. μ-law makes 256 levels enough by spending resolution where hearing is sensitive (quiet sounds) and keeping the softmax small.
Q3Why is training fast but generation slow?
During training all past samples are known, so causal convolutions compute every time step's prediction in one parallel pass. During generation each sample depends on the previous generated sample, so you need one network pass per sample, 16,000 per second of audio at 16 kHz.
Q4What would happen if you removed the dilation and kept the same number of layers?
The receptive field would shrink from about a thousand samples per block to about ten, i.e. under a millisecond. The model could produce locally plausible waveform shapes but no pitch periods or phoneme structure, so it would sound like noise or buzz.
Q5What does the gated activation do that a ReLU does not?
It computes a content signal (tanh) and a separate gate (sigmoid) and multiplies them, so the network can switch channels on or off depending on the input and conditioning. This multiplicative interaction worked better than ReLU in PixelCNN and WaveNet, and it is also where conditioning vectors are injected.
Q6How does WaveNet use linguistic features at 200 frames per second when it runs at 16,000 samples per second?
It upsamples them to the sample rate with learned transposed convolutions (or repetition) and adds a projection of them inside every gated activation. Each sample therefore sees the local linguistic context as well as the past waveform.
When you're back at a laptop
- Implement μ-law encode/decode in NumPy; round-trip a WAV and listen to the 8-bit quantization noise.
- Build a 2-block causal stack in PyTorch (
nn.Conv1dwithdilation=d, left-padding ofd), feed an impulse at position 0, and find the last output position that changes: that is your receptive field. Check it matches1 + Σ d·(k-1).
x = torch.zeros(1, 1, 4000); x[..., 0] = 1.0
y = net(x) # weights all positive
rf = (y.abs() > 0).nonzero()[:, -1].max() + 1- Time 1 second of naive sample-by-sample generation on CPU and on the 3050 to feel why vocoders moved on.