LPCNet

LPCNet: Improving Neural Speech Synthesis Through Linear Prediction

Valin, Skoglund · Mozilla / Xiph · 2018

Skim9 minDay 6 of 10#48 of 86

In one sentence

LPCNet lets a classic DSP filter (linear prediction) predict most of each audio sample and uses a small sparse RNN to predict only what the filter misses, giving good neural vocoder quality in real time on a phone CPU.

Why this paper exists

In 2018 neural vocoders sounded good but were expensive. WaveNet (WaveNet) needed a big GPU; WaveRNN (a single-GRU autoregressive vocoder from DeepMind) was cheaper but still too heavy for low-power devices. Meanwhile, speech codecs had known for decades that speech is very predictable from its own recent past. The network was wasting capacity learning the spectral envelope (the vocal-tract filter shape), which simple math already models well.

The core idea

Speech is roughly a source (glottal pulses or noise) passed through a filter (the vocal tract). Linear prediction (LPC) estimates that filter: each sample is approximately a weighted sum of the previous 16 samples. Compute the filter with DSP from the acoustic features, and let the neural network model only the excitation (the residual), which is much simpler and whiter.

How it works

Inputs per 10 ms frame: 18 Bark-scale cepstral coefficients (a compact spectral envelope) plus 2 pitch parameters (period and correlation). No mel spectrogram.
LPC from features: the cepstrum is converted to LPC coefficients a_1..a_16 with standard DSP, no learning.
Prediction per sample:

p_t = Σ_{k=1..16} a_k · s_{t-k}
e_t = s_t - p_t

s_t is the sample, p_t the linear prediction, e_t the excitation (residual). The network outputs a distribution over e_t (8-bit μ-law, 256 classes), and s_t = p_t + e_t.

Two networks: a frame-rate network (convolutions + dense layers over the features, run once per frame) produces a conditioning vector; a sample-rate network (two GRUs, the main one sparse) runs per sample and gets p_t, the previous sample and previous excitation as inputs.
Efficiency tricks: sparse GRU weight matrices (most weights pruned during training), pre-emphasis to make μ-law quantization noise less audible, and output sampling with a temperature tied to pitch correlation.

Build the intuition

Analogy: If you are asked to guess tomorrow's temperature, "same as today" is already a very good guess. LPC is that guess, done properly. The network only has to learn the surprise, which is a much smaller and simpler signal.

Try it mentally: on a voiced vowel the LPC residual looks like a train of spikes at the pitch period plus low-level noise. Predicting "a spike comes about every 5 ms" is easy for a small GRU; predicting a full formant-shaped waveform is not.

Numbers worth remembering

16 kHz audio, 10 ms frames, LPC order 16.
Complexity around ~3 GFLOPS, real time on a single phone CPU core (check the paper for the exact device).
Quality close to WaveRNN at a fraction of the compute (check the MOS table).

Where it shows up today

The DSP + small-network idea is common in low-power speech: the Opus codec's newer neural features (such as packet-loss concealment) come from the same Xiph line of work (check the Opus 1.5 release notes for which models).
The same philosophy, "let DSP do the part DSP is good at", is what iSTFTNet and Vocos (iSTFTNet + Vocos) do with the inverse STFT.
Relevant if you want a TTS voice running on a cheap Android phone for an Indian-language voice app, where there is no GPU.

What it got wrong, and what came next

Still autoregressive: one network call per sample, so it is slow in absolute terms on fast hardware and cannot use GPU parallelism. GAN vocoders (MelGAN + Parallel WaveGAN, HiFi-GAN) generate all samples in parallel.
Quality is capped by the 20-dim features and 16 kHz; it is a speech-only vocoder. Universal vocoders like BigVGAN (BigVGAN) handle singing, music and wideband audio.

Check yourself

Q1What does the neural network in LPCNet actually predict?

The excitation (residual) e_t, the part of each sample that the linear predictor cannot explain. The sample is reconstructed as s_t = p_t + e_t, where p_t is computed from the previous 16 samples with LPC coefficients derived from the frame features.

Q2Why does offloading the spectral envelope to LPC let the network be smaller?

The envelope (formants) is a smooth, low-dimensional structure that LPC models almost exactly with 16 coefficients. Removing it leaves a whiter, simpler residual, so the network only needs to model pitch pulses and noise, which a small sparse GRU can do.

Q3What would happen if you ran LPCNet on a GPU with large batches?

It would still be limited by the sequential per-sample loop: you can batch many utterances, but each one still needs 16,000 steps per second of audio. It suits a CPU at batch 1; a parallel GAN vocoder suits a GPU.

When you're back at a laptop

Build the reference LPCNet from the xiph/LPCNet repo (or Opus's dnn/ folder) and run its demo on a 16 kHz WAV.
In NumPy, compute LPC coefficients for a 20 ms speech frame with librosa.lpc(frame, order=16), compute the residual, and plot frame vs residual. Compare their variances.

arXiv ↗