iSTFTNet + Vocos

Why this paper exists

In HiFi-GAN (HiFi-GAN) the expensive layers are the last ones: after 8x, 8x, 2x upsampling the network is running convolutions at 22,050 positions per second. Most of the compute goes into turning a frame-rate signal into a sample-rate signal, something the inverse STFT does exactly and almost for free. Earlier spectrogram-based approaches (Griffin-Lim, used by Tacotron Tacotron) failed on phase: estimating it iteratively from magnitude gave metallic audio, and directly predicting phase with a network was considered too hard because phase wraps around at ±π and looks like noise.

The core idea

Speech is redundant in the time domain and compact in the time-frequency domain. If the network outputs, for every frame, a magnitude |X(f)| and a phase φ(f) for each frequency bin, the iSTFT (inverse FFT of each frame, windowing, overlap-add) reconstructs the waveform deterministically. So the network can stay at a low time resolution, and a GAN discriminator plus mel loss teach it to produce phase that sounds right, even if it does not match the original phase.

How it works

The iSTFT, as a reminder

for each frame m:
  X_m(f) = M_m(f) · exp(j·φ_m(f))     (complex spectrum)
  y_m    = irfft(X_m)                 (n_fft samples)
  y[m·hop : m·hop + n_fft] += w · y_m (window, overlap-add)

M is magnitude, φ phase, n_fft the frame length, hop the shift between frames, w the synthesis window. Each frame of n_fft/2 + 1 complex bins produces hop new samples.

iSTFTNet

mel (80 x T)
  │ Conv1d
  ▼
[ConvT x8] → MRF          (HiFi-GAN stages,
[ConvT x8] → MRF           64x upsampling)
  │ Conv1d → magnitude (exp), phase
  ▼
iSTFT (n_fft 16, hop 4)   → 4x more samples
  ▼
waveform (1 x 256T)

• Named by its stages: HiFi-GAN V1 is C8C8C2C2 (four upsampling convs, product 256). iSTFTNet C8C8I keeps the two 8x stages and replaces the two 2x stages with an iSTFT whose hop gives the remaining 4x. C8I removes even more.
• Small FFT sizes (like 16) are enough because the earlier conv layers have already reached a high time resolution; the iSTFT only handles the last short span.
• Same discriminators and losses as HiFi-GAN. Faster and smaller, with quality close to HiFi-GAN for moderate replacement (check the paper for where quality drops).

Vocos

mel or EnCodec tokens (T frames)
  │ embed → Conv1d
  ▼
ConvNeXt blocks x N  (all at frame rate,
  │  ~86-94 frames/s, 512 dims)
  ▼
Linear → m, p   (513 bins each for n_fft 1024)
  │ M = exp(m);  X = M·(cos p + j·sin p)
  ▼
iSTFT (n_fft 1024, hop 256)  → 256 samples/frame
  ▼
waveform

• Backbone: ConvNeXt blocks (depthwise conv with a wide kernel, LayerNorm, pointwise MLP with GELU, residual), borrowed from vision. No upsampling anywhere; time resolution stays at the frame rate.
• Phase head: the network outputs an unbounded value p per bin and uses cos p and sin p as the real and imaginary parts (times the magnitude). Because only cos and sin of p matter, wrapping at ±π is a non-issue: p and p + 2π give the same output.
• Training: multi-period discriminator (HiFi-GAN) + multi-resolution (STFT) discriminator, hinge loss, feature matching, mel L1.
• Two variants: decode a 100-bin mel at 24 kHz, or decode EnCodec tokens (EnCodec) at several bitrates (conditioning on the bandwidth with adaptive layer norm), often sounding better than EnCodec's own decoder at low bitrates.

Build the intuition

Where the compute goes

For 1 second at 24 kHz with hop 256 (~94 frames):

Vocos runs every conv over ~94 positions instead of thousands. The iSTFT itself is an FFT per frame, which is tiny next to a conv stack. That is the whole speed story.

Analogy: Time-domain vocoders paint every pixel of a picture by hand. iSTFT vocoders write down the recipe (which tones, how loud, what phase) for each short moment, and a machine (the inverse FFT) paints the pixels exactly.

Why phase is learnable here but not with Griffin-Lim

Griffin-Lim searches for a phase consistent with a magnitude, with no idea what speech should sound like. A GAN-trained phase head is judged by discriminators on the final waveform, so it learns phase that produces natural-sounding audio, not necessarily the original phase. Many different phases sound the same; the GAN only needs one of them.

Numbers worth remembering

• Vocos: ~13.5M parameters (check the paper), mel at 24 kHz with n_fft 1024, hop 256 → 513 frequency bins per frame.
• Vocos is reported to be roughly an order of magnitude faster than HiFi-GAN on GPU and much faster than BigVGAN, with similar quality on LibriTTS (check its speed table for exact RTFs).
• iSTFTNet C8C8I: two 8x upsampling stages, then iSTFT with n_fft 16 and hop 4 (product 8·8·4 = 256).
• Vocos EnCodec variant: 1.5, 3, 6 and 12 kbps bandwidths.

Where it shows up today

• Kokoro (StyleTTS → StyleTTS 2) uses an iSTFTNet decoder, which is part of why an 82M model runs fast on CPU.
• F5-TTS (F5-TTS) and IndicF5 vocode their mels with Vocos; several other flow-matching TTS systems do too.
• CosyVoice (CosyVoice → CosyVoice 2) uses a HiFi-GAN-style vocoder with an iSTFT output stage (check its paper and repo), the same idea as iSTFTNet.
• Codec decoders increasingly use iSTFT heads to save compute; the idea is central for running TTS on CPU or a 4 GB GPU like your RTX 3050.

What it got wrong, and what came next

• iSTFTNet: replacing too many upsampling stages hurts quality (the frame-level network has to predict too much detail); there is a speed-quality sweet spot rather than a free lunch.
• Vocos at a frame rate of ~94 Hz still needs a decent magnitude and phase per bin; at very low bitrates or for music with sharp transients, time-domain models can still win (check the paper's comparisons).
• Both are offline per utterance in their basic form; streaming needs causal convs and careful overlap-add handling at chunk boundaries, as in codec decoders like Mimi (Moshi).
• Both still need an acoustic model in front. End-to-end systems like VITS (VITS) and StyleTTS 2 (StyleTTS → StyleTTS 2) train decoder and acoustic model together, and Kokoro shows an iSTFTNet decoder fits inside them.

Check yourself

When you're back at a laptop

• pip install vocos, load charactr/vocos-mel-24khz, vocode a mel, and compare RTF with HiFi-GAN on CPU and your 3050.
• Write a minimal Vocos-style head: m, p = linear(h).chunk(2), spec = torch.exp(m) * torch.exp(1j * p), torch.istft(spec, n_fft=1024, hop_length=256, window=torch.hann_window(1024)). Check the output length is ~256 · frames.
• Take a real STFT, keep the magnitude and randomize the phase, invert, and listen; then do Griffin-Lim. This shows why phase prediction matters.