Joint CTC-Attention

Joint CTC-Attention based End-to-End Speech Recognition using Multi-task Learning

Kim, Hori, Watanabe · MERL / Carnegie Mellon University · 2016

Skim8 minDay 1 of 10#5 of 86

In one sentence

Train an attention encoder-decoder with an extra CTC loss on its encoder, so CTC's strictly monotonic alignment keeps attention on track, which gives faster convergence and lower error than either loss alone.

Why this paper exists

Attention encoder-decoders like LAS (Listen, Attend and Spell) learn alignment implicitly, and that is their weakness. Early in training, and on noisy or long utterances, attention can wander: it skips ahead, repeats, or never forms a clean diagonal. Training is slow because the model must discover the alignment from scratch. CTC (CTC) has the opposite profile: its alignment is forced to be monotonic (left to right, never jumping back), but it assumes frame outputs are independent. Each method's strength is the other's weakness.

The core idea

Share one encoder between two heads and train both at once (multi-task learning):

an attention decoder, which models dependencies between output tokens,
a CTC layer on the encoder output, which forces the encoder to produce representations that line up monotonically with the transcript.

Because the encoder must satisfy CTC, the attention decoder receives features that already "know" the order of the labels, and attention locks onto a sensible alignment much sooner.

How it works

audio -> shared ENCODER (BLSTM) -> h_1..h_T
                |               \
           CTC layer         attention decoder
                |                   |
            L_ctc               L_att
                \                  /
      L = λ·L_ctc + (1-λ)·L_att

L_ctc = -log P_ctc(y | x), the standard CTC loss on encoder outputs.
L_att = -log P_att(y | x), the usual teacher-forced cross-entropy of the attention decoder.
λ (between 0 and 1) sets the weight. λ = 0 is pure attention, λ = 1 is pure CTC; the paper compares a few values in between.
The paper used location-aware attention and evaluated on WSJ and CHiME-4 (noisy speech).
In this paper CTC is used only during training. Using the CTC score during decoding too (joint CTC/attention beam search, which rejects hypotheses CTC finds implausible, such as too short or with repeated chunks) came in follow-up work by the same group, collected in Watanabe et al.'s "Hybrid CTC/Attention Architecture" (2017).

Build the intuition

Analogy: training a new transcriber (attention) alongside a strict metronome (CTC). The metronome cannot write good sentences, but it never lets anyone skip ahead or go back, and the transcriber learns the rhythm much faster.

What would happen with λ near 1? The model becomes mostly CTC, and the attention decoder gets little gradient. With λ = 0 you lose the alignment help. The useful range is in between; ESPnet recipes commonly use a CTC weight of around 0.3, but check each recipe.

Numbers worth remembering

Data: WSJ (both a ~15 h subset and the ~81 h full training set) and CHiME-4.
Result: lower character error rate than attention-only and CTC-only baselines on these sets, and visibly faster convergence (check the paper's tables and learning curves for exact values).
Loss is a single weighted sum with one hyperparameter, λ.

Where it shows up today

The default training recipe in ESPnet, including OWSM (OWSM).
Many AED and Conformer (Conformer) systems keep an auxiliary CTC head; NVIDIA NeMo offers hybrid transducer-CTC models, where the CTC head can also be used alone for cheap decoding.
CTC-based forced alignment for timestamps (WhisperX).

What it got wrong, and what came next

Still not streaming. The attention decoder needs the full utterance. Transducers (RNN-T) and cache-aware streaming (Cache-aware streaming Conformer) took that role.
Joint decoding is slower and more complex than plain beam search. Whisper (Whisper) dropped CTC entirely and relied on scale, and pays for it with hallucinations that a CTC check would catch.
Two heads, one more hyperparameter to tune per dataset.

Check yourself

Q1What does the CTC loss contribute that attention lacks?

A hard monotonic alignment constraint. CTC forces the encoder's outputs to line up with the label sequence in order, so the attention decoder gets features that make a clean, left-to-right alignment easy to learn, which reduces skipping and looping.

Q2Write the training objective and explain λ.

L = λ·L_ctc + (1-λ)·L_att, where both terms are negative log-likelihoods of the transcript. λ balances the two: 0 means pure attention, 1 means pure CTC, and intermediate values let CTC regularise the shared encoder while the attention decoder does the main modelling.

Q3How can CTC help at decoding time, not only training?

In joint decoding, each beam hypothesis is scored by both the attention decoder and a CTC prefix score. Hypotheses that CTC finds unlikely, for example ones that end too early or repeat a phrase, are penalised, which removes many typical attention errors. This was developed in follow-up work.

When you're back at a laptop

Add a linear CTC head to the encoder of a small attention model (or your LAS from Listen, Attend and Spell) and train with λ in {0, 0.3, 1}.
Plot validation loss and attention matrices over the first few epochs for λ = 0 versus λ = 0.3, and look for the diagonal appearing earlier.

arXiv ↗