Listen, Attend and Spell

Why this paper exists

By 2015 there were two ways to train neural ASR without frame alignments, and both had gaps:

• CTC (CTC) assumes each frame's output is independent of the others given the audio, so it needs an external language model to spell well.
• RNN-T (RNN-T) fixes that, but at the time it was not widely used or engineered for large tasks.

Meanwhile attention-based sequence-to-sequence models had just transformed machine translation. Bahdanau-style attention lets a decoder look at any part of the input at every output step. The question was whether the same recipe could map thousands of audio frames directly to characters on a real, large task (Google voice search), with one network and one loss.

The obstacle: attention over very long inputs. A few seconds of audio is hundreds of frames; attention over a raw sequence that long trained slowly and poorly.

The core idea

Two pieces:

1. Listen with a pyramidal BiLSTM (pBLSTM) that halves the time resolution at each layer, so the decoder attends over a sequence 8× shorter than the input.
2. Attend and spell with an LSTM decoder that, at each output step, computes attention weights over all encoder states, builds a context vector, and predicts the next character from that context and the previous characters.

The model directly learns P(y | x) as a product of next-character probabilities, the same factorisation as a language model, so language knowledge is learned inside the decoder.

How it works

log-mel frames x_1..x_T
         |
  BLSTM layer
         |
  pBLSTM: concat pairs, T/2
         |
  pBLSTM: concat pairs, T/4
         |
  pBLSTM: concat pairs, T/8
         |
  h_1..h_U'   (listener output)
         |
         v   attention at every step i
  +-------------------------------+
  | s_i = LSTM(s_{i-1}, y_{i-1},  |
  |            c_{i-1})           |
  | e_iu = <phi(s_i), psi(h_u)>   |
  | a_i  = softmax(e_i)           |
  | c_i  = sum_u a_iu * h_u       |
  | P(y_i) = MLP(s_i, c_i)        |
  +-------------------------------+
         |
  characters, one at a time, until <eos>

The factorisation

P(y | x) = Π_i P(y_i | x, y_<i)

y_i is the i-th output character, y_<i are all previous characters. Nothing is assumed independent. Training maximises log P(y | x) with the true previous characters fed in (teacher forcing).

The listener

A pyramidal layer takes the previous layer's outputs at steps 2i and 2i+1, concatenates them, and feeds them to a BLSTM:

h_i^j = pBLSTM(h_{i-1}^j, [h_{2i}^{j-1}, h_{2i+1}^{j-1}])

Three such layers give an 8× time reduction. With 10 ms input frames, the listener outputs one vector per 80 ms. This is the same idea as the 4× and 8× convolutional subsampling in Conformer (Conformer) and Fast Conformer (Fast Conformer).

The speller

• s_i is the decoder LSTM state, updated from the previous state, the previous character and the previous context.
• φ and ψ are small MLPs; the energy e_{i,u} is their dot product, a content-based score of how relevant encoder step u is to output step i.
• α_i is the softmax of those energies over all u: the attention weights, summing to 1.
• c_i is the weighted sum of encoder states: what the decoder "hears" right now.
• An MLP over s_i and c_i gives the character distribution.

Training and decoding tricks

• Sampling the previous token. During training, the previous character fed to the decoder was sometimes taken from the model's own prediction instead of the ground truth (about 10% of the time), to reduce the mismatch between training and inference. This is a form of scheduled sampling.
• Beam search over characters, then rescoring the beam with an external n-gram language model, with a length normalisation term, because the raw model prefers short outputs.

Build the intuition

Analogy: a human transcriber with a recording and a cursor. Before writing each letter they glance back over the whole recording, focus on the part that matters for the next letter, and write it, remembering what they have written so far. CTC writes one symbol per tick of a clock without looking at its notes; LAS writes at its own pace and decides where to look.

What attention learns

On a correct utterance the attention weights form a roughly diagonal band: output character i attends to encoder frames near the time that character is spoken. Nobody tells the model this. The alignment emerges from the loss. When it does not emerge, things go wrong:

• attention gets stuck on one region → the decoder repeats a word or phrase,
• attention jumps ahead → words are skipped,
• the decoder ignores the audio → it produces fluent text that was never said.

These failure modes are the ancestors of Whisper's hallucinations (Whisper) and of Tacotron's word skipping in TTS (Tacotron 2).

A thought experiment: why not attend over raw frames?

A 10-second utterance at 100 frames/s is 1000 frames. Each output character computes 1000 attention scores, and an utterance has maybe 150 characters, so 150,000 scores, each requiring the decoder to find a needle in a long, highly redundant sequence. The paper reported that without the pyramid the model trained slowly and converged poorly. At 125 frames (8× fewer), attention has fewer, more distinct positions to choose from.

Not tied to one spelling

Because the decoder is a conditional language model, the beam can hold genuinely different transcriptions of the same audio. The paper's example is a query containing "AAA" where the beam held both "aaa" and "triple a", something a frame-synchronous CTC model would struggle to do.

Numbers worth remembering

• Task: Google voice search, about 3 million utterances (roughly 2000 hours).
• Time reduction in the listener: 8× (three pBLSTM layers).
• WER: about 14.1% without an LM and about 10.3% with LM rescoring, versus about 8.0% for Google's tuned CLDNN-HMM hybrid system of the time (check Table 1 of the paper). LAS did not win yet; it showed an end-to-end model could get close.
• Output units: characters (letters, digits, punctuation, space, plus start/end tokens).

Where it shows up today

LAS defines the attention encoder-decoder (AED) family. Its descendants swap LSTMs for Transformers and Conformers, but keep "encoder + cross-attending autoregressive decoder":

• Whisper (Whisper) is the large-scale, multitask version; Distil-Whisper (Distil-Whisper) shrinks the decoder.
• OWSM (OWSM) and NVIDIA Canary (Canary).
• Hybrid CTC/attention in ESPnet (Joint CTC-Attention).
• SpecAugment (SpecAugment) was first demonstrated on LAS models.
• Speech-LLMs (SALMONN / Qwen-Audio / Qwen2-Audio) are AED models in which the decoder is a full LLM.
• In TTS, Tacotron (Tacotron, Tacotron 2) is the mirror image: attention decoder from text to spectrogram.

What it got wrong, and what came next

• Unstable alignment and slow convergence. Joint CTC-Attention (Joint CTC-Attention) added a CTC loss on the encoder to force monotonic alignment.
• Not streaming. The decoder attends over the full utterance. Transducers (RNN-T, Streaming E2E ASR for mobile) became the streaming choice; delayed-streams models (Kyutai DSM) are another route.
• Slow decoding. Autoregressive, one token per decoder step, with beam search. Distil-Whisper (Distil-Whisper) and CTC/TDT models (TDT) attack this.
• Hallucination and looping when attention fails, still visible in Whisper (Whisper).
• Accuracy gap to hybrids closed with more data, SpecAugment (SpecAugment), subword units and better encoders (Conformer).

Check yourself

When you're back at a laptop

• Build a tiny LAS in PyTorch: 2-3 pBLSTM layers (concatenate pairs then LSTM), a 1-layer LSTM decoder with dot-product attention, character outputs.
• Train it on a small set (e.g. LibriSpeech dev-clean or a few hours of an Indian-language Kathbath split, IndicSUPERB); it fits on a 4 GB GPU with batch 8-16 and short utterances.
• Plot the attention matrix (output step × encoder step) for a few utterances and watch the diagonal appear during training.
• Feed it silence and see what the decoder does.