Sequence models: why RNNs lost

A recurrent network processes a sequence one element at a time, updating a hidden state: $h_t = \tanh(W_h h_{t-1} + W_x x_t + b)$. The same weights are reused at every step, and the hidden state is the network's memory of everything so far.

Training unrolls the recurrence into a deep feed-forward graph, one layer per time step, and backpropagates through all of it: backpropagation through time (BPTT). Long sequences are often truncated to bound cost.

The structure is inherently sequential: you cannot compute $h_{100}$ before $h_{99}$.

The gradient flowing from step t back to step t−k passes through k copies of the recurrent Jacobian. If its largest eigenvalue is below 1, the product shrinks exponentially (vanishes); above 1, it grows exponentially (explodes).

Consequence: plain RNNs struggle to learn dependencies more than a few dozen steps apart. In 'The keys that the man near the old cabinets left on the table are lost', the verb agreement depends on a word many steps back.

Clipping fixes explosions. Vanishing needs an architectural fix: gates (LSTM/GRU), and ultimately attention, which connects any two positions directly.

An LSTM adds a cell state $c_t$ that runs through time with only element-wise, gated updates, a 'conveyor belt' along which gradients flow more easily. Three sigmoid gates, each between 0 and 1, control it.

The forget gate decides what to erase from the cell; the input gate decides what new candidate information to write; the output gate decides what part of the cell to expose as the hidden state.

Because the cell update is additive ($c_t = f_t \odot c_{t-1} + i_t \odot \tilde c_t$), gradients don't have to pass through a squashing non-linearity at every step: the same insight as ResNet's skip connections.

The GRU merges the cell and hidden state and uses an update gate (how much of the old state to keep vs replace) and a reset gate (how much of the past to use when proposing the new state).

Fewer parameters, faster training, and often comparable results to LSTM. Neither consistently dominates; the choice was usually empirical.

Both are now mostly historical for language, but still appear in small on-device and streaming models.

A bidirectional RNN runs one RNN left-to-right and another right-to-left and concatenates their states, so each position sees both past and future context. Great for tagging and classification; impossible for generation, where the future doesn't exist yet.

Stacked RNNs feed one layer's hidden states as the next layer's input sequence, building more abstract representations.

Encoder side of seq2seq (next week) was typically a bidirectional, stacked LSTM. BERT's 'bidirectional' name echoes this.

No parallelism across time. Step t needs step t−1, so a 1,000-token sequence needs 1,000 sequential steps. GPUs, which excel at huge parallel matrix multiplies, sit mostly idle. Transformers process all positions at once during training.

A fixed-size bottleneck. Everything the model knows about the past must squeeze into one hidden vector. In seq2seq translation, the whole source sentence was compressed into a single vector before decoding began.

Long-range dependencies are hard. Even with gates, information from far back degrades. Attention lets any position look directly at any other in one step, with a path length of 1. Hold these three failures in mind: next week each one gets solved.