A standard feedforward neural network processes one input at a time with no memory of what came before. For sequential data — text, time series, audio — this is a fundamental limitation. Recurrent Neural Networks solve it by maintaining a hidden state that carries information from one step to the next.
The Recurrent Cell
At each time step t, a simple RNN cell takes two inputs: the current input x_t and the hidden state from the previous step h_{t-1}. It combines them to produce a new hidden state:
h_t = tanh(W_h * h_{t-1} + W_x * x_t + b)This hidden state is both the output of the current step and the memory that gets passed to the next step. Unrolled over a sequence of length T, you can think of this as a very deep network where each layer corresponds to a time step and the same weights are shared across all steps.
The Vanishing Gradient Problem
Training RNNs with backpropagation through time (BPTT) requires computing gradients across all T time steps. Because the same weight matrix is multiplied at every step, the gradient either explodes (if the largest eigenvalue of W is greater than 1) or vanishes (if it is less than 1) exponentially with sequence length. In practice, vanishing gradients dominate — the network fails to learn dependencies between events separated by more than a few steps. The RNN forgets.
Gradient clipping partially addresses explosion. Vanishing is harder to fix with a simple RNN, which is why LSTMs were invented.
Long Short-Term Memory (LSTM)
The LSTM, introduced by Hochreiter and Schmidhuber in 1997, adds a cell state — a separate memory channel that runs alongside the hidden state. Information flows through the cell state with only minor linear interactions, which lets gradients propagate much further back in time without vanishing.
The key innovation is the gating mechanism. Three learned gates control what information enters, leaves, and persists in the cell state:
Forget gate: decides what to erase from the cell state.
f_t = sigmoid(W_f * [h_{t-1}, x_t] + b_f)Input gate: decides what new information to write.
i_t = sigmoid(W_i * [h_{t-1}, x_t] + b_i)
g_t = tanh(W_g * [h_{t-1}, x_t] + b_g)
C_t = f_t * C_{t-1} + i_t * g_tOutput gate: decides what to expose from the cell state as the hidden state.
o_t = sigmoid(W_o * [h_{t-1}, x_t] + b_o)
h_t = o_t * tanh(C_t)All three gates use sigmoid activations, producing values between 0 and 1 that act as soft on/off switches. The forget gate output near 0 means "erase everything"; near 1 means "keep everything". This selective memory is what allows LSTMs to maintain relevant context across hundreds of time steps.
Gated Recurrent Unit (GRU)
The GRU simplifies the LSTM by merging the cell state and hidden state, and combining the forget and input gates into a single update gate. It has fewer parameters and trains faster, often matching LSTM performance on tasks where long-range dependencies are not extreme:
z_t = sigmoid(W_z * [h_{t-1}, x_t]) # update gate
r_t = sigmoid(W_r * [h_{t-1}, x_t]) # reset gate
h_tilde = tanh(W * [r_t * h_{t-1}, x_t])
h_t = (1 - z_t) * h_{t-1} + z_t * h_tildeSequence-to-Sequence Architectures
LSTMs became the dominant architecture for sequence-to-sequence tasks like machine translation. An encoder LSTM reads the input sequence and compresses it into a context vector — the final hidden state. A decoder LSTM then generates the output sequence one token at a time, conditioned on that context vector.
The limitation of this approach is the bottleneck: the entire input sequence must be compressed into a single fixed-size vector, which degrades on long sequences. This limitation led directly to the invention of attention mechanisms, and eventually to the Transformer — an architecture that dropped recurrence entirely in favor of attention. But the LSTM's gating ideas remain influential and LSTMs still see use in settings where sequence length is moderate and parallelism matters less than memory efficiency.