Transformers and Attention

The 2017 paper "Attention Is All You Need" by Vaswani et al. introduced the Transformer and changed the trajectory of deep learning. Every major language model today — GPT, BERT, Claude, Gemini — is built on this architecture. Understanding it means understanding the foundation of modern AI.

The Problem with RNNs

Recurrent networks process sequences one step at a time. This makes them inherently sequential — you cannot compute step t until you have computed step t-1. For long sequences, this is slow. More critically, even with LSTMs, capturing dependencies between tokens far apart in a sequence is difficult. The Transformer eliminates recurrence entirely and replaces it with attention, which can directly connect any two positions in a sequence regardless of distance.

Attention: The Core Idea

Attention asks: for each token in the sequence, which other tokens should it attend to, and how much? The answer is computed from three learned projections of the input — Queries (Q), Keys (K), and Values (V):

Attention(Q, K, V) = softmax(Q * K^T / sqrt(d_k)) * V

Intuitively: Q represents what each token is looking for. K represents what each token offers. The dot product Q * K^T computes a compatibility score between every pair of tokens. Dividing by sqrt(d_k) prevents the scores from becoming too large in high dimensions, which would push softmax into a near-zero gradient region. Softmax converts scores into a probability distribution — the attention weights. Finally, these weights are used to take a weighted sum of the Values, producing the output for each token.

The result is that every token's output representation is a mixture of all other tokens' values, weighted by how relevant they are. The word "bank" in "river bank" will attend heavily to "river"; in "bank account" it will attend heavily to "account". The meaning is resolved by context, dynamically.

Multi-Head Attention

A single attention operation captures one type of relationship. Multi-head attention runs h attention operations in parallel, each with different learned Q, K, V projections. This allows the model to simultaneously attend to syntactic structure, semantic similarity, coreference, and other relationships at the same time:

MultiHead(Q, K, V) = Concat(head_1, ..., head_h) * W_O

where head_i = Attention(Q * W_Q_i, K * W_K_i, V * W_V_i)

The outputs of all heads are concatenated and projected back to the model dimension. GPT-3 uses 96 attention heads; each learns to specialize in a different kind of linguistic relationship.

Positional Encoding

Attention has no built-in sense of order — the same computation is applied regardless of whether a token is first or last in the sequence. Positional encodings are added to the input embeddings to inject position information. The original Transformer used fixed sinusoidal encodings:

PE(pos, 2i)   = sin(pos / 10000^(2i/d_model))
PE(pos, 2i+1) = cos(pos / 10000^(2i/d_model))

Modern models often use learned positional embeddings or relative position encodings (RoPE, ALiBi) that generalize better to sequence lengths not seen during training.

The Feed-Forward Layer

After the attention layer, each token's representation passes through a position-wise feed-forward network — a two-layer MLP applied independently to each token:

FFN(x) = ReLU(x * W_1 + b_1) * W_2 + b_2

This is where most of the model's factual knowledge is thought to be stored. The attention layer handles routing and mixing; the FFN handles per-token processing.

The Full Transformer Block

Each Transformer block combines multi-head attention and a feed-forward layer, each wrapped in a residual connection and layer normalization:

x = x + MultiHeadAttention(LayerNorm(x))
x = x + FFN(LayerNorm(x))

Residual connections (adding the input back to the output) allow gradients to flow directly through the network without passing through the attention or FFN computations, enabling training of very deep models. GPT-4 is estimated to have around 120 such blocks.

Encoder vs Decoder

The original Transformer had both an encoder and decoder. The encoder processes the input with bidirectional attention — each token can attend to all others. BERT is encoder-only, making it powerful for understanding tasks (classification, question answering). The decoder uses masked (causal) attention — each token can only attend to previous tokens, enabling autoregressive generation. GPT is decoder-only. Modern frontier models like Claude and GPT-4 are all decoder-only Transformers trained to predict the next token.

Why Transformers Dominate

Three properties explain their dominance. First, parallelism: unlike RNNs, all attention computations can run simultaneously on GPU hardware, making training on massive datasets tractable. Second, direct long-range dependencies: any two tokens can interact in a single layer regardless of distance. Third, scalability: Transformer performance scales predictably with compute, data, and model size — the scaling laws discovered by Kaplan et al. showed that larger models trained on more data consistently outperform smaller ones, which drove the race to GPT-4 and beyond.