One-sentence summary: Attention is a learned routing system: Query-Key dot products score allowed position pairs, then scaling, masking, and Softmax turn those scores into weights used to blend Values.

9.1 Review: What We Know So Far

Before opening up Attention, let's map what we have built:

We now have all the prerequisites. This chapter finally opens up the Attention mechanism — the part that makes the Transformer genuinely different from everything that came before it.

9.2 Attention in the Architecture

9.2.1 The Transformer Block Structure

Each Transformer block contains two major sub-layers:

Input
  ↓
LayerNorm
  ↓
Masked Multi-Head Attention   <- this chapter
  ↓
Residual connection
  ↓
LayerNorm
  ↓
Feed Forward Network (FFN)
  ↓
Residual connection
  ↓
Output

This figure shows a GPT-2-style pre-norm block. Other architectures change the norm type, ordering, or parallelism. In this example, Attention is the first major sub-layer and directly mixes information across allowed positions; one Attention operation is not a complete account of what the model understands.

9.2.2 The Internal Flow of Scaled Dot-Product Attention

Inside the Attention sub-layer (details in Chapter 10, overview here):

Input X
  ↓
Project X into Q, K, V using learned weight matrices W_Q, W_K, W_V
  ↓
Compute Q @ Kᵀ  (pairwise compatibility scores)
  ↓
Scale by 1/√d_k
  ↓
Mask (causal masks block future positions; padding masks can also apply)
  ↓
Softmax (normalize each row into attention weights)
  ↓
Weighted sum of V using attention weights
  ↓
Concatenate across heads
  ↓
Output projection W_O after heads are concatenated
  ↓
Output

This chapter focuses on the geometric intuition: why dot product, what do the heatmaps show, and what does "attention weight" actually mean. Chapter 10 covers Q, K, and V in detail.

9.3 Why Attention Exists

9.3.1 The Core Problem in Language Understanding

Consider this sentence:

"The luthier carried the theorbo into the workshop before repairing the instrument."

When the model processes “the instrument” near the end, it needs to connect that phrase to “the theorbo” several tokens earlier, not to the luthier or the workshop.

Every word's meaning depends on its relationship to other words. Resolving references, understanding subject-verb agreement, tracking long-range dependencies — all of these require information flow across positions in the sequence.

9.3.2 Why RNN Struggled with This

Before the Transformer, the standard approach for sequence processing was the Recurrent Neural Network (RNN):

token₁ → token₂ → token₃ → token₄ → token₅ → ...
              ↘         ↘         ↘
              hidden state flows forward

Problems with RNNs:

1. Sequential computation: the model must process token 1 before token 2, token 2 before token 3. No parallelism. Training on long sequences is slow.

2. Long-range dependency decay: information from token 1 must survive many hidden-state transitions to influence token 100. In practice, it often doesn't. The model forgets long-ago context.

3. Gradient problems: backpropagating through long sequences leads to vanishing or exploding gradients, making training difficult.

9.3.3 Attention's Solution

Attention shortens the path between positions. Bidirectional Attention can connect every pair; causal decoder Attention connects each position only to itself and earlier positions:

         token₁  token₂  token₃  token₄  token₅
token₁     ↔       ↔       ↔       ↔       ↔
token₂     ↔       ↔       ↔       ↔       ↔
token₃     ↔       ↔       ↔       ↔       ↔
token₄     ↔       ↔       ↔       ↔       ↔
token₅     ↔       ↔       ↔       ↔       ↔

Allowed positions do not need to communicate through a chain of recurrent hidden states. In the computation graph, a permitted distant position is one Attention operation away. That direct path does not guarantee the model will use distant information, and learned positional effects can still favor some distances.

The analogy: RNNs process sequences like a phone chain — information passes one person to the next. Attention is a direct broadcast over permitted positions. In causal mode, later speakers can hear earlier speakers, but earlier speakers cannot hear the future.

9.4 Dot Product as a Compatibility Score

9.4.1 Recap from Chapter 8

In Chapter 8 we established:

Q · K = |Q| × |K| × cos(θ). The score depends on both direction and magnitude, so it is not cosine similarity.

Attention uses this as a fast, differentiable, learned compatibility score for “how strongly should this Query read this Key?”

9.4.2 Finding Related Tokens with Dot Products

Here are hand-built 3D toy vectors to illustrate the arithmetic. Real Attention first projects the input separately into Q and K:

luthier     = [0.2, 0.8, 0.3]
carried     = [0.3, 0.7, 0.4]
theorbo     = [0.1, 0.9, 0.2]
instrument  = [0.8, 0.2, 0.7]

Treat the toy “instrument” vector as a Query and the others as toy Keys:

instrument · luthier = 0.8×0.2 + 0.2×0.8 + 0.7×0.3 = 0.16 + 0.16 + 0.21 = 0.53
instrument · carried = 0.8×0.3 + 0.2×0.7 + 0.7×0.4 = 0.24 + 0.14 + 0.28 = 0.66
instrument · theorbo = 0.8×0.1 + 0.2×0.9 + 0.7×0.2 = 0.08 + 0.18 + 0.14 = 0.40

The toy ranking is “carried” (0.66), “luthier” (0.53), then “theorbo” (0.40). It is not a measurement from a trained model and proves nothing about the true semantic ranking. The real model learns W_Q and W_K so that these scores become useful for its task.

9.4.3 Matrix Multiply Computes All Compatibility Scores at Once

Computing Query-Key scores one pair at a time is slow. Matrix multiplication batches them:

Q [n, d_k] @ K.T [d_k, n] = score matrix [n, n]

Entry (i,j) is the dot product between Query i and Key j. For a sequence of length 512, this produces a 512×512 score matrix in one call. Dense matrix-multiplication kernels are optimized for exactly this operation.

9.5 Attention Heatmaps: Visualizing the Weight Matrix

9.5.1 What Scaling, Masking, and Softmax Produce

After computing QKᵀ / √d_k, adding the causal mask, and applying Softmax, we get an attention weight matrix. The teaching figure has 8 positions, so it is 8×8. Each row sums to 1 over the allowed Key positions.

This matrix can be visualized as a heatmap:

• X-axis (columns): the Key positions — which token is being attended to.
• Y-axis (rows): the Query positions — which token is doing the attending.
• Color: bright (yellow) = high attention weight; dark = low weight.

9.5.2 What to Look For in a Heatmap

Possible patterns:

1. A bright diagonal cell: a Query gives its own position substantial weight. Self-attention is allowed, but not every head must prefer itself.

2. A bright off-diagonal cell: Query i gives Key j a relatively large weight in this head for this input.

3. A dark upper triangle: the causal mask gives all future positions exactly zero weight.

9.5.3 What Heatmaps Do Not Tell You

Attention heatmaps are useful for intuition and debugging. But they are not a complete explanation of what the model understands or why it produced an output.

A single heatmap shows one Attention head in one layer for one input. Other heads, residual connections, FFNs, and later layers all continue to transform the representation. Research has also shown that very different attention distributions can sometimes yield equivalent predictions.

Use heatmaps as a local diagnostic, not as a standalone causal explanation.

9.6 From Compatibility Scores to Attention Weights

9.6.1 The Problem with Raw Dot Products

Raw dot product scores have no fixed range:

raw scores: [3.5, -2.1, 8.7, 0.3, ...]

These scores can be positive or negative, and their absolute scale depends on the vector magnitudes. We cannot interpret them as "how much attention to pay" without normalizing.

9.6.2 Softmax to the Rescue

Softmax (Chapter 6) converts arbitrary scores into normalized nonnegative weights:

Softmax([3.5, -2.1, 8.7, 0.3]) ≈ [0.0055, 0.0000, 0.9943, 0.0002]

Now:

• Every weight is between 0 and 1.
• All weights sum to 1.
• The highest raw score gets the largest weight.
• They are the row-wise coefficients used to mix Value vectors; they are not probabilities that a token is semantically relevant.

9.6.3 The Scaling Step: Why Divide by √d_k

In the full formula, there is a scaling step before Softmax:

Why divide by √d_k?

The original Transformer gives a statistical intuition. If the components of Q and K are independent with mean 0 and variance 1, their dot product sums d_k terms, so its variance is d_k and its standard deviation is √d_k.

Large inputs to Softmax cause the distribution to become extremely peaked:

Softmax([100, 50, 40]) ≈ [1.0, 0.0, 0.0]       <- saturated
Softmax([5.0, 2.5, 2.0]) ≈ [0.883, 0.073, 0.044] <- less saturated

When Softmax saturates, most derivatives become very small and optimization becomes harder.

Dividing by √d_k (e.g., √512 ≈ 22.6) keeps the typical score scale comparable as head width grows. It does not normalize Q and K to unit length and does not turn the dot product into cosine similarity.

9.7 The Complete Attention Formula

9.7.1 The Formula

9.7.2 Step-by-Step Breakdown

Step 1: Q @ Kᵀ

• Shape: [seq_len, d_k] @ [d_k, seq_len] = [seq_len, seq_len]
• Each entry (i, j) is the dot product between Query token i and Key token j.
• Interpretation: a raw learned compatibility score for Query i and Key j.

Step 2: / √d_k

• Scalar division.
• Keeps the scores in a range where Softmax has healthy gradients.

Step 3: Add mask M, then Softmax

• Future or padding positions receive -∞ before Softmax.
• Applied row-by-row.
• Each row becomes normalized weights over allowed Key positions.
• Entry (i,j) is a mixing coefficient, not a probability that Key j is truly relevant.

Step 4: × V

• Shape: [seq_len, seq_len] @ [seq_len, d_v] = [seq_len, d_v]
• Each output token is a weighted sum of all Value vectors.
• Tokens with high attention weight contribute more to the output.

9.7.3 An Analogy: Search with Weighted Results

Think of Attention as a search system:

1. Query (Q): the search query — "what am I looking for?"
2. Key (K): the index entry for each document — "what does this token advertise?"
3. Compatibility (Q @ Kᵀ): learned matching scores for each Query-Key pair.
4. Softmax: normalize scores into a distribution over results.
5. Value (V): the actual content of each document — "what information do I contribute if selected?"
6. Output (attention_weights @ V): a blend of all documents, weighted by relevance.

Unlike a search engine that returns discrete ranked results, Attention returns a soft weighted blend. Every unmasked token can contribute; masked positions contribute exactly zero, and some allowed weights can be effectively zero.

9.8 Why Dot Product Is a Common Choice

9.8.1 Computational Efficiency

Dot product is expressible as matrix multiplication, and dense matrix multiplication is highly optimized on modern accelerators. A single Q @ Kᵀ call computes all seq_len² pairwise scores in parallel.

# One line computes all Query-Key scores
attention_scores = Q @ K.transpose(-2, -1)

The original Transformer also compared additive attention: its theoretical complexity is similar, but dot-product attention was faster and more space-efficient in practice because it maps directly to matrix multiplication. Dot product is a practical design choice, not the only mathematically valid scoring function.

9.8.2 Geometric Clarity

From Chapter 8: Q · K = |Q||K|cos(θ). Query “asks a question” and Key “advertises” features. Their dot product combines directional alignment with magnitude.

The model learns W_Q and W_K so that useful Query-Key pairs receive relatively high scores. Direction is part of that geometry, but vector lengths also matter; the score is not cosine similarity.

9.8.3 Learned Flexibility

Although the dot product operation is fixed, Q, K, and V are learned projections of the input:

Q = X @ W_Q   (shape: [seq_len, d_k])
K = X @ W_K   (shape: [seq_len, d_k])
V = X @ W_V   (shape: [seq_len, d_v])

The model learns W_Q, W_K, W_V during training. This means the model can learn:

• Which aspects of a token's representation should be used when asking a question (Q).
• Which aspects should be advertised to other tokens (K).
• What information to contribute when selected (V).

The dot product itself is a fixed operation, but the projected spaces it operates in are fully learned. This combination of a simple fixed operation with rich learned projections is what makes Attention so powerful.

9.9 Self-Attention vs. Cross-Attention

9.9.1 Self-Attention

In a decoder-only model such as GPT or Llama, Q, K, and V all come from the same input sequence:

input: "The luthier carried the theorbo into the workshop."

Q = input @ W_Q
K = input @ W_K
V = input @ W_V

Because Q, K, and V come from the same sequence, this is Self-Attention. In a causal decoder, each position can attend only to itself and earlier positions.

9.9.2 Causal Masking in Decoder Self-Attention

In a language model, the model should not be able to see future tokens when predicting the current one. If the model is generating token 5, it must not attend to tokens 6, 7, 8, ...

This is enforced by a causal mask: before Softmax, set the attention scores for all future positions to -∞. Mathematically, those positions get exactly zero weight after Softmax and effectively do not exist.

Masked attention matrix for a 5-token sequence (lower triangle):
token 1 attends to: [1]
token 2 attends to: [1, 2]
token 3 attends to: [1, 2, 3]
token 4 attends to: [1, 2, 3, 4]
token 5 attends to: [1, 2, 3, 4, 5]

Positions above the diagonal are masked out.

9.9.3 Cross-Attention

In encoder-decoder models (original Transformer, translation models), Q comes from the decoder sequence and K, V come from the encoder's output:

Encoder input: "The luthier repaired the theorbo."
Decoder input: "Le luthier a réparé le"

Q = decoder_hidden @ W_Q
K = encoder_output @ W_K
V = encoder_output @ W_V

The decoder asks questions about the encoder's representation. This is Cross-Attention.

9.9.4 This Book's Focus

This book focuses on documented decoder-only architectures such as GPT and Llama because that is the shape many engineers encounter first. Chapter 10 digs deeper into the QKV details.

9.10 Chapter Summary

9.10.1 Key Concepts

9.10.2 The Attention Formula

9.10.3 Core Takeaway

Attention applies dot-product compatibility to learned projections. Q asks, K advertises, their dot product scores the pair, scaling and masking prepare those scores, Softmax turns each row into weights, and the weights blend V. Those weights describe routing in one head and layer—not a complete explanation of the model's reasoning.

Chapter Checklist

After this chapter, you should be able to:

• Explain why Attention gives each position a direct path to every allowed position, and why that matters for long-range dependencies.
• Explain why dot product is used as a learned Query-Key compatibility score.
• Trace through the Attention formula: Q @ Kᵀ, scale, mask, Softmax, and weighted sum of V.
• Explain why we divide by √d_k before Softmax.
• Distinguish Self-Attention (same sequence) from Cross-Attention (two sequences).
• Explain what causal masking does in a decoder model.

See You in the Next Chapter

That covers the geometry of Attention. If you can explain the full pipeline — dot product scores, scaling, Softmax weights, blended Values — without looking at the formula, you are ready for what comes next.

Chapter 10 answers the question this chapter deliberately left open: what exactly are Q, K, and V? Where do they come from? What are the weight matrices W_Q, W_K, and W_V learning? And why does splitting into multiple heads help?