One-sentence summary: Q is “what I am looking for,” K is “what label I advertise,” and V is “what content I contribute.” Query-Key dot products produce learned compatibility scores; normalized weights then blend the allowed Value vectors.

10.1 What This Chapter Covers

The previous chapter built the geometric intuition: a dot product can serve as a learned, magnitude-aware Query-Key compatibility score.

But several questions are still open:

What do Q, K, and V actually mean?
How are they generated from the input?
How does the shape change at each step of the computation?

This chapter traces the complete Attention computation, step by step, keeping track of every dimension change.

10.2 The Input Shape

10.2.1 Understanding the Input Dimensions

Input tensor shape: batch, sequence, model dimension

In practice, training processes multiple sequences at once. The input tensor has shape:

X: [batch_size, ctx_length, d_model]

Using concrete numbers from the diagram:

batch_size = 4: four sequences processed in parallel
ctx_length = 16: each sequence has 16 tokens
d_model = 512: each token is represented as a 512-dimensional vector

10.2.2 A Concrete Example

Imagine four prompts going through training at the same time:

1. "The luthier carried the theorbo into the..."
2. "The viol player tuned a gut string before..."
3. "A sackbut rested beside the cornett while..."
4. "The continuo part ended on a suspended..."

For this teaching example, each prompt is padded or truncated to 16 token positions, and each position is represented by a 512-dimensional vector.

Total input shape: [4, 16, 512]

4 sequences
16 positions each
512 dimensions per position

10.2.3 The Three Dimensions

Dimension	Name	Meaning
`batch_size`	batch	how many sequences we process at once
`ctx_length`	context length	how many tokens per sequence
`d_model`	model dimension	width of each token vector

10.3 Generating Q, K, and V

10.3.1 The Core Idea

Q, K, V generated from the same input via three weight matrices

Q, K, and V all come from the same input X, through three different weight matrices:

Q = X @ W_Q
K = X @ W_K
V = X @ W_V

These weight matrices — W_Q, W_K, and W_V — are learnable parameters updated during training.

10.3.2 Dimension Calculation

Combined projection X @ W_Q with dimension tracking before Q is split into heads

Take generating Q as an example:

X:  [4, 16, 512]   (batch_size, ctx_length, d_model)
W_Q:[512, 512]     (d_model, h × d_k)
Q:  [4, 16, 512]   (batch_size, ctx_length, h × d_k)

The matrix multiplication rule is [..., A, B] @ [B, C] = [..., A, C], so:

[4, 16, 512] @ [512, 512] = [4, 16, 512]

In this conventional example, h × d_k = h × d_v = d_model = 512, so the combined Q, K, and V tensors happen to have the same shape as X. That is a design choice, not a rule: Q and K must share d_k within a head, while V may use a different d_v.

10.3.3 Why Three Different Matrices?

You might ask: if the output shape is the same, why bother with three separate matrices?

Because Q, K, and V play different roles:

Q (Query): "What information am I looking for?"
K (Key): "What information can I be found by?"
V (Value): "What content do I offer when selected?"

By learning separate W_Q, W_K, and W_V, the model learns different views of the same input. Q and K land in compatible per-head spaces so their dot product is defined; V carries the features that will be mixed into the output.

10.3.4 An Analogy Worth Keeping

Think of a library search:

Role	Analogy	Function
Query (Q)	The reader's search terms	“I want a book about early electronic instruments”
Key (K)	Each book's catalogue tags	“theremin, performance, history”
Value (V)	The book's actual content	The material the reader ultimately retrieves

When you search:

your Query is compared with each book's Key
books with higher match scores receive larger normalized weights
their Value content is blended according to those weights

Attention works the same way.

10.4 First Matrix Multiplication: Q @ K^T

10.4.1 Computing the Compatibility-Score Matrix

Q @ K^T produces a sequence-by-sequence score matrix

After the combined projections are reshaped into heads, the next step is computing Query-Key compatibility scores within each head:

scores = Q @ K^T

K needs to be transposed (K^T) so that Q's rows (one per token) dot-product against K's rows (also one per token).

10.4.2 Dimension Change

Q:      [4, 4, 16, 128]   (batch, heads, seq, d_k)
K^T:    [4, 4, 128, 16]   (batch, heads, d_k, seq)
scores: [4, 4, 16, 16]    (batch, heads, query_seq, key_seq)

This example uses h = 4 and d_k = d_model / h = 512 / 4 = 128. Chapter 11 explains the head split itself; here we expose the head axis so the shape path stays honest.

10.4.3 What the Result Means

The result is a [4, 4, 16, 16] tensor:

4 sequences
4 heads per sequence
each head has a 16×16 raw score matrix
entry (i,j) is the dot product of Query position i and Key position j

Here is a hand-built 4×4 example for the short prompt “luthier repairs theorbo .” The values below are illustrative raw dot products before scaling or masking; they were not measured from a trained model:

             luthier  repairs  theorbo   .
luthier  [   8.2,     3.1,     6.5,    1.2  ]
repairs  [   3.4,    11.7,     7.8,    2.0  ]
theorbo  [   6.8,     7.2,    12.3,    1.9  ]
.        [   1.3,     2.1,     1.7,    9.4  ]

The table was chosen only to make the arithmetic visible. Real heads need not have large diagonals, and a high raw score is not ground-truth semantic relevance. After scaling, masking, and row-wise Softmax, the scores become normalized mixing weights over allowed Key positions.

10.5 Scale: Why Divide by $\sqrt{d_k}$

10.5.1 The Scaling Step

Raw Query-Key scores divided by sqrt(d_k) to stabilize their typical scale

The raw scores from Q @ K^T are scaled by the head width:

scaled_scores = (Q @ K^T) / sqrt(d_k)

The goal is not to force scores into a fixed interval. It is to keep their typical scale comparable as d_k changes.

10.5.2 Why Scale?

The problem: as d_k grows, the typical magnitude of an unscaled dot product grows too.

dot product = sum(q_i × k_i)   # summing 128 multiplied pairs

If the components of Q and K are independent with mean 0 and variance 1, the dot product has variance d_k and standard deviation √d_k.

The consequence: large values drive Softmax to extremes.

Softmax([100, 1, 2])     ≈ [1.000, 0.000, 0.000]   # saturated
Softmax([1.0, 0.1, 0.2]) ≈ [0.539, 0.219, 0.242]   # less saturated

When Softmax saturates, most derivatives become very small and optimization becomes harder.

The fix: divide by $\sqrt{d_k}$ so the idealized score standard deviation stays around 1 as head width changes.

dot product / sqrt(128) ≈ dot product / 11.3

10.5.3 Where It Sits in the Formula

\text{Attention}(Q, K, V) = \text{softmax}\!\left(\frac{QK^T}{\sqrt{d_k}} + M\right) V

The $\sqrt{d_k}$ is the Scale step; M represents any causal or padding mask applied before Softmax.

10.6 Mask: Preventing "Peeking" at the Future

10.6.1 Why Masking Is Needed

Causal mask blocks future positions with -inf

In a GPT-style autoregressive model, predicting the next token must not use future tokens. Q @ K computes raw scores for all position pairs, so a causal mask must block the future pairs. Other architectures may also use padding or structural masks.

Think of it this way: if a model is training on “The luthier carried the theorbo and repaired it,” it must not, when processing “repaired,” be able to see the later token “it.”

10.6.2 How the Mask Works

The solution is a triangular mask that fills future positions with negative infinity:

Before mask:                     After mask:
[0.3, 0.2, 0.1, 0.4]     →     [0.3, -inf, -inf, -inf]
[0.2, 0.5, 0.2, 0.1]     →     [0.2, 0.5,  -inf, -inf]
[0.1, 0.3, 0.4, 0.2]     →     [0.1, 0.3,  0.4,  -inf]
[0.2, 0.1, 0.3, 0.4]     →     [0.2, 0.1,  0.3,  0.4 ]

The upper-right triangle (future positions) becomes -inf.

10.6.3 Why -inf?

Because Softmax maps -inf to exactly 0:

Softmax([0.3, -inf, -inf, -inf]) = [1.0, 0.0, 0.0, 0.0]

After Softmax, future positions carry zero weight. The model cannot read from them.

10.7 Softmax: Converting Scores to Weights

10.7.1 The Conversion

Row-wise Softmax turns scaled, masked scores into normalized Attention weights

After masking, we apply Softmax row-by-row:

Before Softmax: [0.32, 1.87, 0.94, -inf]
After Softmax:  [0.132, 0.622, 0.246, 0.000]

10.7.2 What Softmax Does

Normalizes: each row sums to 1
Preserves score ordering: larger scores receive larger weights, with ratios determined exponentially
Handles -inf: maps them to exactly 0

10.7.3 Reading the Pattern

For the first token in a causal sequence, it can only attend to itself, so its row becomes [1.00, 0.00, 0.00, ...]. The second token can attend to positions 0 and 1, so its row might look like [0.32, 0.68, 0.00, ...]. Later tokens have more available positions, but a learned head may still concentrate most weight on only one or two of them.

This is the attention weight matrix: the row-wise coefficients used to mix allowed Value vectors. They are not probabilities that a token is semantically relevant.

10.8 Second Matrix Multiplication: Attention Weights @ V

10.8.1 Weighted Sum

Attention weights multiplied by V to produce the output

With the attention weights computed, the final step is using them to blend the value vectors:

Output = Attention_Weights @ V

10.8.2 Dimension Change

Attention_Weights: [4, 4, 16, 16]   (batch, heads, ctx_len, ctx_len)
V:                 [4, 4, 16, 128]  (batch, heads, key_seq, d_v)
head_output:       [4, 4, 16, 128]  (batch, heads, query_seq, d_v)

The multi-head structure here (4 heads) is the topic of the next chapter.

10.8.3 What This Step Does

Each output position is a weighted sum of the allowed V rows, where the weights come from row-wise Softmax:

output[i] = sum(attention_weight[i, j] × V[j])

If token i puts 70% of its attention on token j and 30% on token k:

output[i] = 0.7 × V[j] + 0.3 × V[k]

The output is a context-aware blend, not a copy of any single token.

10.9 What the Attention Output Means

10.9.1 Output Dimensions

Attention output: each token gets an updated vector representation

After one head computes weights @ V, each position has a per-head output vector:

head_output: [batch_size, ctx_length, d_v] = [4, 16, 128]

Across four heads, those outputs are concatenated to [4, 16, 512] and then projected by W_O back to d_model. Chapter 11 opens up that merge.

10.9.2 The Semantic Shift

Here is the important part. In the first block, X starts from token and position information. In later blocks, X is already contextual. An Attention head adds a new weighted mixture of the allowed projected V rows.

For example, the position for “repaired” can receive projected information from earlier positions such as “luthier” and “theorbo.” The learned weights determine how much each allowed V row contributes.

This is one mechanism by which the model builds contextual representations. It is not a literal proof that a particular head resolved the subject or “understood” the sentence.

10.9.3 The Loop

After the per-head outputs are concatenated and projected through W_O, the Attention branch is combined with the residual stream. The resulting representation then continues to the next sub-layer and is refined block by block.

Each block adds more context into each token's representation.

10.10 The Full Attention Computation

10.10.1 Step-by-Step

Complete Attention flow with shapes at each step

Step 1: Generate combined Q, K, V
        Q = X @ W_Q    [4, 16, 512]
        K = X @ W_K    [4, 16, 512]
        V = X @ W_V    [4, 16, 512]
              ↓
        Reshape into heads (4 heads, d_k = d_v = 128 in this example)
        Q, K: [4, 4, 16, 128]   V: [4, 4, 16, 128]
              ↓
Step 2: Compute compatibility scores (per head)
        scores = Q @ K^T    [4, 4, 16, 16]
              ↓
Step 3: Scale
        scores = scores / sqrt(d_k)    [4, 4, 16, 16]
              ↓
Step 4: Add mask M (causal and/or padding, when required)
        scores = scores + M    [4, 4, 16, 16]
              ↓
Step 5: Softmax
        weights = softmax(scores)    [4, 4, 16, 16]
              ↓
Step 6: Weighted sum (per head)
        head_output = weights @ V    [4, 4, 16, 128]
              ↓
        Concat heads: [4, 16, 512]
        output = concat @ W_O: [4, 16, 512]   (batch, seq, d_model)

10.10.2 PyTorch Implementation

import torch
import torch.nn.functional as F

def attention(Q, K, V, allowed_mask=None):
    """
    Scaled Dot-Product Attention.

    Q: [..., query_len, d_k]
    K: [..., key_len, d_k]
    V: [..., key_len, d_v]
    allowed_mask: boolean tensor broadcastable to
                  [..., query_len, key_len]; True means readable

    Returns:
        output: [..., query_len, d_v]
        attention_weights: [..., query_len, key_len]
    """
    d_k = Q.size(-1)

    # Step 2: Q @ K^T
    scores = torch.matmul(Q, K.transpose(-2, -1))

    # Step 3: Scale
    scores = scores / (d_k ** 0.5)

    # Step 4: Mask
    if allowed_mask is not None:
        # Every query row must retain at least one readable Key.
        scores = scores.masked_fill(~allowed_mask, float('-inf'))

    # Step 5: Softmax
    attention_weights = F.softmax(scores, dim=-1)

    # Step 6: Weighted sum
    output = torch.matmul(attention_weights, V)

    return output, attention_weights

10.11 Deeper Understanding of Q, K, and V

10.11.1 Role Summary

Role	Generated by	Purpose	Used in
Q	X @ W_Q	"What I am looking for"	Q @ K^T
K	X @ W_K	"What I advertise"	Q @ K^T
V	X @ W_V	"What I carry"	weights @ V

10.11.2 Why Separate K from V?

K and V both come from the same input. Why use two different matrices?

To decouple matching from extraction.

K controls which positions get attention
V controls what information flows when attention is given

This separation gives the model flexibility: the features used to decide routing can differ from the features sent along that route. Which Query-Key pairs score highly is learned from data, not fixed by the analogy.

10.11.3 An Example

Consider the hand-built sentence: “The luthier repaired the theorbo.”

When processing “repaired”:

Q("repaired") may look for “who performed this action?”
K("luthier") may signal “I am the action's subject”
V("luthier") carries the luthier position's current projected features

This is an intuition for a pattern a trained head could learn, not a claim that every model or head encodes those exact roles.

10.12 Chapter Summary

10.12.1 Key Concepts

Concept	Shape	Meaning
X	[batch, seq, d_model]	Input tensor
W_Q / W_K	[d_model, h × d_k]	Learnable Query/Key projections
W_V	[d_model, h × d_v]	Learnable Value projection
Q, K	[batch, h, seq, d_k]	Per-head Query and Key tensors
V	[batch, h, seq, d_v]	Per-head Value tensor
Scores	[batch, h, query_seq, key_seq]	Raw compatibility scores
Weights	same as Scores	Row-normalized mixing weights after mask + Softmax
Head output	[batch, h, query_seq, d_v]	Per-head weighted mixtures of V
Block-width output	[batch, seq, d_model]	Concatenated heads projected by W_O

10.12.2 Computation Flow

X → [W_Q, W_K, W_V] → combined Q, K, V → split into heads
              ↓
        Q @ K^T (compatibility scores)
              ↓
        / sqrt(d_k) (scale)
              ↓
        + M (causal/padding mask when needed)
              ↓
        Softmax (normalize)
              ↓
        @ V (per-head weighted sum)
              ↓
        concat heads → W_O → Output

10.12.3 Core Takeaway

Q, K, and V are the three players in Attention. Q asks, K advertises, and their dot product supplies a learned compatibility score. Scaling, masking, and Softmax turn each row into routing weights; those weights blend V. The result is then merged across heads and projected back into the residual stream.

Chapter Checklist

After this chapter, you should be able to:

Explain what Q, K, and V each represent.
Describe how they are generated from the same input X.
Trace the dimension changes through each step of Attention.
Explain the causal mask and why it uses -inf.
Explain why the scale factor $\sqrt{d_k}$ exists.

See You in the Next Chapter

That was the QKV path end to end, with the head axis exposed just enough to keep every shape consistent.

Real Transformers run this whole process from multiple angles simultaneously. Chapter 11 explains Multi-Head Attention: how the model looks at relationships in parallel and combines everything back together.