One-sentence summary: Attention does not rewrite the embedding table during a forward pass. It mixes V vectors into a temporary representation for the current context. During training, parameters change only after the loss, backpropagation, and an optimizer step.

12.1 Today Is an Easy Chapter

Today is an easy chapter, and it brings our discussion of Attention to a close.

In the previous chapter, we split Multi-Head Attention into separate heads and saw how each head computes independently. Only two questions remain:

How do we merge the results from all heads?
What does the Attention output actually change?

The second question is where people often get confused. I made the same mistake when I first learned this: I treated the following two events as if they were one:

the representation of a token in the current sentence changes;
the embedding parameters stored in the model change.

They are not the same.

The first happens in every forward pass. The second happens only during training, after backpropagation computes gradients and the optimizer applies an update.

Once that boundary is clear, QKV, residual connections, and the training process all fit together.

12.2 Let Us Give A a More Precise Name

For convenience, earlier chapters called the Attention result A. Different sources use A for different things, however: some mean the attention weights, while others mean the output after mixing V.

In this chapter, we separate them:

P = \operatorname{Softmax}\left(\frac{QK^\top}{\sqrt{d_k}} + M\right)

H_{\text{heads}} = PV

Here:

P is the attention-weight tensor;
M is the mask, so blocked positions have weight 0 after Softmax;
H_heads is the output after each head mixes its V vectors.

So Q×K does not immediately produce percentages. The complete sequence is:

QKᵀ produces raw compatibility scores;
divide by √d_k to scale them;
add the mask;
apply Softmax to obtain P, whose rows sum to 1;
use P to take weighted sums of V.

These weights are useful for saying how much each position contributes when V is mixed. They are not a rigorous probability judgment about semantic relationships.

12.3 The Shape of H_heads: Four Dimensions Are Not Scary

Continue with our earlier example:

H_heads: [4, 4, 16, 128] │ │ │ └── output width of each head │ │ └────── Query sequence length │ └────────── number of heads └───────────── batch size

The four dimensions mean:

4 sequences in the batch;
4 heads for each sequence;
each head produces an output for 16 Query tokens;
each token receives a 128-dimensional result from that head.

In ordinary self-attention, Query, Key, and Value have the same sequence length, so P has shape:

[batch, heads, seq_len, seq_len]

It is still better to keep two different names in mind:

[batch, heads, query_len, key_len]

Then cross-attention will not feel strange when the Query and Key lengths differ.

12.4 Concatenate: Merging the Heads

12.4.1 The Merge

Concatenate places the dimensions from all heads back beside one another:

Before: [4, 4, 16, 128] Reorder: [4, 16, 4, 128] After: [4, 16, 512]

In code, we do not usually flatten two arbitrary dimensions. We first put the token and head dimensions in the correct order, then reshape.

Each token had 128 dimensions in each of 4 heads. After the merge, it has one 512-dimensional vector again.

12.4.2 Why Split and Then Merge?

This confused me when I first learned it: why cut a vector into pieces, calculate Attention separately, and then join the pieces again?

Each head has its own slices of the projection parameters, so it can compute Attention in a different representation subspace. Some trained heads may respond strongly to position, syntax, or a particular relationship, but no one assigns those jobs in advance, and every head is not guaranteed to develop a role that can be named neatly.

Merging puts the results from these different subspaces back into one token representation.

12.5 W_O: Learning How to Combine the Merged Results

Concatenation only places the head outputs next to one another. It does not decide how their features should interact. That is the job of the output projection W_O:

H_{\text{attn}} = \operatorname{Concat}(H_1,\ldots,H_h)W_O

In our equal-width example:

Concatenated output: [batch, 16, 512] W_O: [512, 512] H_attn: [batch, 16, 512]

W_O is a trainable parameter. It does not simply average the four heads; it learns how to recombine the features they provide.

If h×d_v is not equal to d_model, the general shape of W_O is:

[h × d_v, d_model]

So W_O is 512×512 only in this particular equal-width example.

12.6 What Do QK and V Do?

12.6.1 QKᵀ Decides How to Mix

For one head:

Q: [query_len, d_k] K: [key_len, d_k] QKᵀ: [query_len, key_len]

Each entry of QKᵀ is a compatibility score between one Query position and one Key position. It becomes an attention weight only after scaling, masking, and Softmax.

12.6.2 V Supplies the Content Being Mixed

V is not the original text, and it is not the embedding vectors left untouched. It is a projection of the current layer input X through W_V:

V = XW_V

Then:

H_{\text{heads}} = PV

For each Query position, the output is a weighted sum of the V vectors.

The shortest way to remember it:

Q and K decide how much to take from which positions;
V supplies the features that are actually taken.

12.7 The Most Common Confusion: Hidden States Are Not the Embedding Table

The distinction among embedding parameters, hidden states for the current sequence, and Attention output

12.7.1 What Is the Embedding Table?

The embedding table E is a model parameter. Given token IDs, it returns the initial vectors:

X_0 = E[\text{token IDs}]

The same token ID retrieves the same base vector. After positional information is added and the vector passes through Transformer blocks, it gradually becomes a representation for this particular context.

12.7.2 What Is a Hidden State?

X_0, X_1, X_2, and so on are intermediate results created during this forward pass. They are usually called hidden states or activations.

Think of them as temporary working notes the model writes while processing the current sequence. Change the sequence, and the same token may have a different hidden state in later layers.

12.7.3 Does Attention Rewrite Its Input?

No.

The Attention branch computes a new tensor H_attn from X. As the next chapter explains, a common residual connection adds the branch output back to the main stream:

X_{\text{next}} = X + \operatorname{AttentionBranch}(X)

This creates a new hidden state. Attention does not rewrite the original X in place, and it does not rewrite the embedding table E during this forward step.

So it is inaccurate to say that PV adjusts the initialized value of the original token. A more precise statement is:

PV creates a temporary, context-dependent representation for this sequence, layer, and head.

12.8 When Do Parameters Actually Change?

The three stages of a training step: forward pass, backpropagation, and optimizer update

Splitting one training step into three stages removes the ambiguity.

12.8.1 Stage One: Forward Pass

The model reads the embedding table E and weights such as W_Q, W_K, W_V, and W_O. It computes hidden states, predictions, and a loss.

This stage creates many activations, but the parameter values have not changed yet.

12.8.2 Stage Two: Backpropagation

Starting from the loss, backpropagation computes a gradient for each trainable parameter:

\nabla_E L,\;\nabla_{W_Q}L,\;\nabla_{W_K}L,\;\nabla_{W_V}L,\;\nabla_{W_O}L

A gradient says which direction a parameter should move and by how much it affects the loss. Computing it is not the parameter update itself.

12.8.3 Stage Three: Optimizer Update

The optimizer uses those gradients to modify the parameters. With the simplest gradient-descent notation:

\theta \leftarrow \theta - \eta\nabla_\theta L

Only after this step do the embedding table and layer weights have new values. The next batch reads the updated parameters.

Inference normally performs only the forward pass: no backward call and no optimizer step. Normal text generation therefore does not rewrite model parameters while you chat. A KV Cache stores intermediate K and V values, but it is a cache for the current inference process, not a set of training parameters.

12.9 A Less Generic Example: Clara Rockmore

Suppose the training corpus contains the name “Clara Rockmore” many times. Rockmore was a theremin virtuoso, so this is a proper name that might occur in a small, specialized corner of the corpus rather than in every language-model tutorial.

The first time the model encounters it:

each input token retrieves a base vector from the embedding table;
Attention and the later network create contextual hidden states for the full sentence;
the model computes a loss from its prediction and target;
backpropagation computes gradients;
the optimizer makes a small update to relevant embeddings and other weights.

When “Clara Rockmore” appears again, the model does use the parameters updated by the previous optimizer step. But that is not because the previous Attention output was written directly into the embedding table. Training completed the entire chain:

forward pass → loss → backpropagation → optimizer update

After many training steps, base embeddings learn information that can be reused across the corpus. The hidden state at each occurrence still changes according to the surrounding tokens.

Remember both levels:

embedding parameters: base representations learned over time and stored in the model;
hidden states: temporary, context-dependent representations computed for the current input.

12.10 Which Values Are Parameters?

Name	Trainable parameter?	When it exists or changes
Embedding table E	Yes	Changes after an optimizer update
W_Q, W_K, W_V, W_O	Yes	Change after an optimizer update
Q, K, V	No	Recomputed on each forward pass
Attention weights P	No	Recomputed on each forward pass
Attention output	No	Recomputed on each forward pass
Hidden state X_l	No	Recomputed on each forward pass
KV Cache	No	Stored and extended during inference

This table is the most important boundary in the chapter.

It is better to phrase this question more carefully.

A common implementation uses one large linear layer to compute the full Q tensor, then reshapes it into heads. That large matrix contains a different column slice for each head:

W_Q = [W_Q^{(1)}\;W_Q^{(2)}\;\cdots\;W_Q^{(h)}]

Therefore:

in storage and computation, the head projections may come from one large W_Q;
in learned parameter values, each head uses a different slice, not the same numbers repeated across all heads.

The same is true for W_K and W_V.

Different Transformer blocks normally have independent Attention parameters. Architectures with cross-layer parameter sharing also exist, but that is not the standard case discussed here.

12.12 Counting Attention Parameters

In the common equal-width form of Multi-Head Attention, ignoring bias terms:

|W_Q|+|W_K|+|W_V|+|W_O| = 4d_{\text{model}}^2

If d_model = 512:

Attention weights per layer = 4 × 512 × 512 = 1,048,576

For 12 layers:

12 × 1,048,576 = 12,582,912

That count covers the Attention projection weights only. It excludes biases, embeddings, FFNs, LayerNorms, and the output layer.

Running more training steps does not create more parameters. Training changes the values of existing parameters, not how many parameters the architecture contains. Model size is determined mainly by vocabulary size, depth, width, FFN dimension, and the architecture itself.

12.13 Connecting the Whole Causal Chain

We can now write the last part of Multi-Head Attention from beginning to end:

Current hidden state X ↓ Project X into Q, K, and V ↓ QKᵀ → scale → mask → Softmax ↓ Multiply weights P by V ↓ Concatenate the head outputs ↓ Apply W_O ↓ Obtain the Attention branch output

During an ordinary forward pass, this chain creates activations only.

During training, it continues:

prediction → loss → backpropagation → optimizer update

Only then do the embedding table and W_Q, W_K, W_V, and W_O actually change.

Chapter Checklist

QKᵀ creates compatibility scores; scaling, masking, and Softmax turn them into attention weights
P decides how to mix, while V supplies the features being mixed
Concatenate restores one token representation, and W_O learns how to recombine the head outputs
An Attention forward pass creates new hidden states; it does not directly rewrite the embedding table
Backpropagation computes gradients; the optimizer updates parameters
Q, K, V, attention weights, and the KV Cache are runtime values, not model parameters
One large projection matrix can hold different parameter slices for different heads

Part 3 Summary

We have now connected the geometry of Attention, QKV, head splitting, merging, and the output projection.

When you see:

\operatorname{Attention}(Q,K,V)=\operatorname{Softmax}\left(\frac{QK^\top}{\sqrt{d_k}}+M\right)V

you should know more than how to calculate it. You should also know which values are activations produced for the current input, which are temporary mixing weights, and exactly when training changes a parameter.

That is what the QKV output really means.

See You in the Next Chapter

Chapter 13 covers residual connections and Dropout.

A residual connection leaves a direct path through the network, making deep models easier to optimize.
Dropout randomly removes some activations during training as a form of regularization.

Whether LayerNorm sits before or after the residual branch also gives us Pre-Norm and Post-Norm. We will look at those next.

That is enough for today. See you in the next chapter.