One-sentence summary: Text becomes token IDs, then the sum of token and position embeddings. Twelve Pre-Norm blocks transform that residual stream before a tied output projection produces vocabulary logits. Softmax, greedy decoding, or sampling can then choose the next token. Following those shapes is the clearest way to understand one GPT-2 computation.

15.1 The Big Picture: Decoder-Only Architecture

15.1.1 What Does “Decoder-Only” Remove?

GPT-1, GPT-2, GPT-3, and LLaMA are autoregressive Decoder-Only Transformers. This does not mean taking the original Transformer decoder unchanged. That decoder also contained cross-attention over encoder output. A standalone language model has no encoder and therefore no cross-attention sublayer. What remains is causal self-attention, a position-wise MLP, residual connections, and normalization.

This chapter uses GPT-2 Small as one consistent reference:

• d_model = 768
• n_layers = 12
• n_heads = 12, so d_head = 64
• d_ff = 3072
• vocab_size = 50,257
• learned absolute positions up to 1,024

15.1.2 GPT-1 and GPT-2 Place LayerNorm Differently

Both models use causal self-attention, but their normalization order differs:

GPT-2 also changed initialization, vocabulary, and context length. LayerNorm placement is an important difference, not the only difference. We choose GPT-2 here simply to keep the entire trace concrete.

15.2 Steps 1–3: Prepare the Input

15.2.1 Step 1: Tokenization

Instead of another tutorial cat, keep the book's more distinctive electronic-music example:

Input: "Clara Rockmore played the"

Measured with GPT-2's own byte-level BPE encoding:

Token IDs: [2601, 3301, 4631, 3549, 2826, 262]
Sequence length T = 6

The pieces decode to Cl, ara, Rock, more, played, and the. Spaces can belong to a token, and a word can span several tokens. The model receives a token sequence, not an array of words.

15.2.2 Step 2: Token Embeddings

Each ID selects one row from the learned matrix E ∈ R^(50257×768):

token_ids                         [B, 6]
token_embedding(token_ids)       [B, 6, 768]

These rows are context-free learned parameters. Training can give them lexical and statistical structure, but one row is not the model's complete “understanding” of a token. Contextual information forms in the blocks that follow.

15.2.3 Step 3: Add Position Embeddings

GPT-2 uses a learned absolute-position table P ∈ R^(1024×768):

positions                        [0, 1, ..., 5]
position_embedding(positions)    [6, 768]

X = token_embedding + position_embedding
X                                [B, 6, 768]

The position vectors broadcast across the batch. GPT-2 can also apply embedding dropout to the sum during training; that dropout is disabled in evaluation mode.

As Chapter 14 established, addition makes both signals affect X, but it is not a uniquely reversible packing scheme. Models using RoPE or ALiBi inject position elsewhere and do not follow this exact input operation.

15.3 Steps 4–6: Inside One Transformer Block

One GPT-2 block fits into two equations:

U = X + Dropout(MHA(LN₁(X)))
Y = U + Dropout(MLP(LN₂(U)))

Dropout is active only in training mode, with rates controlled by configuration. The important distinction is this: the block interface stays [B, T, 768], but internal dimensions do change.

15.3.1 Step 4: Masked Multi-Head Attention

Normalize X, then form Q, K, and V. Written with the head axis exposed:

LN₁(X)                            [B, 6, 768]
Q, K, V                           [B, 12, 6, 64]
Q @ Kᵀ                            [B, 12, 6, 6]

Each head computes:

scores = Q @ Kᵀ / √64 + causal_mask
weights = Softmax(scores, dim=-1)
head_output = weights @ V         [B, 12, 6, 64]

The Q–K dot product is a learned compatibility score. It is neither cosine similarity nor a probability that two words are semantically related. The causal mask adds -∞ at future positions, so their weights become 0 after Softmax. A position can still attend to itself.

The twelve head outputs are concatenated back to [B, 6, 768], mixed through W_O, and only then added to the residual stream:

concat(heads)                     [B, 6, 768]
attention_delta = concat @ W_O    [B, 6, 768]
U = X + attention_delta           [B, 6, 768]

15.3.2 Step 5: The First Residual Path

The identity path usually makes a deep network easier to optimize and lets a sublayer temporarily contribute an update near zero. It does not guarantee that gradients can never vanish or explode.

The bypass carries the current residual stream X, not an untouched copy of the original token embedding all the way through the model.

15.3.3 Step 6: Position-Wise MLP and the Second Residual Path

GPT-2 applies the same MLP independently at every position:

LN₂(U)                            [B, 6, 768]
Linear                            768 → 3072
GELU
Linear                            3072 → 768
Y = U + mlp_delta                [B, 6, 768]

The MLP really does widen to 3,072 internally. “Dimensions never change in a block” is therefore wrong; the residual-stream interface is what remains fixed.

In GPT-2 Small, MLP weights make up about 45.5% of all parameters. Research has linked MLPs to factual associations and feature transformations, but knowledge is not stored in MLPs alone. Embeddings, attention, and other layers also participate.

15.4 Step 7: Stack 12 Blocks, Then Apply Final LayerNorm

X₀ [B, 6, 768]
  → Block 1 → Block 2 → ... → Block 12
X₁₂ [B, 6, 768]
  → Final LayerNorm
H [B, 6, 768]

Every layer owns separate parameters; the layers merely share a structure and shape contract. Some models show local, syntactic, or abstract tendencies at different depths, but there is no hard-coded curriculum in which early layers must do syntax and late layers must do reasoning.

15.5 Step 8: From Hidden States to Vocabulary Logits

GPT-2 ties the output projection to its token-embedding weights. If the embedding table is E [50257, 768]:

H                               [B, 6, 768]
logits = H @ Eᵀ                [B, 6, 50,257]

Logit i is the dot product between the current hidden state and E[i]. It is an unnormalized compatibility score, affected by direction and magnitude. It is not yet a semantic-similarity probability.

15.5.1 Training Uses All Positions

Targets are the same sequence shifted by one token. Cross-entropy implementations normally consume logits directly:

input : token₀, token₁, ..., token₄
target: token₁, token₂, ..., token₅

loss = CrossEntropy(logits[:, :-1, :], targets[:, 1:])

Numerically stable cross-entropy performs log_softmax internally. Passing already-normalized probabilities would be the wrong contract for APIs such as PyTorch's CrossEntropyLoss.

15.5.2 Generation Usually Uses the Final Position

next_logits = logits[:, -1, :]    [B, 50,257]

• Greedy decoding takes argmax(next_logits).
• Sampling transforms logits with temperature, Softmax, top-k/top-p, or related controls and draws a token.

So “the highest-probability token is the output” describes greedy decoding only. Sampling may choose another candidate.

15.6 Complete Shape Trace

Input text                          B strings
GPT-2 BPE                          [B, 6]
Token embedding                    [B, 6, 768]
+ position embedding               [B, 6, 768]

Residual stream at each block:     [B, 6, 768]
  Q/K/V, split by head             [B, 12, 6, 64]
  Attention scores                 [B, 12, 6, 6]
  MLP hidden activations           [B, 6, 3072]

After 12 blocks                    [B, 6, 768]
Final LayerNorm                    [B, 6, 768]
Vocabulary logits                  [B, 6, 50,257]

Final position during generation  [B, 50,257]
Selected or sampled token ID       [B]

Now “the shape stays fixed” can be stated precisely: hidden width is 768 at each block's entrance and exit. Head splits, T×T attention matrices, and the wider MLP have their own internal shapes.

15.7 GPT-2 Small Parameter Count

Directly enumerating weights, biases, embeddings, and LayerNorm parameters from the public configuration gives:

The total is about 124.4M. The GPT-2 paper and early descriptions called Small “117M.” OpenAI later added a note to the official repository explaining that the original parameter counts were wrong. That is why the same checkpoint appears under the historical 117M label and a direct count near 124M.

15.8 Backpropagation During Training

Keep these four stages separate:

logits = model(input_ids)          # 1. forward: parameters unchanged
loss = cross_entropy(logits, y)    # 2. compute scalar loss
loss.backward()                    # 3. compute/accumulate gradients; parameters unchanged
optimizer.step()                   # 4. apply a parameter update
optimizer.zero_grad()              # clear gradients for the next step

When the output head and token embeddings are tied, they are one parameter, so gradients from both uses accumulate into the same table. Backpropagation also reaches the final LayerNorm, all twelve blocks, and position embeddings; the optimizer then applies their updates.

15.9 Chapter Summary

15.9.1 Eight Stages

15.9.2 Core Insight

A Transformer does not keep one shape everywhere inside a block. The precise statement is that the residual stream has a fixed entrance and exit width, while heads, T×T attention matrices, and the wider MLP use internal shapes. Final hidden states become vocabulary logits; training computes next-token loss across positions, while generation usually uses the final position to choose one new token.

Chapter Checklist

After this chapter, you should be able to:

• Trace text all the way to vocabulary logits.
• Distinguish residual-stream shapes from internal block shapes.
• Write the two equations of a Pre-Norm block.
• Explain the separate roles of the causal mask, logits, Softmax, and decoding.
• Explain why backward and the optimizer update are not the same step.
• Estimate GPT-2 Small's parameter count from its configuration.

Code Implementation

Part 5 turns this trace into code:

• Chapter 18: model.py — model definition
• Chapter 19: train.py — training loop
• Chapter 20: inference.py — inference logic

See You in the Next Chapter

The forward function can be the same while training and generation call it very differently. Training predicts many positions in parallel; generation feeds each new token back into the next call. Chapter 16 separates those modes and prepares the ground for KV caching.