One-sentence summary: A residual connection leaves an identity path beside each sublayer, so information and gradients do not have to travel only through the transformed branch. Dropout randomly zeroes some activations during training as an optional regularizer.

13.1 Revisiting the Transformer Block

Before diving into residual connections and Dropout, here is a Pre-Norm decoder block in its most compact form:

u = X + Dropout(Attention(LayerNorm(X)))
y = u + Dropout(FFN(LayerNorm(u)))

Attention and the FFN each have a residual path, so there are two additions. The two Dropout calls are a common training recipe, not a law of the Transformer architecture. Some models also apply Dropout to attention weights or input embeddings; others set every Dropout rate to zero. GPT-2-style Pre-Norm models also apply one final normalization after the whole stack, outside these per-block additions.

13.2 Residual Connections: Information's Bypass Lane

13.2.1 The Problem with Deep Networks

Optimization becomes harder as networks grow deeper. Backpropagation multiplies many layer Jacobians; those products can make gradients too small or too large. Deep plain networks can also show a degradation problem in which adding layers raises even the training error. Residual connections do not cure every optimization problem, but they give the optimizer a much simpler route through the stack.

Imagine information flowing from layer 1 through to layer 12:

Layer 1 → Layer 2 → Layer 3 → ... → Layer 12

If every layer offers only a transformed branch, information and gradients must pass through every transformation in sequence. As the stack grows, that long product becomes difficult to optimize.

13.2.2 The Fix: A Bypass Lane

The residual connection idea is simple: let the input skip the layer and be added directly to the output.

Input X ──────────────────────┐
    ↓                         │  (bypass lane)
  Sublayer                    │
    ↓                         │
  Output ←────────────────────┘ + X

The formula:

output = sublayer(X) + X

Instead of only outputting sublayer(X), we add the original input back.

13.2.3 Numeric Example

The diagram uses illustrative values to make the element-wise addition concrete. They are not claimed to be a trace from a published model run.

Attention output (after Dropout):

[4, 16, 512] tensor
First values: -0.07005,  0.09600,  0.03522, ...

Original input X:

[4, 16, 512] tensor
First values:  0.50748, -1.96800,  5.14941, ...

After residual connection:

output = Attention_output + X
       = [-0.07005 + 0.50748,  0.09600 + (-1.96800), ...]
       = [ 0.43743,           -1.87200,              ...]

It is element-wise addition. Nothing exotic.

13.2.4 Why Residual Connections Work

1. Gradient flow

Let $y=x+F(x)$. In vector notation, backpropagation gives:

The identity matrix $I$ is the direct path; $J_F$ is the Jacobian of the transformed branch. This usually improves gradient flow. It does not guarantee that gradients can never vanish or explode: the two terms can partially cancel, and products across many blocks still matter.

2. Identity mapping as a fallback

If a layer does not yet know what to learn, it can default to outputting near-zero:

sublayer(X) ≈ 0
output = 0 + X = X

This effectively makes the layer a no-op. The information passes through unchanged. This is much easier to achieve than learning a perfect identity transformation from scratch.

3. Information preservation

Each block adds its input directly to an update, so the network does not have to reconstruct an identity mapping at every layer. That creates a direct information path, but it does not mean the original token signal remains perfectly recoverable at arbitrary depth; later updates keep changing the residual stream.

13.2.5 Where Residual Connections Sit in the Transformer

First residual connection: after Attention

X → LayerNorm → Attention → Dropout → (+X) → output1

Second residual connection: after FFN

output1 → LayerNorm → FFN → Dropout → (+output1) → output2

13.2.6 PyTorch Implementation

from torch import nn

class TransformerBlock(nn.Module):
    def __init__(self, d_model, num_heads, d_ff, dropout=0.1):
        super().__init__()
        self.attention = MultiHeadAttention(d_model, num_heads)
        self.ffn = FeedForward(d_model, d_ff)
        self.norm1 = nn.LayerNorm(d_model)
        self.norm2 = nn.LayerNorm(d_model)
        self.dropout1 = nn.Dropout(dropout)
        self.dropout2 = nn.Dropout(dropout)

    def forward(self, x, mask=None):
        # First residual connection
        attn_output = self.attention(self.norm1(x), mask)
        x = x + self.dropout1(attn_output)  # residual here

        # Second residual connection
        ffn_output = self.ffn(self.norm2(x))
        x = x + self.dropout2(ffn_output)   # residual here

        return x

This skeleton reuses the MultiHeadAttention and FeedForward classes from earlier chapters. The key lines are x = x + ...; separate Dropout modules simply make configuration and debugging explicit.

13.3 Dropout: Random Masking as Regularization

13.3.1 The Overfitting Problem

Neural networks can overfit: training performance keeps improving while performance on held-out data stalls or worsens.

Think of a model like a student who memorizes every practice problem without understanding the underlying concepts. The student scores 100% on practice tests, then fails on any unfamiliar exam question.

Overfitting is the model's version of that.

13.3.2 Dropout: Random Deactivation

Dropout's idea is surprising in its simplicity: randomly zero elements of an activation tensor during training.

During each forward pass, Dropout samples a new binary mask. No weights are deleted and no unit is permanently switched off; the zeroed activation positions usually change on the next pass.

Normal:  [0.5, 0.3, 0.8, 0.2, 0.6]   ← all displayed activations kept
Dropout: [0.5, 0.3, 0.0, 0.2, 0.6]   ← 0.8 is zeroed out

Which activation positions get dropped? Different ones each time, randomly.

13.3.3 The Intuition

Imagine a small early-music consort rehearsing a viol, recorder, theorbo, and harpsichord part. If the same instrument always carries every cue, the arrangement becomes brittle. If one part is occasionally silent during rehearsal, the other players must learn enough of the structure to keep the piece together.

The analogy has limits, but it captures the intended pressure: Dropout perturbs fixed collaborations and discourages excessive co-adaptation. It is a stochastic regularizer, not a guarantee that every unit becomes independent or that every model generalizes better.

13.3.4 The Math

Let the drop probability be $p$ and the keep probability be $q=1-p$. PyTorch uses inverted Dropout:

Here is one random draw with dropout_rate = 0.1. One of five displayed positions happens to be dropped; small samples do not contain exactly 10% zeros every time.

input  = [0.5, 0.3, 0.8, 0.2, 0.6]
mask   = [1,   1,   0,   1,   1  ]    # 0.8 gets dropped
output = [0.5, 0.3, 0.0, 0.2, 0.6] / 0.9
       = [0.56, 0.33, 0.00, 0.22, 0.67]

During inference:

output = input    # no dropout, pass everything through

13.3.5 Why the Rescaling?

The division by (1 - dropout_rate) preserves the conditional expectation of each element:

If 10% of activations are dropped on average, survivors are multiplied by 1/0.9 ≈ 1.11. Inference then uses the input unchanged. This preserves an element-wise expectation; it does not make every sampled vector's norm identical.

13.3.6 Where Dropout Sits in the Transformer

In the simplified residual-Dropout recipe used in this chapter:

1. After Attention: Attention → Dropout → residual addition
2. After FFN: FFN → Dropout → residual addition

The sublayer update is perturbed before it is added to the residual stream. The original Transformer also applied Dropout to the embedding-plus-position sum, and many implementations separately drop attention weights. Other models use no Dropout at all, so this placement is a recipe, not a universal law.

13.3.7 PyTorch Implementation

import torch
import torch.nn as nn

x = torch.tensor([0.5, 0.3, 0.8, 0.2, 0.6])
dropout = nn.Dropout(p=0.1)

dropout.train()
train_output = dropout(x)  # random mask; survivors divided by 0.9

dropout.eval()
eval_output = dropout(x)   # identical to x

When Dropout is registered inside a model, model.train() and model.eval() switch all child modules recursively. For a standalone layer like this demonstration, switch the layer itself.

13.4 Pre-Norm vs Post-Norm

13.4.1 Two Layouts

One important architectural choice is which side of the residual addition contains the normalization.

Post-Norm (original Transformer, 2017):

u = LayerNorm(X + Dropout(Attention(X)))
y = LayerNorm(u + Dropout(FFN(u)))

LayerNorm comes after the residual addition.

Pre-Norm (the layout used by GPT-2):

u = X + Dropout(Attention(LayerNorm(X)))
y = u + Dropout(FFN(LayerNorm(u)))

LayerNorm comes before each sublayer. GPT-2 also adds a final LayerNorm after the entire stack. LLaMA keeps the Pre-Norm placement but uses RMSNorm instead of LayerNorm.

13.4.2 Why Pre-Norm Is Common

The identity path in Pre-Norm does not pass through a normalization. Theory and practice show that this often gives better-behaved gradients at initialization, makes deep stacks easier to optimize, and can reduce dependence on a delicate learning-rate warmup.

That does not mean Pre-Norm is better in every respect. Post-Norm can train well with suitable initialization, warmup, or architectural changes; Pre-Norm can develop its own residual-stream scale growth. Modern systems also use RMSNorm, parallel branches, and newer normalization layouts. The careful conclusion is that Pre-Norm is a common starting point for decoder-only LLMs, not the only valid answer.

13.5 How Residual Connections and Dropout Work Together

13.5.1 Tracing Data Through the Block

Here is a complete data flow trace for one Transformer block:

Input X  [4, 16, 512]
         ↓
LayerNorm(X)                      # stabilize the input
         ↓
Attention(LayerNorm(X))           # compute context-aware updates
         ↓
Dropout(Attention(...))           # drop some updates (training only)
         ↓
X + Dropout(...)                  # residual: original signal + updates
         ↓
Output1  [4, 16, 512]             # shape preserved

The second sub-block (FFN) follows the same pattern.

13.5.2 Why This Combination Works

Together:

1. LayerNorm stabilizes the input before computation
2. Attention or FFN learns the features
3. If configured, Dropout regularizes the proposed update
4. The residual connection adds that update back to the residual stream

Residual paths and some stabilization strategy are close to architectural necessities for this family. Dropout is not: whether it helps depends on the model, data, and training objective.

13.6 Dropout Rates in Practice

13.6.1 Common Configurations

These rows do not establish a law that larger models always need less Dropout. The rate is part of a training recipe and depends on dataset size and repetition, training duration, task, other regularizers, and optimization behavior. Some large autoregressive pretraining runs choose zero; fine-tuning or smaller datasets may need a new sweep.

13.6.2 Residual Variants

The original Transformer uses plain addition. Some research has explored variations:

Scaled residual branch (schematic):

x = x + alpha * sublayer(x)

Gated residual:

gate = torch.sigmoid(linear(x))
x = x + gate * sublayer(x)   # learn how much to trust the sublayer output

Different methods fix, initialize, or learn alpha, and gating equations also vary. The original Transformer uses plain addition; later architectures need not. These snippets show the design space, not a universal recipe.

13.7 Chapter Summary

13.7.1 Key Concepts

13.7.2 Block Layout

Input X
    ↓
LayerNorm → Attention → Dropout → (+X) → output1
                                    ↑
                              residual here

output1
    ↓
LayerNorm → FFN → Dropout → (+output1) → output2
                                  ↑
                            residual here

13.7.3 Core Takeaway

A residual block learns what to add to the current residual stream instead of rewriting the whole representation from scratch. Its identity path usually makes deep optimization easier. Dropout has a different job: when the training recipe calls for it, a random activation mask adds regularization. Residuals are fundamental to this architecture; Dropout is optional.

Chapter Checklist

After this chapter, you should be able to:

• Explain why residual connections usually improve gradient flow, and why that is not an absolute guarantee.
• Describe the identity-mapping fallback that residual connections enable.
• Explain inverted Dropout's random mask, scaling, and training/inference behavior.
• State where residual connections and Dropout sit inside a Transformer block.
• Distinguish Pre-Norm from Post-Norm and explain why Pre-Norm is preferred for modern LLMs.

See You in the Next Chapter

Each residual addition uses the current sublayer input, not the original token embedding forever. Before the first block, however, the residual stream does begin with a combination of the token embedding and positional encoding.

Chapter 14 asks a question that seems obvious but turns out to be subtle: why do we combine these two signals by adding them, rather than concatenating them?