One-sentence summary: Token embeddings provide content features; positional encodings provide order cues. The original Transformer adds them to keep d_model fixed and avoid an extra projection. The result is a useful compact superposition, not a mathematically lossless package.

14.1 Two Inputs, One Step Before Everything Else

In Chapters 4 and 5, we covered the two components that prepare input for the Transformer:

This chapter goes one level deeper: why are these two signals combined by addition rather than concatenation?

The answer involves a few geometric intuitions, a parameter-count argument, and one honest acknowledgment that neural networks are not spreadsheets.

14.2 The Nature of Each Signal

14.2.1 Token Embeddings: Semantic Information

Recall the embedding lookup table from Chapter 4:

Each token has a trainable row in a table of shape [vocab_size, d_model]. Training makes these vectors carry lexical and statistical features that we loosely call semantics. Nearest-neighbor relations vary by model, so treat the following as intuition rather than a guaranteed geometry:

• "theremin" and "ondes Martenot" may share features as early electronic instruments
• "theremin" and "turnip" should have fewer such features

The embedding answers: what does this token mean?

14.2.2 Positional Encodings: Location Information

The sinusoidal scheme from Chapter 5 computes a position-dependent vector for each sequence index:

• Position 0 has its own vector
• Position 1 has a different vector
• Position 15 has yet another

The positional encoding answers: where in the sequence does this token appear?

14.2.3 Why Both Matter

Consider these two sentences:

"The luthier tuned the theorbo."
"The theorbo tuned the luthier."

Same tokens. Different order. Completely different meaning.

The Transformer needs both signals simultaneously:

1. What each token is (Embedding)
2. Where each token sits (Positional Encoding)

The question is how to combine them.

14.3 Addition vs Concatenation

14.3.1 Two Options

There are two intuitive ways to combine two vectors:

Option 1: Concatenation

input = [Embedding; Positional Encoding]
resulting dimension: d_model + d_model = 2 × d_model

Option 2: Addition

input = Embedding + Positional Encoding
resulting dimension: d_model (unchanged)

The Transformer uses addition. Why?

14.3.2 The Case Against Concatenation

Concatenation doubles the vector width.

If d_model = 512, concatenation produces a 1024-dimensional vector. There are then two honest choices:

• Keep the downstream network 1024 wide. Activations double in width; square projections whose input and output both follow model width use about four times the parameters and matrix-multiply work.
• Project 1024 back to 512. This is a valid architecture, and a learned fusion can sometimes be useful, but it adds a 1024 × 512 projection.

Concatenation is therefore not mathematically wrong, and its cost is not one universal multiplier for every tensor. Addition is simply the cheaper interface: no new projection and no wider Transformer stack.

Addition keeps the contract simple:

[d_model] + [d_model] → [d_model]

Everything downstream sees the same shape it expected.

14.3.3 The Case For Addition

1. Dimensions do not grow

The model width d_model stays fixed from the embedding layer through the entire Transformer stack. Every block, every projection, every norm layer operates on the same shape. This consistency is architecturally clean.

2. Both signals influence the result

Element-wise addition superimposes the two signals in one vector.

A concrete example:

embedding = [0.5,  0.3, -0.2,  0.8, ...]  # semantic signal
position  = [0.1,  0.0,  0.1, -0.1, ...]  # position signal
combined  = [0.6,  0.3, -0.1,  0.7, ...]  # influenced by both

This is not a lossless packing scheme. Many different (embedding, position) pairs can produce the same sum, so the two inputs cannot generally be recovered from combined alone. Recovery is not the training objective; useful prediction is.

3. The system trains around this interface

Token embeddings and Attention projections — plus position embeddings when they are learned — adapt together. They need not reconstruct two pristine source vectors. They only need to turn the shared representation into useful attention patterns and predictions.

14.3.4 An Analogy

Picture a Victorian hydrographic chart. Coastlines and depth soundings are printed onto the same sheet instead of widening the paper and placing two maps side by side. A navigator reads the joint pattern: a shoal matters precisely because of where it sits relative to the coast.

The finished chart does not let you reconstruct the original printing plates exactly. That is fine; the point is to navigate from the overlaid information. Addition plays a similar role here.

14.4 A Concrete Calculation

14.4.1 Step by Step

The diagram traces a one-dimensional toy calculation. It uses plain SGD to keep the arithmetic visible; real training commonly uses optimizers such as AdamW and updates many parameters at once.

Before training (forward pass):

embedding_value   = 0.9    # one dimension of the token's embedding
positional_value  = 0.1    # same dimension in the position vector

combined_value = 0.9 + 0.1 = 1.0

During training (suppose SGD updates this embedding value):

new_embedding_value = old_embedding_value - lr * gradient
                    = 0.9 - 0.1 * (-0.4)
                    = 0.9 + 0.04
                    = 0.94

Next forward pass:

new_combined_value = new_embedding_value + positional_value
                   = 0.94 + 0.1
                   = 1.04

14.4.2 Key Observations

From this trace, three things stand out:

1. Token embeddings are trainable — they update via backpropagation at each training step
2. The positional value here is fixed — that is true for the original Transformer's sinusoidal scheme; a learned position embedding would receive gradients too
3. Addition happens every forward pass — it is not a one-time preprocessing step

14.5 Why This Design Is Sound

14.5.1 Start With One Linear Projection

Let the token embedding be E, the position vector be P, and their sum be Z:

Z = E + P
Q = Z @ Wq = E @ Wq + P @ Wq
K = Z @ Wk = E @ Wk + P @ Wk

This follows from the distributive law, not from an orthogonality assumption. During training, Wq and Wk adapt so that the mixture is useful.

14.5.2 Attention Sees Four Kinds of Interaction

Expanding QKᵀ gives four terms:

(E Wq)(E Wk)ᵀ   # content–content
(E Wq)(P Wk)ᵀ   # content–position
(P Wq)(E Wk)ᵀ   # position–content
(P Wq)(P Wk)ᵀ   # position–position

This is more precise than saying that content occupies one set of dimensions and position an orthogonal set. Attention can use the relationship between thereminist and theremin, their order and distance, and the cross-terms between identity and location.

14.5.3 Exact Separation Is Not Required

The map (E, P) → E + P is many-to-one. Attention cannot generally recover a unique E and P from the sum. A language model does not need to reconstruct its two input tables, however; it needs enough jointly trained signal to predict the next token.

14.6 Variants: Different Positional Encoding Methods

14.6.1 Original Method: Fixed Sinusoidal Encoding

input = Embedding(token_ids) + PositionalEncoding(positions)

The original 2017 Transformer used fixed sinusoidal functions. No positional parameters are learned — the encoding is computed deterministically from the position index.

14.6.2 Learned Positional Embeddings

GPT-1, GPT-2, and GPT-3 use learned absolute position embeddings. That does not imply that every later model with “GPT” in its name, or every decoder-only Transformer, uses the same method.

import torch
from torch import nn

class TransformerInput(nn.Module):
    def __init__(self, vocab_size, d_model, max_len, dropout=0.1):
        super().__init__()
        self.token_embedding = nn.Embedding(vocab_size, d_model)
        self.position_embedding = nn.Embedding(max_len, d_model)   # learned
        self.dropout = nn.Dropout(dropout)

    def forward(self, x):
        # x: [batch_size, seq_len] token IDs
        if x.size(1) > self.position_embedding.num_embeddings:
            raise ValueError("sequence length exceeds max_len")
        token_emb = self.token_embedding(x)        # [batch, seq, d_model]

        positions = torch.arange(x.size(1), device=x.device)
        pos_emb = self.position_embedding(positions)   # [seq, d_model]

        combined = token_emb + pos_emb             # [batch, seq, d_model]
        return self.dropout(combined)

Learned position embeddings can fit positional patterns in the training data rather than using hand-designed frequencies. Their lookup table also has a native maximum index; extending it usually requires resizing and further training or another extension method. This example deliberately follows the learned-absolute-position convention and does not mix in the original Transformer's separate √d_model token-embedding scale.

14.6.3 RoPE: Rotary Position Embedding

More recent models use RoPE (Rotary Position Embedding), which takes a different approach. Instead of adding a position vector to the token embedding, RoPE applies a rotation to the Q and K vectors based on position:

Q_rotated = rotate(Q, position)
K_rotated = rotate(K, position)

RoPE properties:

• Absolute indices determine the rotation angles, while the rotated Q·K explicitly depends on relative displacement
• It does not add another position vector to the residual stream
• It is used by named model families including LLaMA, GPT-NeoX, and Mistral

RoPE does not guarantee cost-free extrapolation to arbitrary lengths. Going beyond the trained context often requires position interpolation, frequency scaling, or additional training. We will revisit those details in Chapter 25.

14.6.4 Comparing Approaches

14.7 Dimension Tracking

Input token_ids: [4, 16]           # 4 sequences, 16 tokens each

Token Embedding:
  lookup: token_embedding([4, 16])
  output: [4, 16, 512]             # each token → 512-dim vector

Position Embedding:
  positions: [0, 1, 2, ..., 15]
  lookup: position_embedding([16])
  output: [16, 512]                # each position → 512-dim vector
  broadcast: [4, 16, 512]          # extend across batch

Addition:
  [4, 16, 512] + [4, 16, 512] = [4, 16, 512]

Final output: [4, 16, 512]         # semantic + positional, same shape

The shape never changes. Every component downstream sees [batch, seq, d_model].

14.8 Common Questions

14.8.1 Does Addition Lose Information?

It loses unique invertibility in the mathematical sense. The sum alone does not determine a unique E and P.

That is not the model's objective. Token representations, position signals, and downstream projections are designed or trained around the sum, and the original Transformer experiments show that this compact interface can work well on the target task.

14.8.2 Why Does Relative Scale Matter?

If the positional encoding values are much larger than the token embedding values, they overwhelm the semantic signal:

embedding = [0.5,  0.3, -0.2]   # semantic
position  = [10,   20,  -15]    # positional — too large!
combined  = [10.5, 20.3, -15.2] # mostly position, semantics drowned out

If the scales are extremely mismatched, the larger term can dominate the initial representation. There is no universal law that position encodings “must be small,” though. Sinusoids lie in [-1, 1], while the original Transformer also multiplies token embeddings by √d_model. Learned position-embedding scales depend on initialization and training.

14.8.3 Learned vs Fixed: Which Is Better?

There is no universal winner. In the original Transformer paper's translation experiment, learned and sinusoidal positions produced nearly identical results. The right choice depends on architecture, training length, and long-context goals.

14.9 Chapter Summary

14.9.1 Key Concepts

14.9.2 Why Addition, Not Concatenation

1. Dimension stability: d_model stays fixed throughout the architecture
2. No extra fusion projection: same-width vectors enter the residual stream directly
3. Interactions are learnable: Attention can use content, position, and their cross-terms

14.9.3 Core Takeaway

Token embeddings provide “what” clues and position encodings provide “where” clues. The original Transformer adds them inside one d_model. The sum is not reversible compression and does not require an unproven orthogonality story; it is a parameter-free, shape-preserving interface from which Attention can learn several useful interactions.

Chapter Checklist

After this chapter, you should be able to:

• Explain what token embeddings and positional encodings each represent.
• Explain why concatenation increases model width and why that is a problem.
• Explain why addition is useful even though it is not uniquely reversible.
• Describe how Attention uses the combined signal.
• Name at least three positional encoding variants and their tradeoffs.

See You in the Next Chapter

We now have all the pieces:

• Token embeddings (Chapter 4)
• Positional encodings (Chapter 5, revisited here)
• Attention with Q, K, and V (Chapters 9–12)
• Residual connections and Dropout (Chapter 13)
• The addition operation that combines content and position (this chapter)

Chapter 15 assembles these into a complete forward pass — tracing a sequence of raw text all the way through to output probabilities, dimension by dimension.