One-sentence summary: token embeddings carry content but no address; positional encoding supplies an order signal that the standard Transformer can use.

5.1 Why Position Matters

The previous chapter gave us this pipeline:

raw text -> token IDs -> embeddings -> Transformer blocks

We have embeddings now. But there is a quiet problem lurking in them.

5.1.1 A Critical Gap

Consider two sentences from our running example:

the thereminist thanked the inventor.
the inventor thanked the thereminist.

Both sentences contain the same tokens. If we only hand the model token embeddings, the representation of “thereminist” is identical in both sentences — same vector, same numbers. The same is true for “inventor” and “thanked.”

The content vectors alone do not say which arrangement they came from. The two sentences absolutely do not mean the same thing.

The gap is missing position. Standard self-attention has neither recurrence nor convolution to carry sequence order, so the original Transformer explicitly injects an order signal at its input. Token generation can still be sequential and a decoder can still use a causal mask; neither fact puts an absolute or relative address inside the token embedding itself.

5.1.2 Three Things Positional Encoding Fixes

Positional encoding solves three related problems:

1. Absolute location: the model receives a signal that this token is at index 0, 14, or another position in the sequence.

2. Relative distance: “thereminist” and “thanked” are adjacent; two tokens (thanked, the) sit between “thereminist” and “inventor.” That spacing carries grammatical and semantic information.

3. Consistent patterns: a deterministic encoding can be computed at positions not stored in a learned table. This makes extrapolation possible, but it does not guarantee good behavior beyond the lengths seen in training.

5.2 The Simple Idea and Its Limits: Raw Integers

Before explaining what the Transformer actually does, let's look at the most naive approach and why it is a poor representation in this form.

5.2.1 Stacking Integer Position Numbers

The simplest possible idea: give each position a number, then add it to the token embedding at that position.

Here is an example with the tokens ["the", "thereminist", "thanked", "the", "inventor", "."] using a toy d_model = 4:

Embedding vectors (semantic content):

Integer position vectors:

After adding:

5.2.2 Why This Is Awkward

Copying the raw index into every dimension is not mathematically impossible, but it creates two practical problems:

1. Unbounded scale: position 1000 adds 1000 to every dimension. If embeddings are centered near zero, the position signal can swamp their content and make optimization harder. That does not prove gradients must explode; it is simply a poor scale match.

2. A one-direction signal: every dimension receives the same scalar. A network could learn from that linear signal, but it lacks the multi-scale structure that makes different offsets easy to represent.

The original Transformer paper needed something more principled.

5.3 The Transformer's Answer: Sinusoidal Encoding

5.3.1 Core Idea

The original Transformer uses sine and cosine waves at different frequencies to construct positional vectors. The way I remember this: every position gets a barcode made from stacked waves. Low-frequency waves encode broad location; high-frequency waves encode fine-grained distance between nearby positions.

The formula:

even dimensions:  PE(pos, 2i)   = sin(pos / 10000^(2i / d_model))
odd dimensions:   PE(pos, 2i+1) = cos(pos / 10000^(2i / d_model))

Do not panic at the formula. There are only two ideas inside it:

• Even dimensions use sine, odd dimensions use cosine.
• Each pair of dimensions uses a different frequency — controlled by the exponent 2i/d_model.

5.3.2 Wave Visualization

If you plot the position encoding along a sequence, each dimension traces out a wave. A low-frequency dimension changes slowly across positions — like a slow clock. A high-frequency dimension oscillates quickly — like a fast clock. Put many clocks together and each position in the sequence has a unique combination of readings, which is what we need.

5.3.3 Concrete Numbers

Here are the actual values for the first four dimensions, for our six-token sequence (positions 0 through 5):

Note: positions are zero-indexed (pos = 0, 1, 2, ...). At pos = 0, sin(0) = 0 and cos(0) = 1.

For d_model = 4, dimensions 0–1 use sin(pos) and cos(pos), while dimensions 2–3 use sin(pos/100) and cos(pos/100). That is why the last two columns change much more slowly.

Observations:

• All values stay in [-1, 1] — the natural range of sine and cosine. No value explosion at long positions.
• Within this example and practical context ranges, positions receive distinguishable fingerprints.
• The patterns change smoothly — nearby positions are numerically similar.

5.3.4 Why Sine and Cosine Specifically?

The paper authors chose sin/cos for three reasons:

1. Bounded values: always in [-1, 1]. Position 10,000 does not blow up the numbers.

2. The formula extrapolates: the encoding can be calculated beyond the training length. Whether the trained model can use those unseen positions well is a separate question.

3. A linear relation for fixed offsets: for any fixed k, PE(pos + k) can be obtained from PE(pos) by a rotation that depends only on k. The original paper hypothesized that this would make fixed relative offsets easier to learn.

A formula that continues forever does not guarantee that the trained model will work forever. Schemes such as RoPE and ALiBi put relative position or distance bias more directly into attention, but their usable context still depends on training length, scaling, and model configuration. Chapter 25 compares them. For now, sinusoidal encoding is the clearest place to learn the basic idea.

5.4 Embedding + Position = Input

Now let's see the full addition step.

5.4.1 Vector Addition

Three matrices, one shape each: [seq_len, d_model].

Embedding matrix (semantic content):

the:         [0.62, -0.51,  0.09,  0.85]
thereminist: [0.15,  0.73, -0.38,  0.46]
thanked:     [0.07,  0.31, -0.44,  0.12]
...

Position matrix (location):

pos 0:    [0.00,  1.00,  0.00,  1.00]
pos 1:    [0.84,  0.54,  0.01,  1.00]
pos 2:    [0.91, -0.42,  0.02,  1.00]
...

Input embeddings (their sum):

the (pos 0):         [0.62+0.00, -0.51+1.00, 0.09+0.00, 0.85+1.00]
thereminist (pos 1): [0.15+0.84,  0.73+0.54, -0.38+0.01, 0.46+1.00]
...

The critical observation: the two occurrences of “the” start with the same embedding but receive different position vectors, so their combined inputs differ. Attention can now combine content with order and distinguish “the thereminist thanked the inventor” from “the inventor thanked the thereminist” even though the two sentences share their tokens.

This toy arithmetic shows direct Embedding + Position. The original Transformer first scales token embeddings by sqrt(d_model): Input = sqrt(d_model) × Embedding + PE. Modern architectures vary in the details; the core idea here is to combine content and position without changing the tensor shape.

5.4.2 Geometric Intuition

In a 2D sketch, vector addition follows the parallelogram rule:

embedding vector = [1, 3]   (blue arrow: semantic direction)
position vector  = [2, 1]   (red arrow: positional shift)
input vector     = [3, 4]   (result: diagonal of the parallelogram)

The resulting vector carries both pieces of information, encoded in its direction and magnitude. In 768 or 4096 dimensions, there is far more room for this combined representation to remain coherent.

5.4.3 Relative Distance Matters

For many tasks the model cares about relative distance, not just absolute position. Whether “Clara” comes immediately before “Rockmore” matters. The sinusoidal scheme preserves some of that structure mathematically, and learned position schemes preserve it empirically. Chapter 25 goes into the specifics of each approach.

Position alone does not decide a token's meaning. It lets attention read order and neighborhood, which can then resolve ambiguity. Consider the word bow:

• “Tighten the bow” — bow is the bow used on a string instrument.
• “Bow after the recital” — bow is an action performed for the audience.

The token is identical. Its starting embedding is identical. The position signal tells attention how the surrounding words are arranged; the later contextual hidden state, not position by itself, separates the noun and verb uses.

This is the core reason positional encoding matters in the standard architecture: content embeddings alone carry no address, so order-sensitive relationships need another signal.

5.5 Training: What Learns and What Doesn't

5.5.1 Fixed Encodings vs. Learned Parameters

There is an important asymmetry in the original Transformer:

• Embedding matrix: trainable parameters. Gradients flow back through the lookup and update token vectors. Useful geometric structure can emerge, although no rule says every semantically related token must become a nearest neighbor.

• Position matrix (sinusoidal version): deterministic and fixed. It can be computed or cached from the formula, but it is not an optimizer parameter and receives no parameter update.

During training, the model learns how to interpret the position signal embedded in the vectors — but it does not change the signal itself.

Some models, including BERT, use learned positional embeddings: the position matrix is a parameter just like the token embedding table and is updated by gradient descent. The tradeoff is that learned embeddings often do not generalize beyond the training context length, while sinusoidal ones can in principle.

5.5.2 Where It Sits in the Architecture

The flow in the full model:

raw text
   |
   | tokenization
   v
token IDs
   |
   | embedding lookup
   v
embedding matrix [seq_len, d_model]
   |
   | + positional encoding [seq_len, d_model]
   v
input embeddings [seq_len, d_model]
   |
   | feed into Transformer blocks
   v
   ...

For the additive sinusoidal scheme in this chapter, the position signal is added before the first Transformer block. RoPE-style models differ: they rotate Q and K inside attention instead of adding a position vector to the input.

5.6 Why Add Instead of Concatenate?

This question comes up every time I teach this chapter. The intuition is worth spelling out.

5.6.1 Concatenation vs. Addition

Concatenation:

• Append the position vector after the embedding vector.
• If both parts retain width d_model, the result [embedding | position] is 2 × d_model wide.
• Clean separation of information.
• Downside: subsequent layers must handle wider matrices and more compute, or an extra projection must compress the result back to d_model.

Addition:

• Add embedding and position element-wise.
• Result: same shape [d_model] — no dimension change.
• Downside: the two signals are mixed in the same dimensions.

5.6.2 Why Addition Works

The useful intuition is that high-dimensional spaces are spacious. In 768 or 4096 dimensions, learned projections can interpret the mixed content and position signals together. They may organize partly separate directions, but orthogonal subspaces are not guaranteed by addition itself.

Think of it as two engineers sharing a whiteboard instead of each having their own. It sounds messy, but if you have a large enough whiteboard and organized people, it works fine — and you saved the cost of a second whiteboard.

Empirically, addition works. The architecture is simpler and the parameter count stays the same. That is a good engineering trade.

5.7 Chapter Summary

5.7.1 Key Concepts

5.7.2 Data Flow

Embedding [seq_len, d_model]    <- semantic content
     +
Position  [seq_len, d_model]    <- sinusoidal position encoding
     =
Input     [seq_len, d_model]    <- fed into the first Transformer block

5.7.3 Core Takeaway

Positional encoding fills the address missing from token embeddings. By injecting an order signal, it lets attention combine content, direction, and distance, so “the thereminist thanked the inventor” and “the inventor thanked the thereminist” no longer look like the same arrangement.

Chapter Checklist

After this chapter, you should be able to:

• Explain why token embeddings need a position signal even when the sequence is stored in an ordered tensor.
• Describe sinusoidal encoding as a multi-frequency wave barcode: each position gets a unique combination of sine and cosine values.
• Reproduce the formula PE(pos, 2i) = sin(pos / 10000^(2i/d_model)) and explain the two main ideas it contains.
• Explain why addition is used instead of concatenation, and what the practical trade-off is.
• Distinguish between fixed sinusoidal encodings (original Transformer) and learned positional embeddings (BERT, etc.).

See You in the Next Chapter

That is enough for positional encoding. If you can explain why “the thereminist thanked the inventor” and “the inventor thanked the thereminist” produce different model outputs even though they share every token, you have internalized this chapter.

The input to the Transformer blocks is now complete: semantic information from embeddings, plus location information from positional encoding.

Chapter 6 introduces two small but essential mathematical tools that appear everywhere inside the Transformer: LayerNorm, which keeps numbers in a well-behaved range, and Softmax, which turns raw scores into probability distributions.