One-sentence summary: computers do not process text directly. Tokenization converts text into token IDs, which are then mapped into vectors.

4.1 Why Tokenization Exists

In the previous chapter, the first step in the Transformer map was:

text -> token IDs

This chapter explains that step.

4.1.1 Computers Need Numbers

A computer does not see a sentence the way we do. It does not know that:

Clara Rockmore played the theremin.

is made of meaningful words. It needs numeric units.

Tokenization is the process that turns text into a sequence of numbers. Each numeric unit is called a token ID.

4.1.2 Where It Sits in the Architecture

Tokenization is the entry point:

raw text -> token IDs -> embeddings -> position -> Transformer blocks

Without tokenization, the rest of the model has nothing to process.

4.2 Two Ways to Tokenize

The simplest idea is to assign a number to every character. Real LLMs usually do something smarter.

4.2.1 Method One: Character IDs

For an English sentence, a naive character-level tokenizer might assign:

C -> 1
l -> 2
a -> 3
r -> 4
space -> 5
R -> 6
o -> 7
c -> 8
...

This is easy to understand. Every character becomes a number.

But it has problems:

1. Too many tokens: one word becomes many characters.
2. Weak semantic units: Clara Rockmore is split into letters even though it is one proper name.
3. Inefficient context use: long text consumes context length quickly.

Character tokenization is not wrong, but it is rarely the best choice for modern LLMs.

4.2.2 Method Two: Byte-Level BPE

OpenAI's tiktoken uses BPE, Byte Pair Encoding. Its byte-level base means arbitrary text can fall back to byte sequences instead of requiring a token for every possible Unicode character.

The idea is:

• common chunks become single tokens
• rare words can still be split into smaller pieces
• the vocabulary stays finite
• the model can handle unseen text

Using OpenAI's cl100k_base tokenizer, this text:

Clara Rockmore played the theremin.

becomes:

[5176, 5169, 9305, 6518, 6476, 279, 9139, 26768, 13]

The token pieces are:

5176  -> "Cl"
5169  -> "ara"
9305  -> " Rock"
6518  -> "more"
6476  -> " played"
279   -> " the"
9139  -> " ther"
26768 -> "emin"
13    -> "."

Notice that spaces often become part of the token. That is normal.

4.2.3 Context Length

Context length is the model's configured sequence budget. In generation APIs, the prompt and generated tokens usually share that budget.

If a model supports 128,000 tokens, that does not mean 128,000 English words. It means 128,000 tokenizer units.

Context length belongs to a specific model and serving configuration, so a product table becomes stale quickly. Two stable paper examples are enough to build intuition:

Modern product models can support much longer windows, but the exact input/output budget must be checked in the current model documentation.

Different languages and writing systems have different token efficiency, but there is no fixed “tokens per Chinese character” ratio. With cl100k_base, 中华人民共和国 is 7 characters and 7 tokens, while 小沈阳江西演唱会邀请了， is 12 characters and 16 tokens. A different sentence or tokenizer can produce another ratio.

LLM APIs meter tokens because tokens are the model's processing units. The consequence is that price and usable context can differ across languages even for text with similar human reading length.

4.3 From Token to Embedding

Token IDs are still not enough. The model must convert each ID into a vector.

This is called Embedding.

4.3.1 Embedding Lookup Table

The model contains a large table:

[vocab_size, d_model]

Where:

• vocab_size is the number of token IDs the tokenizer knows.
• d_model is the vector width used by the model.

For a deliberately round toy example, if:

vocab_size = 100000
d_model = 64

then the embedding table contains:

100000 x 64 = 6,400,000 numbers

Those numbers are trainable parameters.

4.3.2 Lookup Process

Take the sentence:

Clara Rockmore played the theremin.

Tokenization gives:

[5176, 5169, 9305, 6518, 6476, 279, 9139, 26768, 13]

Then the model performs table lookup:

token 5176  -> row 5176  -> vector
token 5169  -> row 5169  -> vector
token 9305  -> row 9305  -> vector
...

The result is a matrix:

[context_length, d_model]

If the sentence has 9 tokens and d_model = 64, the matrix shape is:

[9, 64]

This matrix is the numeric representation sent into the Transformer blocks.

4.3.3 Why Use Vectors?

Why not use token IDs directly?

Because IDs have no geometry. Token ID 791 is not "closer" to token ID 792 in a meaningful semantic way.

Learned vectors can encode useful relationships, but keep two levels separate:

• the embedding table gives each token ID a context-free starting vector
• Transformer blocks turn those starting vectors into contextual hidden states
• a name such as Clara Rockmore is normally composed across multiple tokens and layers

Embedding vectors make language available to matrix math without pretending that token IDs themselves have meaning. Similarity in the initial embedding table can be useful, but it is not a guarantee that every semantically related word or phrase will be nearest neighbors there.

4.4 Try It With tiktoken

You can inspect tokenization with OpenAI's tokenizer library:

import tiktoken

enc = tiktoken.get_encoding("cl100k_base")

text = "Clara Rockmore played the theremin."
tokens = enc.encode(text)

print(f"Token IDs: {tokens}")
print(f"Token count: {len(tokens)}")
print(f"Decoded: {enc.decode(tokens)}")

for token_id in tokens:
    token_bytes = enc.decode_single_token_bytes(token_id)
    token_text = token_bytes.decode("utf-8", errors="replace")
    print(f"{token_id} -> bytes={token_bytes!r}, text={token_text!r}")

Expected shape of the result:

Token IDs: [5176, 5169, 9305, 6518, 6476, 279, 9139, 26768, 13]
Token count: 9
Decoded: Clara Rockmore played the theremin.
5176 -> bytes=b'Cl', text='Cl'
5169 -> bytes=b'ara', text='ara'
...

This small experiment is worth doing. decode_single_token_bytes() is intentional: an individual token can contain only part of a UTF-8 character, so decoding each token directly as standalone text can be lossy.

4.5 Parameter Count in the Embedding Layer

The embedding layer can hold a meaningful number of parameters.

4.5.1 Formula

embedding parameters = vocab_size x d_model

4.5.2 Examples

Embedding is not a tiny pre-processing detail. It is a learned parameter table that matters.

Some architectures tie the input embedding matrix to the output projection. In that case the same weights serve two roles; do not count them twice when calculating the whole model.

4.6 Chapter Summary

4.6.1 Key Concepts

4.6.2 Flow

"Clara Rockmore played the theremin."
        |
        | Tokenization
        v
[5176, 5169, 9305, 6518, ...]
        |
        | Embedding lookup
        v
[context_length, d_model] matrix

4.6.3 Core Takeaway

Tokenization plus embedding is how text enters the Transformer. Tokenization cuts text into model-readable units; embedding turns those units into vectors that can participate in matrix computation.

Chapter Checklist

After this chapter, you should be able to:

• Explain why tokenization is needed.
• Describe the difference between character tokenization and BPE-style tokenization.
• Explain what vocab_size, d_model, and context_length mean.
• Explain why token IDs are converted into vectors.
• Calculate the parameter count of an embedding table.

See You in the Next Chapter

That is it for Tokenization. The next time an API charges you by token, you should know exactly what it is counting.

Now text has become vectors. But one key thing is still missing: position.

The sentences:

The thereminist thanked the inventor.
The inventor thanked the thereminist.

contain nearly the same words but mean different things. Chapter 5 explains how the model knows where each token sits in the sequence.