Tokenization: From Text to Model Input

Computers execute floating-point matrix multiplications (y = Wx + b), which cannot directly ingest variable-length alphanumeric strings like "Hello world". Tokenization acts as the compilation layer: it segments the raw Unicode character stream into discrete atomic pieces (tokens) and maps each piece bijectively to an integer index tᵢ ∈ 0, 1, ṡ, |V|-1. These indices serve as memory addresses for retrieving continuous embedding vectors from the model's weight table.

In OpenAI's `tiktoken` (used in GPT-4o) and Hugging Face's `tokenizers` library (used in Llama-3), tokenization executes in Rust at millions of characters per second before handing tensors off to GPU CUDA kernels.

Claude Shannon's 1948 landmark paper A Mathematical Theory of Communication first treated written text as sequences of discrete symbols generated by stochastic Markov processes. Early computational linguistics attempted full dictionary parsing, but modern deep learning rediscovered Shannon's statistical symbol view to feed neural networks fixed integer IDs.

Mnemonic: "Strings to Slots to Weights."

You cannot multiply strings; text must turn into integer slot IDs before weights can multiply them.

In modern transformer architectures, the vocabulary V is intentionally diverse. A single token can be a full frequent word ("the"), a subword morpheme ("ing"), punctuation ("..."), a whitespace-prefixed word (" world", encoded with a marker like Ġworld), a chunk of digits ("2026"), or a fallback raw byte (<0xFF>). By treating whitespace and punctuation as first-class token components, tokenizers preserve lossless reconstruction without relying on language-specific delimiter assumptions.

In GPT-4's `cl100k_base` vocabulary, "Apple" (capitalized, no space), " apple" (leading space), and "apple" (lowercase, no space) are assigned three distinct token IDs (26732, 17180, 24508) because leading spaces are bound directly into the tokens.

In 2018, Taku Kudo at Google introduced SentencePiece, demonstrating that treating whitespace as an ordinary symbol (_) rather than a hard boundary enabled unified multilingual tokenization without language-specific segmenters like Moses or MeCab.

Mental rule: "Tokens absorb their surroundings."

A token isn't just letters; it absorbs leading spaces, capitalization, and punctuation into one atomic symbol.

Word-level models require massive vocabularies (|V| > 500,000) to cover a natural language dictionary, which causes the input/output projection matrices (|V| × d) to consume billions of parameters while still failing catastrophically on out-of-vocabulary (OOV) slang, typos, and names. Character-level models have a tiny vocabulary (|V| ≈ 256), eliminating OOV errors, but a 1,000-word essay explodes into 5,000+ tokens. Because standard Transformer self-attention computes an L × L attention matrix, quadratic scaling O(L²) makes character models computationally unaffordable for long contexts. Subwords provide the optimal Pareto point.

Llama-3 expanded its vocabulary from 32,000 (Llama-2) to 128,256 tokens. This reduced sequence length L by roughly 15% across multilingual text, allowing Llama-3 to generate more text per GPU second despite the larger embedding matrix.

Early neural machine translation models (Bahdanau et al., 2014) used fixed 30,000-word vocabularies and mapped every unknown word to an [UNK] token, producing translation outputs riddled with blanks. In 2015, Rico Sennrich realized that subwords could completely solve the open-vocabulary problem without expanding L to character lengths.

Mnemonic: "Words choke the dictionary, characters choke the attention."

Words make the embedding table explode (|V| · d); characters make the context window explode (L²). Subwords sit comfortably in between.

Natural languages form new words through morphology: roots combine with prefixes and suffixes. A model does not need distinct vocabulary entries for "play", "playing", "playful", "replay", and "unplayable". By keeping the root "play" and common affixes ("un", "re", "ing", "ful", "able"), a vocabulary of size M + N represents M × N combinations. Even rare technical words or neologisms (e.g., "cryptozoologist") are segmented into known pieces ("crypto", "zoo", "logist"), allowing transformer attention to synthesize meaning from familiar constituent parts.

When biomedical LLMs encounter newly synthesized chemical compounds or gene sequences (e.g., "CRISPR-Cas9"), tokenizers break them into subword fragments rather than collapsing to an [UNK] token, preserving downstream reasoning capability in models like Med-PaLM and BioGPT.

Zellig Harris in 1955 proposed that morpheme boundaries could be discovered computationally by counting phoneme and letter transition branching factors. Sennrich, Haddow, and Birch (2016) adapted this linguistic insight into modern NLP by applying data-driven subword segmentation to neural machine translation.

Mental hook: "Lego Bricks of Language."

Instead of molding a solid plastic toy for every possible castle, subwords give you modular Lego blocks that snap together to build any word imaginable.

BPE training is an unsupervised greedy compression routine. The corpus is first initialized with characters split by spaces (often appended with a boundary marker </w>). At each iteration step:

1. The algorithm scans all adjacent token pairs (p, q) across the tokenized corpus.
2. It identifies the pair (p^, q^) with the maximal co-occurrence frequency: (p^, q^) = argmax Count(p, q).
3. It merges every occurrence of (p^, q^) into a single new subword token p^q^ and appends this merge rule to a priority rulebook.
4. The process repeats until the predefined vocabulary capacity |V| is saturated.

Hugging Face's `tokenizers` crate optimizes this counting step using parallel hash maps and indexed inverted lists, compiling a 100,000-token BPE vocabulary over hundreds of gigabytes of raw text in minutes.

Philip Gage invented Byte Pair Encoding in 1994 as a lightweight heuristic algorithm for lossless data compression in The C Users Journal. Twenty-two years later, in 2016, Rico Sennrich realized Gage's compression heuristic was the exact data-driven compression needed to segment text into open-vocabulary subwords for machine translation.

Mnemonic: "Count, Merge, Repeat."

Count adjacent pairs; merge the winner; repeat until your vocabulary budget runs out.

Once the text is split into a sequence of token IDs [t₁, t₂, ṡ, t_N], each integer tᵢ indexes row tᵢ of a trainable embedding matrix E ∈ R^(|V| × d). Mathematically, this lookup is equivalent to multiplying a one-hot vector tᵢ by E: xᵢ = tᵢ^T E. The resulting vector xᵢ ∈ R^d places the token into a continuous semantic coordinate space. At this input stage, the embedding is static and context-free: "bank" in "river bank" and "bank deposit" begins with the exact same initial vector until self-attention mixes it with contextual clues.

In PyTorch, `torch.nn.Embedding(num_embeddings, embedding_dim)` performs this operation as an O(1) indexed memory access on GPU memory, bypassing actual matrix multiplication to retrieve batch embedding tensors X ∈ R^(B × L × d).

Tomas Mikolov et al. (2013) at Google revolutionized NLP with Word2Vec, showing that discrete words could be mapped to continuous vector spaces where geometric operations captured linguistic semantics (e.g., vec(v)_(King) - vec(v)_(Man) + vec(v)_(Woman) ≈ vec(v)_(Queen)). Transformers adopt this embedding matrix as their foundational input layer.

Mental rule: "Token ID is the key; Embedding is the treasure."

The token ID is just a shelf number (index); the embedding vector on that shelf contains all the learned geometric meaning.