From Multilingual Text to Model Representations
How does one model turn many writing systems into reusable tokens and context-aware vectors?
How this lesson unfolds.
Begin with How can a single tokenizer ingest Latin alphabets, Devanagari script, and emoji without requiring separate language-specific alphabets? The sequence then connects A shared vocabulary does not force identical pieces and builds toward Where text representation fits in an LLM build.
How can a single tokenizer ingest Latin alphabets, Devanagari script, and emoji without requiring separate language-specific alphabets?
Every written script in human history—from Japanese kanji and Cyrillic glyphs to mathematical operators and emojis—transforms into a sequence of numbers between 0 and 255 under standard UTF-8 rules. By initializing the vocabulary with this exact 256-number palette, a system can digest any text stream on Earth without needing separate language-specific hardware or out-of-vocabulary fallback tags.
- $$c \in \text{Unicode} \xrightarrow{\text{UTF-8}} [b_1, \dots, b_k], \quad b_i \in \{0, 1, \dots, 255\}, \quad 1 \le k \le 4$$
- Learning goalExplain how UTF-8 byte serialization unifies diverse scripts into universal starting vocabularies.
- Helpful foundationsUnicode Code Points · UTF-8 Serialization · Subword Foundations · Attention Is All You Need · A Mathematical Theory of Communication (Shannon, 1948)
- Common trapLLMs do not require per-language alphabets; byte tokenizers reduce all scripts to 256 base bytes.
- Next connectionAnalyze how shared vocabularies allocate subword pieces across distinct writing systems.
More context & mechanics
Human writing systems differ dramatically in script structure, but any digital text can be deterministically encoded into UTF-8 bytes. An English letter 'A' serializes into 1 byte (0x41), a Hindi Devanagari character like 'ह' (U+0939) serializes into 3 bytes (0xE0 0xA4 0xB9), and an emoji like '🙂' (U+1F642) into 4 bytes. By anchoring the bottom layer of the tokenizer to these 256 byte values, every script in existence can be represented without an unknown token [UNK].
Byte-level BPE in LLaMA-3 and GPT-4 avoids language-specific character vocabularies by initializing with 256 base byte tokens. This allows these models to seamlessly ingest arbitrary multilingual text, code, and emoji without out-of-vocabulary exceptions.
Ken Thompson and Rob Pike designed UTF-8 in 1992 on a placemat in a New Jersey diner to unify character encoding for the Plan 9 operating system. Decades later, Radford et al. (2019) adopted byte-level encoding for GPT-2, forever banishing out-of-vocabulary errors from modern language models.
Mnemonic: "256 Bytes Rule the World."
No matter what language is spoken, every script melts down into the same 256 bytes before merging into tokens.
Why does a shared multilingual vocabulary tokenize identical semantic concepts into vastly different numbers of tokens across languages?
If 90% of a model pretraining data is English and 0.1% is Hindi, the algorithm spends thousands of merge slots on long English phrases like "internationalization" (1 token). Meanwhile, the Hindi translation gets fragmented into a dozen tiny subwords and raw byte pieces. Even though both words express the identical semantic concept, the low-resource script consumes twelve times the context length and compute budget.
- $$\operatorname{Fertility}(w) = \frac{\operatorname{Tokens}(w)}{\operatorname{Words}(w)}, \quad \operatorname{Fertility}_{\text{English}} \approx 1.2 \ll \operatorname{Fertility}_{\text{Hindi}} \approx 3.8$$
- Learning goalDefine token fertility and quantify cross-lingual efficiency disparities in vocabulary allocation.
- Helpful foundationsCorpus Frequency Distribution · Subword Segmentation · Token Fertility · Attention Is All You Need · A Mathematical Theory of Communication (Shannon, 1948)
- Common trapMultilingual vocabularies are not equally efficient across languages; high-resource scripts enjoy high compression while low-resource scripts suffer high fertility.
- Next connectionDemonstrate why token boundaries are engineered heuristics rather than natural linguistic truths.
More context & mechanics
In a joint multilingual vocabulary of fixed size (e.g., 32,000 tokens), greedy BPE allocates merge rules to the most frequent character sequences in the training mix. If 85% of the pretraining data is English, English words frequently compress into single tokens (e.g., "mathematics" = 1 token). In contrast, languages representing only 0.5% of data rarely have frequent multi-character pairs, causing their words to fragment into 3 to 6 subword tokens (known as high token fertility). Thus, shared storage does not produce equal segmentation depth.
This phenomenon, often termed the 'token tax', causes non-Latin script users to consume context windows 2 to 4 times faster in ChatGPT or Claude. To combat this, modern models like LLaMA-3 expanded vocabulary from 32k to 128k to explicitly allocate dedicated tokens to Indic, Arabic, and East Asian scripts.
Conneau et al. (2019) introduced XLM-R with a 250,000-token shared SentencePiece vocabulary covering 100 languages. They demonstrated that balancing training sampling probabilities across low-resource languages was critical to prevent the vocabulary from being monopolized by English.
Mental rule: "Corpus share dictates token size."
High-frequency languages get whole words as tokens; low-frequency languages pay a token tax with smaller pieces.
Why is there no single "correct" tokenization for a given sentence?
Feed the phrase "She didn't run" into two different commercial parsers. One splits on apostrophes yielding ["She", " did", "n", "'t", " run"], while another preserves contractions producing ["She", " didn", "'t", " run"]. Neither reflects an absolute linguistic truth; each is simply the deterministic consequence of arbitrary regex whitespace splits, casing rules, and merge priorities.
- $$\mathcal{S}(x; \theta_A) \ne \mathcal{S}(x; \theta_B) \quad \text{for differing objectives } \theta$$
- Learning goalAnalyze how pre-tokenization regex rules and optimization objectives determine arbitrary token boundary placements.
- Helpful foundationsPre-tokenization Regex · Tokenization Objectives · Segmentation Discrepancies · Attention Is All You Need · A Mathematical Theory of Communication (Shannon, 1948)
- Common trapToken boundaries are not objectively correct linguistic morphemes; they are algorithmic artifacts produced by chosen pre-segmentation heuristics.
- Next connectionExamine the distinction between token vocabulary capacity and natural language lexicon size.
More context & mechanics
Consider the string "unplayable". A BPE tokenizer trained on morphologically rich text might segment it into ["un", "play", "able"]. A WordPiece tokenizer optimizing maximum likelihood on Wikipedia might segment it into ["un", "##playable"]. A byte-level tokenizer with a smaller vocabulary might split it into ["un", "pl", "ay", "able"]. None of these boundaries are objectively wrong; each represents an optimal solution to a particular constrained optimization problem defined by corpus frequencies, regex splits, and vocabulary size |V|.
OpenAI migrated from `r50k` (GPT-2/3) to `cl100k_base` (GPT-4) and `o200k_base` (GPT-4o), altering regex split patterns for numbers and punctuation. Consequently, identical prompt strings produce different token sequences, affecting model reasoning on arithmetic and code.
In 2018, Google researchers developing BERT selected WordPiece (maximizing corpus language model likelihood gain), whereas OpenAI selected BPE (maximizing pair co-occurrence count). This created a historical divergence in subword philosophies across modern transformer families.
Mnemonic: "Tokens are Engineered, not Discovered."
Nature didn't create token boundaries; engineers chose them to balance sequence length against vocabulary memory.
Why does an LLM vocabulary size of 100,000 not mean the model knows 100,000 dictionary words?
A standard paper lexicon with 100,000 entries fails the instant it encounters internet slang, medical jargon, or typos. An LLM vocabulary of 100,000 items is fundamentally a set of modular building blocks—common roots, syllables, punctuation marks, and individual bytes. Just as 26 letters can generate every novel sentence ever written, 100,000 subword fragments synthesize an infinite combinatorial variety of ideas.
- $$|\mathcal{V}| = |\mathcal{V}_{\text{words}}| + |\mathcal{V}_{\text{subwords}}| + |\mathcal{V}_{\text{bytes}}| + |\mathcal{V}_{\text{special}}|$$
- Learning goalDifferentiate fixed tokenizer vocabulary capacity from the unbounded combinatorial space of human natural language.
- Helpful foundationsEmbedding Matrix Size · Subword Composition · Dictionary vs Inventory
- Common trapVocabulary capacity measures discrete token IDs, not dictionary words; subwords combine combinatorially to express unbounded lexicons.
- Next connectionDebunk the assumption of constant token-to-character conversion ratios.
More context & mechanics
In an embedding table E ∈ R^(|V| × d), |V| is the number of rows. These rows do not store a static dictionary of complete English words. Instead, they store a mixed bag of building blocks: common full words ("the", "cat"), subword syllables ("ing", "ation"), isolated letters, whitespace symbols, and raw bytes. A word like "tokenization" may consume two token slots ("token" and "ization"). With just 32,000 or 100,000 stored tokens, an LLM possesses an infinite receptive vocabulary because any unknown word can be assembled from smaller pieces.
The embedding table parameter count is exactly |V| × d. In Llama-3-70B (d = 8192, |V| = 128,256), the input embedding table alone contains over 1.05 billion parameters (128256 × 8192 ≈ 1.05 × 10⁹), highlighting why vocabulary size is carefully budgeted.
Early machine translation in the 1990s used exhaustive bilingual dictionaries containing millions of inflected word entries, creating immense storage bottlenecks. The subword revolution showed that a compact vocabulary of 32,000 statistical pieces outperforms a dictionary of 1,000,000 static words.
Mental rule: "Rows in the table, not entries in Webster's."
|V| is just the row count of the embedding matrix, storing modular building blocks that combine into infinite words.
Why is the common heuristic "1 token ≈ 4 characters" merely a rough statistical average rather than a mathematical law?
Examine three vocabulary entries in a single prompt: the piece for a single space spans 1 character; the chunk for "anthropomorphic" spans 15 characters; and a single byte from a 4-byte musical symbol spans only one-quarter of a visual glyph. The standard rule of thumb that one piece equals 4 characters is merely a rough empirical average for English prose, not a physical invariant.
- $$1 \le \operatorname{len}_{\text{chars}}(t) \le 30+, \quad \operatorname{len}_{\text{bytes}}(t) \ge 1$$
- Learning goalDeconstruct the variable character and byte spans of individual tokens across scripts and symbols.
- Helpful foundationsCharacter Length · Byte-Level BPE · Variable-Length Encodings
- Common trapTokens do not have a uniform character width; token-to-character ratios fluctuate widely between single bytes, short morphemes, and entire words.
- Next connectionTrace how static embedding lookups evolve into dynamic contextual representations.
More context & mechanics
The character length of a token is completely fluid:
- Frequent words: " unfortunately" is 13 characters long but forms a single token in modern tokenizers.
- Numbers: A 6-digit number "123456" might be split into two tokens ("123", "456"), giving 3 characters per token.
- Non-Latin scripts: A single Devanagari character 'क्ष' may require 2 to 3 tokens (composed of multiple bytes).
- Punctuation: Single characters like . or , are 1 character per token.
The 4-characters-per-token rule of thumb is merely an empirical aggregate across English prose.
When calculating API costs for OpenAI or Anthropic endpoints, sending 1,000 characters of Python code or JSON might consume 300 to 400 tokens, whereas 1,000 characters of English prose consumes roughly 250 tokens, and 1,000 characters of Korean can consume over 700 tokens.
Early fixed-width encodings like ASCII assumed 1 character = 1 byte = 1 atomic unit. Modern AI broke this one-to-one assumption twice: first with UTF-8 variable-width bytes (1 to 4 bytes per character), and second with BPE variable-width subwords (1 to many characters per token).
Mnemonic: "Tokens Expand and Contract."
Frequent phrases shrink into 1 giant token; rare characters stretch across multiple tiny tokens.
Why is the static input embedding vector only the starting point for a token's representation inside an LLM?
In a static lookup table, the entry for "apple" always retrieves the exact same vector whether reading "apple pie" or "Apple stock". But as that initial vector travels through successive transformer layers, multi-head attention queries surrounding words, dynamically pulling information from "pie" or "stock" to reorient the representation toward fruit orchards or Silicon Valley finance.
- $$\mathbf{h}_i^{(0)} = E[t_i] + \mathbf{p}_i, \quad \mathbf{h}_i^{(l+1)} = \operatorname{TransformerBlock}(\mathbf{h}_1^{(l)}, \dots, \mathbf{h}_N^{(l)})_i$$
- Learning goalExplain how transformer self-attention contextualizes static input embedding vectors across hidden layers.
- Helpful foundationsInput Embeddings · Positional Encodings · Self-Attention Mechanism
- Common trapInput embeddings do not fix final token meaning; static embeddings are dynamically contextualized across successive attention layers.
- Next connectionSituate text tokenization within the broader lifecycle of transformer model architecture.
More context & mechanics
In an embedding matrix E, the token ID for "apple" always pulls the exact same numerical vector x_(apple) ∈ R^d, regardless of whether the sentence is "Apple unveiled the new iPhone" or "I ate a juicy red apple". However, as the vector passes through L successive transformer layers, multi-head self-attention attends to surrounding tokens ("iPhone" vs "juicy red"). By the final layer, the hidden state h_(apple)^((L)) has shifted geometrically in activation space to encode either a multinational tech corporation or a fresh fruit.
Probing experiments on BERT and Llama show that early layers (layers 1–4) encode syntactic part-of-speech information, middle layers capture semantic relationships, and late layers synthesize task-specific high-level reasoning.
Peters et al. (2018) introduced ELMo, followed immediately by Vaswani et al. (2017) and Devlin et al. (2018) with BERT. This demonstrated that static embeddings were inadequate for polysemous language, sparking the modern paradigm of deep contextualized representations.
Mental hook: "Static at the gate, dynamic through the maze."
A token enters the transformer with a fixed passport (static embedding), but picks up all its knowledge as it walks through the layers (contextual states).
Where does the tokenization and text representation pipeline sit within the end-to-end lifecycle of a large language model?
Before a neural network multiplies a single weight tensor or computes cross-entropy gradients, text must pass through the tokenizer gateway. If the tokenizer chops non-English text inefficiently, strips critical whitespace in code, or bottlenecks sequence length, no amount of subsequent parameter scaling or reinforcement learning can recover the lost information downstream.
- $$\text{Text} \xrightarrow{\text{Tokenize}} \mathbf{t} \xrightarrow{\text{Embed}} X^{(0)} \xrightarrow{\text{Transformer}} X^{(L)} \xrightarrow{\text{Unembed}} \mathbf{z} \xrightarrow{\text{Softmax}} P(t_{N+1})$$
- Learning goalDiagram where tokenization sits in the LLM architecture lifecycle and assess its downstream impact on training and inference.
- Helpful foundationsTokenization Pipeline · Transformer Architecture · LLM Training Lifecycle
- Common trapTokenization is not a swappable preprocessing utility; it fixes vocabulary dimension and sequence lengths permanently for the model lifetime.
- Next connectionTransition to the statistical foundations of reinforcement learning and sequential decision-making.
More context & mechanics
Tokenization is the foundational bedrock of the LLM pipeline:
- Data Preparation: Massive web corpora are normalized and used to train a tokenizer vocabulary.
- Encoding & Embedding: Text is converted to token IDs, paired with positional encodings, and projected to dimension d.
- Pretraining: Transformer blocks compute self-attention to predict the next token via cross-entropy loss.
- Post-Training (SFT & RLHF): Special tokens (`<|im_start|>`, `<|user|>`) structure conversational turns.
- Inference & De-tokenization: Logits are sampled, converted back to token IDs, and de-tokenized back into readable human strings.
Because the vocabulary |V| dictates the output classification layer (d → |V|), tokenizer design is locked in before pretraining begins. OpenAI, Meta, and Anthropic spend weeks benchmarking candidate tokenizers before launching multi-million-dollar training runs.
The evolution from character-level RNNs in 2011 to subword transformers in 2017 established tokenization as a permanent first-class engineering discipline in AI, standardizing the modern LLM build pipeline.
Mnemonic: "Tokenize, Vectorize, Modernize."
Tokenize to discretize strings; vectorize to calculate in space; modernize through transformer layers to produce intelligence.