Tokenization & Vector Embedding Space
How do discrete token identifiers translate into continuous vector embeddings along high-dimensional semantic axes?
How this lesson unfolds.
Computers cannot process human words directly. Before attention mechanisms or feedforward layers can compute relationships, discrete token integers must be projected into continuous geometric coordinate space where Euclidean distance and directional alignment reflect semantic meaning.
How does embedding discrete token symbols into continuous geometric coordinates establish semantic proximity?
Think of a massive library catalog where every book is originally given an arbitrary integer ticket: apple is 42, banana is 9871, and nuclear reactor is 43. Storing words as plain integers places apple right beside nuclear reactor, which makes no sense. The embedding layer converts that arbitrary integer into a multi-dimensional GPS coordinate: apple becomes [0.82, 0.15], banana becomes [0.80, 0.18], and nuclear reactor shifts far away to [-0.91, 0.44]. Suddenly, words with shared meaning live in the exact same geographic neighborhood.
- $$Map discrete categorical vocabulary indices into continuous dense vectors in R^d such that semantic similarity corresponds to Euclidean distance and cosine angle.$$
- Learning goalMap discrete integer token identifiers into continuous dense vectors and evaluate semantic similarity using Euclidean distance and cosine metrics.
- Helpful foundationsHigh-school Cartesian coordinates · Efficient Estimation of Word Representations in Vector Space (Mikolov et al., 2013)
- Common trapAssuming that token IDs possess intrinsic numerical order where token ID 100 is 'greater' or 'more important' than token ID 50; categorical IDs are purely arbitrary labels until projected into embedding space.
- Next connectionOnce tokens become continuous vectors, how do we measure the contextual relevance between pairs of vectors across a sequence? We turn to the dot product.
More context & mechanics
An embedding table W_E ∈ R^|V| × d_(model) acts as a continuous lookup matrix. Given a one-hot vector x ∈ 0, 1^(|V|) representing token index i:
eᵢ = x^T W_E = W_E[i, :]
Through backpropagation, gradient updates pull tokens appearing in similar linguistic contexts toward each other, minimizing the distance:
D(eᵢ, eⱼ) = \|eᵢ - eⱼ\|₂
While maximizing directional alignment measured by cosine similarity:
cos(θ) = (eᵢ · eⱼ) / (\|eᵢ\| \|eⱼ\|)
Every modern language model (GPT-4, LLaMA-3, Claude) starts by passing input token IDs through this lookup table. At d_(model) = 4096, each token occupies a point in a 4,096-dimensional hyperspace where grammatical tense, gender, and conceptual associations form parallel geometric offset vectors.
In 2013, Tomas Mikolov et al. at Google published Word2Vec, showing that neural networks trained on skip-gram prediction produced emergent vector arithmetic: vec(v)_(King) - vec(v)_(Man) + vec(v)_(Woman) ≈ vec(v)_(Queen).
- Integers are arbitrary labels; vectors are semantic coordinates.
- Proximity equals synonymy; direction equals relationship.