GPT-2 (124M)
How does GPT-2 (124M) construct its unified input tensor from discrete token IDs and positional embeddings?
How this lesson unfolds.
Begin with How does GPT-2 (124M) construct its unified input tensor from discrete token IDs and positional embeddings?. Follow the visual diagrams to build an intuitive mental model.
How does GPT-2 (124M) construct its unified input tensor from discrete token IDs and positional embeddings?
Imagine labeling mail in an automated sorting depot. Every package needs two labels: what the package contains (its semantic identity, e.g. "apple"), and which slot in the conveyor belt it occupies (its sequence position, e.g. slot 4). In GPT-2, token index 42 looks up a 768-dimensional identity vector from the token table, while position index 3 looks up a 768-dimensional location vector from the position table. By adding these two vectors element-by-element, the resulting vector knows both what it is and where it is in the sentence.
- Learning goalFormulate the input tensor calculation X = W_E[tokens] + W_P[positions] in R^(B x T x 768) and calculate the total parameter count of GPT-2 124M input embeddings.
- Helpful foundationsLanguage Models are Unsupervised Multitask Learners / GPT-2 (Radford et al., 2019) · Attention Is All You Need (Vaswani et al., 2017) · Embedding Matrix Lookup & Positional Encodings
- Common trapAssuming GPT-2 uses sinusoidal positional encodings like the original 2017 Transformer. GPT-2 replaced fixed sinusoids with fully learnable positional embeddings (W_P), learned via gradient descent like any other weight parameter.
- Next connectionWith the input tensor X in R^(B x T x 768) constructed, it passes directly into the first of GPT-2's 12 stacked Pre-LN Transformer blocks.
More context & mechanics
Given a batch of token ID sequences t ∈ N^(B × T) with vocabulary size V = 50,257 and context window T_(max) = 1024:
1. Token Embedding Lookup:
The token embedding matrix W_E ∈ R^V × d_(model) stores a 768-dimensional vector for each vocabulary token:
E_(token) = W_E[t] ∈ R^(B × T × 768)
2. Positional Embedding Lookup:
GPT-2 defines a learned positional embedding table W_P ∈ R^T_(max) × d_(model).
For sequence positions p = [0, 1, 2, ṡ, T-1] broadcast across the batch:
E_(pos) = W_P[p] ∈ R^(B × T × 768)
3. Element-wise Summation & Dropout:
The input tensor X fed to the first Transformer block is the element-wise sum:
X = Dropout(E_(token) + E_(pos)) ∈ R^(B × T × 768)
Embedding Parameter Count Calculation:
- Token Embeddings: 50,257 × 768 = 38,597,376 parameters.
- Positional Embeddings: 1,024 × 768 = 786,432 parameters.
- Total Embedding Parameters: 39,383,808 (approx. 39.4M parameters, nearly 32% of GPT-2 124M's total weight budget!).
This learned embedding scheme is used across GPT-2, GPT-3, and early BERT models. Later architectures (like LLaMA) transitioned to Rotary Position Embeddings (RoPE), but the token embedding lookup W_E[t] remains universal.
In 2019, Radford et al. released the GPT-2 family (124M, 355M, 774M, 1.5B parameters). The 124M model demonstrated that pure autoregressive next-token prediction on diverse web corpora (WebText) naturally induces zero-shot multi-task problem-solving.
- Golden Axiom: "Token what plus position where equals input tensor."
- X = W_E[tokens] + W_P[positions] ∈ R^(B × T × 768). Words provide the identity, indices provide the order.
How are the 12 transformer blocks, token embeddings, positional encodings, and unembedding head assembled into the complete GPT-2 124M pipeline?
Imagine building a skyscraper. The foundation is a bedrock slab of word meanings and room numbers (Token + Positional Embeddings). Above this foundation, the building consists of 12 identical luxury office floors stacked vertically (the 12 Transformer Blocks). Each floor has its own communication center (Attention) and private analytical labs (MLP). At the very top floor, an executive penthouse suite (Final LayerNorm) inspects the final output document, and an observation telescope (Unembedding Head) views all 50,257 possible words in the sky to select the next sentence.
- Learning goalDescribe the end-to-end macro architecture of GPT-2 124M from token input through the 12-block stack to vocabulary logit prediction.
- Helpful foundationsPre-LN transformer block structure · Language Models are Unsupervised Multitask Learners (Radford et al., 2019)
- Common trapBelieving that GPT-2 has 124 million parameters plus the embedding matrix. The 124M parameter count includes the embedding matrix; because weight tying shares W_E with W_U, the unembedding head adds zero additional parameters.
- Next connectionNext, trace the exact component census and 38.6M weight-tying reduction calculus in Part B.
More context & mechanics
The GPT-2 124M (small) architecture is the canonical baseline of modern autoregressive language modeling:
- Input Encoding Stage:
- Input integer tokens t ∈ R^(B × T).
- Token embedding matrix W_E ∈ R^(50,257 × 768).
- Positional embedding matrix W_P ∈ R^(1024 × 768).
- Initial representation: x₀ = W_E[t] + W_P[0..T-1] ∈ R^(B × T × 768).
- The 12-Block Transformer Stack:
- 12 sequentially stacked Pre-LN transformer blocks.
- Each block contains:
- Multi-Head Causal Self-Attention (h = 12, dₖ = 64).
- Position-wise Feed-Forward Network (768 → 3072 → 768, with GELU).
- Two LayerNorm layers and two residual connections.
- Output Unembedding Stage:
- Final LayerNorm: x_(norm) = LayerNorm(x_(12)) ∈ R^(B × T × 768).
- Vocabulary Linear Head: Z = x_(norm) W_U ∈ R^(B × T × 50,257).
- Weight Tying: W_U = W_E^T, reusing the input embedding matrix.
GPT-2 124M is widely used as an educational and empirical research benchmark. It runs efficiently on consumer laptops and single GPUs, providing the exact architectural blueprint scaled up into GPT-3 (175B) and Llama-3 (8B/70B).
Alec Radford and the OpenAI team published "Language Models are Unsupervised Multitask Learners" in February 2019, introducing GPT-2. By demonstrating that unsupervised next-token prediction across large web corpora enabled zero-shot task transfer, GPT-2 sparked the modern LLM revolution.
- 12 blocks, 768 hidden, 12 heads, 1024 context: The iconic dimensions of GPT-2 124M.
- Embed, stack 12 blocks, normalize, and project: The four-movement symphony of GPT-2.
How do we derive the exact parameter count for every sublayer in GPT-2 124M and prove the 38.6M savings from weight tying?
Think of buying a dictionary that translates English to Spanish. In an untied system, the publisher prints two separate giant 50,000-page volumes: Volume 1 translates English to Spanish, and Volume 2 translates Spanish to English. In a tied system, the publisher prints only Volume 1 and explains: 'To translate back, simply read the dictionary in reverse.' You save an entire 50,000-page book of printing costs, shelf space, and paper.
- Learning goalDerive the exact parameter count for every sublayer in GPT-2 124M and calculate the 38.60M parameter savings achieved by weight tying.
- Helpful foundationsGPT-2 architecture blueprint · Weight tying formulation (Press & Wolf, 2016)
- Common trapAssuming that weight tying halves the parameters of the model. Weight tying only shares the embedding and unembedding matrices (38.6M), reducing total parameters by roughly 24\%, not 50\%.
- Next connectionNext, learn how logits produced by the unembedding head drive the autoregressive generation loop in Chapter 25 Screen 1.
More context & mechanics
#### 1. Exact Parameter Accounting Ledger
For GPT-2 small (V = 50,257, T = 1024, d = 768, h = 12, d_(ff) = 3072):
- Embedding Layer:
- Token Embeddings: W_E ∈ R^(50,257 × 768) = 38,597,376
- Positional Embeddings: W_P ∈ R^(1024 × 768) = 786,432
- Embedding Subtotal: 39,383,808 parameters.
- Single Transformer Block:
- Attention QKV Projection (W_(QKV)): 768 × (3 · 768) + 3 · 768 = 1,769,472 + 2,304 = 1,771,776
- Attention Output Projection (W_O): 768 × 768 + 768 = 589,824 + 768 = 590,592
- LayerNorm 1: 2 × 768 = 1,536
- FFN Layer 1 (W₁): 768 × 3072 + 3072 = 2,359,296 + 3,072 = 2,362,368
- FFN Layer 2 (W₂): 3072 × 768 + 768 = 2,359,296 + 768 = 2,360,064
- LayerNorm 2: 2 × 768 = 1,536
- Total per block: 1,771,776 + 590,592 + 1,536 + 2,362,368 + 2,360,064 + 1,536 = 7,087,872 parameters.
- All 12 Blocks Subtotal: 12 × 7,087,872 = 85,054,464 parameters.
- Final Head & Output:
- Final LayerNorm: 2 × 768 = 1,536
- Unembedding Head: Tied to W_E^T (0 new parameters!).
- Master Total:
Total Parameters = 39,383,808 + 85,054,464 + 1,536 = 124,439,808 ≈ 124.44M
If the unembedding matrix W_U were untied and learned independently:
Untied Parameters = 124,439,808 + (50,257 × 768) = 124,439,808 + 38,597,376 = 163,037,184 ≈ 163.04M
Weight tying eliminates 38,597,376 parameters (23.67\% memory reduction) with zero loss in expressivity!
Weight tying is implemented in PyTorch by setting `self.lm_head.weight = self.transformer.wte.weight`. In addition to saving 154 MB of VRAM, weight tying acts as a powerful regularizer, preventing word embeddings from diverging between the input and output spaces.
Using tied input-output embeddings was proposed independently by Hakan Inan et al. and Ofir Press & Lior Wolf in 2016. Vaswani et al. incorporated weight tying into the original Transformer in 2017, and Radford retained it in GPT-2.
- 39M + (12 × 7.08M) + 1.5k = 124.4M: The exact census of GPT-2.
- Tie the weights to save a quarter: Reusing W_E saves nearly 25% of the total parameter budget.
How does a single linear layer translate 768-dimensional contextual vectors into raw prediction scores across a 50,257-token vocabulary?
Imagine a translator standing before a wall of 50,257 small numbered pigeonholes, each containing one word. The translator holds a single 768-dimensional keycard (the contextual vector). Sliding this keycard against all 50,257 magnetic locks simultaneously generates an electrical voltage (logit) at each slot; the slot with the strongest voltage is the top next-word guess.
- $$The unembedding head computes an unconstrained inner product projection Z = H_out W_u + b_u, transforming contextual representations (B, T, 768) into vocabulary logits (B, T, 50257).$$
- Learning goalCompute the linear projection from final hidden state to vocabulary logits, verify output tensor dimensions (B, T, 50257), and trace how logit values map to next-token probabilities.
- Helpful foundationsUsing the Output Embedding to Improve Language Models / Weight Tying (Press & Wolf, 2017) · Language Models are Unsupervised Multitask Learners / GPT-2 (Radford et al., 2019) · The Illustrated Transformer (Jay Alammar, 2018)
- Common trapBelieving that the model predicts a single token for the entire input sequence. The unembedding head projects every single token position t in [1, ..., T] in parallel. Position t predicts token t+1. In training, all T predictions are evaluated simultaneously against target tokens shifted by one.
- Next connectionStoring this unembedding projection matrix W_u in R^(768 x 50257) requires 38.6 million parameters; Screen 4 demonstrates how Weight Tying eliminates this duplicate memory cost.
More context & mechanics
The output from the 12th Transformer block and final LayerNorm is H_(out) ∈ R^B × T × d_(model).
The unembedding head (or language modeling head) applies a linear transformation without activation function:
Z = H_(out) W_u + b_u
where:
- H_(out) ∈ R^(B × T × 768)
- W_u ∈ R^(768 × V) (where vocabulary size V = 50257)
- b_u ∈ R^(V) (bias vector; in standard GPT-2, bias is omitted, b_u = 0)
- Z ∈ R^(B × T × 50257) (the raw unnormalized logits)
Toy worked arithmetic trace:
Suppose batch size B=1, sequence length T=1, and toy d_(model)=3, with vocabulary V=4:
h = [1.0, 2.0, -1.0].
Let unembedding weight matrix columns be candidate word vectors:
W_u = 1 & 0 & -1 & 2 \ 2 & 1 & 0 & -1 \ 0 & -2 & 3 & 1
Logits are the dot product of h with each column:
- Word 0: (1)(1) + (2)(2) + (-1)(0) = 1 + 4 + 0 = 5.0
- Word 1: (1)(0) + (2)(1) + (-1)(-2) = 0 + 2 + 2 = 4.0
- Word 2: (1)(-1) + (2)(0) + (-1)(3) = -1 + 0 - 3 = -4.0
- Word 3: (1)(2) + (2)(-1) + (-1)(1) = 2 - 2 - 1 = -1.0
Raw logits: z = [5.0, 4.0, -4.0, -1.0]. The network expresses highest confidence in Word 0.
During autoregressive text generation, only the logit vector at the very last token position Z[:, -1, :] is extracted to sample the next token. During parallel training, all T positions are projected to compute Cross-Entropy Loss against shifted targets.
Bengio et al. (2003, 'A Neural Probabilistic Language Model') first introduced the linear vocabulary projection layer over distributed representations, laying the mathematical cornerstone for modern autoregressive prediction.
- Golden Axiom: "Unembedding is simply 50,257 simultaneous dot products: whichever vocabulary vector aligns best with the context vector earns the highest logit."
- Tensor contract: (B, T, 768) × (768, 50257) → (B, T, 50257).
How does sharing weights between token embeddings and the unembedding head save 38.6 million parameters in GPT-2?
Think of a two-way pocket dictionary: English to French in the front, French to English in the back. Instead of printing two separate 50,000-word volumes, weight tying flips the same pages in reverse, cutting physical paper volume by nearly a third while guaranteeing that a word definition matches its translation.
- $$Weight tying enforces W_u = W_emb^T, allowing the 50257 x 768 embedding table to serve dual duty as both input encoder and output logit projector, saving 38.6M parameters and regularizing semantic representations.$$
- Learning goalCalculate the complete parameter accounting ledger for GPT-2 (124M), derive the exact 38,597,376 parameter savings from weight tying, and analyze the regularization effect of transpose sharing.
- Helpful foundationsUsing the Output Embedding to Improve Language Models (Press & Wolf, 2017) · Weight Tying in Pre-trained Language Models (Inan, Khosravi & Socher, Salesforce Research, 2016) · Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism (Shoeybi et al., NVIDIA, 2019)
- Common trapBelieving that weight tying reduces computational FLOPs during inference. Weight tying shares memory storage, not matrix multiplication operations. The GPU must still compute the full (B, T, 768) x (768, 50257) matrix product; weight tying saves VRAM parameter storage and accelerates gradient learning via dual-gradient feedback.
- Next connectionWith parameter allocation rigorously counted, Screen 5 audits the runtime VRAM memory budget required to train and run inference on this 124M model across FP32 and FP16 precisions.
More context & mechanics
In an untied architecture, the model allocates two separate matrices of size V × d_(model):
- Input token embedding table: W_(emb) ∈ R^(50257 × 768) (38,597,376 parameters)
- Output unembedding projection: W_u ∈ R^(768 × 50257) (38,597,376 parameters)
Weight tying binds them by transposition:
W_u = W_(emb)^T
Complete GPT-2 Small (124M) parameter ledger breakdown:
- Token Embeddings: 50257 × 768 = 38,597,376
- Positional Embeddings: 1024 × 768 = 786,432
- Single Transformer Block:
- Multi-Head Attention Q, K, V projections: 3 × (768 × 768 + 768) = 1,771,776
- Attention Output projection W_O: 768 × 768 + 768 = 590,592
- MLP expansion W₁: 768 × 3072 + 3072 = 2,362,368
- MLP contraction W₂: 3072 × 768 + 768 = 2,360,064
- LayerNorm 1 (γ, β): 2 × 768 = 1,536
- LayerNorm 2 (γ, β): 2 × 768 = 1,536
- Total per block: 7,087,872 parameters
- 12 Stacked Blocks: 12 × 7,087,872 = 85,054,464
- Final LayerNorm: 2 × 768 = 1,536
- Output Unembedding Head: Tied to W_(emb) (0 additional unique parameters)
Total Unique Trainable Parameters:
Total = 38,597,376 + 786,432 + 85,054,464 + 1,536 = 124,439,808 ≈ 124.44M
Without weight tying, total count would be 124.44M + 38.60M = 163.04M. Weight tying reduces parameter footprint by 23.7\%.
Weight tying was standardized in GPT-2, GPT-3, RoBERTa, and ALBERT. While modern large models like Llama 3 separate input and output embeddings due to vocabulary expansion (128,000 tokens in Llama 3), small and edge models heavily utilize weight tying to fit inside tight mobile caches.
Hakan Inan et al. (Salesforce Research, 2016, 'Tying Word Vectors and Word Classifiers: A Loss Framework for Language Modeling') and Ofir Press & Lior Wolf (Tel Aviv University, 2017, 'Using the Output Embedding to Improve Language Models') concurrently proved that tying input and output matrices improves perplexity by constraining input semantics to match output predictive space.
- Golden Axiom: "One matrix, two duties: embed input tokens on the way in, project output logits on the way out."
- Memory trick: 50257 × 768 ≈ 38.6M params. Multiply by 2 without tying (77.2M); with tying, pay once.
Why does training a 124M model require over 2 GB of GPU VRAM when the model weights themselves take only 498 MB?
Think of purchasing a desk. A laptop takes up only two square feet of surface space. But to actually work, you need room for open notebooks, draft papers, textbooks, and a trash bin for discarded ideas. The working environment demands ten times more space than the computer sitting on it.
- Learning goalCompute the precise byte-level VRAM allocation for parameters, gradients, AdamW optimizer states, and forward activations across FP32, FP16, and mixed-precision regimes.
- Helpful foundationsMixed Precision Training (Micikevicius et al., NVIDIA & Baidu, ICLR 2018) · ZeRO: Memory Optimizations Toward Training Trillion Parameter Models (Rajbhandari et al., Microsoft DeepSpeed, 2019) · FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning (Dao, 2023)
- Common trapAssuming that an 8 GB consumer GPU can easily train a 1-billion parameter model because 1B x 2 bytes (FP16) = 2 GB. Training requires storing gradients (2 GB), AdamW FP32 master weights (4 GB), momentum buffers (4 GB), and variance buffers (4 GB) totaling 16 GB before even counting forward activation memory.
- Next connectionThis concludes the architectural assembly of GPT-2 (124M); Chapter 25 implements the autoregressive generation loop to produce real output tokens using these exact weights and memory buffers.
More context & mechanics
Data representation sizes:
- FP32 (Full Precision IEEE 754): 4 bytes per parameter
- FP16 / BF16 (Half Precision / Brain Float): 2 bytes per parameter
Static VRAM breakdown for GPT-2 (124.44M parameters):
- Pure FP32 Training:
- Model weights: 124.44M × 4 bytes = 497.76 MB
- Gradients: 124.44M × 4 bytes = 497.76 MB
- AdamW Optimizer States:
- FP32 First Moment (m_t, momentum): 124.44M × 4 bytes = 497.76 MB
- FP32 Second Moment (v_t, variance): 124.44M × 4 bytes = 497.76 MB
- Total Static State: 4 + 4 + 4 + 4 = 16 bytes/parameter = 1.991 GB ≈ 2.0 GB
- Mixed Precision (FP16 / BF16 AMP) Training:
- FP16 model weights (forward/backward): 124.44M × 2 bytes = 248.88 MB
- FP16 gradients: 124.44M × 2 bytes = 248.88 MB
- FP32 Master Weights (for numerical stability): 124.44M × 4 bytes = 497.76 MB
- FP32 AdamW m_t: 497.76 MB
- FP32 AdamW v_t: 497.76 MB
- Total Static State: 2 + 2 + 4 + 4 + 4 = 16 bytes/parameter ≈ 1.991 GB
- Dynamic Activation Memory:
For batch size B=4, context length T=1024, layers L=12, storing forward activations for backpropagation without activation checkpointing requires an additional 2.5 GB to 4.0 GB, pushing total VRAM requirement to ≈ 5–6 GB.
In pure inference (no gradients, no optimizer), GPT-2 requires only the model weights:
- FP32 Inference: 498 MB
- FP16 Inference: 249 MB
- INT8 Quantized Inference: 124 MB
Memory accounting governs all distributed LLM training strategies, including DeepSpeed ZeRO (Zero Redundancy Optimizer, Rajbhandari et al., 2019), which partitions optimizer states (ZeRO-1), gradients (ZeRO-2), and parameters (ZeRO-3) across multiple GPUs.
Paulius Micikevicius and NVIDIA researchers introduced 'Mixed Precision Training' at ICLR 2018, establishing the master-weight FP32 accumulator technique that enabled deep learning to scale onto modern Tensor Cores.
- Golden Axiom: "Weights are just the tip of the iceberg: AdamW keeps three hidden copies of every parameter in memory."
- The 16-byte rule: In FP32 or Mixed-Precision AdamW training, multiply parameter count by 16 to get minimum static VRAM (124M × 16 B ≈ 2 GB).
How does sharing weights between the input embedding and output classification head reduce parameters while enforcing semantic symmetry?
Think of a bilingual English-French dictionary. The front half translates English words into French (Encoding). The back half translates French words back into English (Decoding). Writing two completely separate dictionaries doubles printing costs and risks contradictions where word A means word B, but word B maps to word C. Weight tying is like using a single reversible dictionary: looking up word coordinates from left to right encodes tokens, and reading from right to left decodes vectors back into token probabilities.
- $$Bind the output projection matrix W_out to the transpose of the input embedding matrix W_E^T, halving parameter storage for large vocabularies.$$
- Learning goalFormulate the mathematical equivalence of weight tying and quantify memory savings across large vocabulary architectures.
- Helpful foundationsMatrix transpose · Using the Output Embedding to Improve Language Models (Press & Wolf, 2016)
- Common trapAssuming weight tying degrades model expressiveness; by tying representations, the semantic space learned by the input is directly aligned with output prediction targets, often improving test perplexity.
- Next connectionBefore we build massive multi-layer networks, how does a single artificial neuron make a decision? We start with the elementary line equation.
More context & mechanics
In standard language models:
- Input embedding: x → e = W_E[i, :] ∈ R^(d), where W_E ∈ R^(|V| × d).
- Output projection: h → z = W_(out) h ∈ R^(|V|), where W_(out) ∈ R^(|V| × d).
Under Weight Tying:
W_(out) = W_E ⟹ z = W_E h
The logit for token i is the direct Euclidean dot product between final hidden state h and token embedding wᵢ:
zᵢ = wᵢ · h
This guarantees that if token wᵢ is closely aligned with hidden context h, its output emission probability is maximized.
Weight tying is implemented in GPT-2 (124M), where vocabulary size |V| = 50257 and d = 768. Storing two distinct matrices would consume 2 × 50257 × 768 ≈ 77.2M parameters (over 60\% of the model!). Tying saves 38.6M parameters.
Ofir Press and Lior Wolf proved the theoretical and empirical advantages of tying input and output embeddings in their 2016 paper 'Using the Output Embedding to Improve Language Models'.
- Two sides of the same coin: W_out is W_E transpose.
- Input: integer to vector; Output: vector dot-product to logit.
How does GPT-2 124M Architecture Specifications & Parameter Budget operate mathematically at scale?
A structured parameter ledger table:
- Learning goalMaster the algebraic and geometric derivation of GPT-2 124M Architecture Specifications & Parameter Budget.
- Helpful foundationsFoundations of Build Llm From Scratch · Coordinate Invariance & Geometric Projections · Radford et al. (2019), Language Models are Unsupervised Multitask Learners, OpenAI
- Common trapAssuming GPT-2 124M Architecture Specifications & Parameter Budget is an arbitrary computational trick rather than an invariant spatial law.
- Next connectionConnects upstream foundational lemmas to modern high-dimensional computational systems.
More context & mechanics
A parameter accounting infographic in square 1:1 format on warm cream parchment (#fcf0db) with an 8px border.
Top section: Title in navy serif: "GPT-2 (124M): Architectural Accounting". Subtitle: "Exact mathematical breakdown of parameters across embedding, attention, and MLP layers."
Middle section: A structured parameter ledger table:
- Vocabulary & Positional Embeddings:
- Token Embeddings: 50,257 × 768 = 38.6M parameters.
- Positional Embeddings: 1,024 × 768 = 0.8M parameters.
- Transformer Core (12 Identical Blocks):
Per Block Parameters:
- Self-Attention (Q, K, V + Output): 4 × (768 × 768) = 2.36M.
- Feed-Forward MLP: (768 × 3072) + (3072 × 768) = 4.72M.
- LayerNorms & Biases: ~0.01M.
Total per block = 7.08M × 12 blocks = 85.0M parameters.
- Total Model: 38.6M + 0.8M + 85.0M = 124.4 Million Parameters!
Bottom section: Weight tying note: "If embedding and output head are tied, total parameters drop to 85.8M; in original GPT-2, they are untied."
Clean ledger card styling, crisp monospace math numbers, balanced square frame.
Every transformation preserves structural invariants while mapping coordinate spaces. When operating in high dimensions, numerical precision and computational efficiency require adhering strictly to canonical mathematical laws.
Formal derivation and peer-reviewed analysis:
Radford et al. (2019), Language Models are Unsupervised Multitask Learners, OpenAI
"GPT-2 124M Architecture Specifications & Parameter Budget: Invariance, symmetry, and structure govern mathematical truth across continuous and discrete domains."