Preface

ELIZA relied on decomposition rules and reassembly rules:

Input: [I am X] →(Regex Matching) Output: [How long have you been X?]

In contrast, a modern autoregressive Transformer models language as a joint probability distribution decomposed via the chain rule of probability:

P(w₁, w₂, ṡ, w_T) = ∏_(t=1)^T P(w_t | w₁, ṡ, w_(t-1))

Each token w_t is projected into continuous space R^d, passed through self-attention layers that compute contextual dependencies across all T positions simultaneously, and transformed via an unembedding projection W_U ∈ R^(d × |V|) into logits z_t. A softmax function normalizes these logits into a categorical probability distribution over vocabulary V (|V| = 50,257 in GPT-2).

While customer support systems in the 1990s used rule-based decision trees, modern conversational agents (from GitHub Copilot to ChatGPT and Claude) utilize deep Transformer backbones. Understanding this distinction explains why modern models can generalize to unseen questions, translate between programming languages, and write poetry—capabilities impossible under rigid regex reassembly.

In 1966 at MIT, Joseph Weizenbaum created ELIZA (featuring the famous 'DOCTOR' script simulating a Rogerian psychotherapist). Weizenbaum was shocked that human users, including his own secretary, attributed genuine empathy and human intelligence to simple string-matching rules—a cognitive bias now known as the 'ELIZA effect.'

• ELIZA reflects strings with rigid regex mirrors; LLMs navigate trajectories through continuous geometric spaces.
• Rule-based systems match syntax; transformer models predict probability over latent semantics.

An open-weight model provides the complete parameter tensor set:

\Theta = W_E, W_Q^((l)), W_K^((l)), W_V^((l)), W_O^((l)), W_(1)^((l)), W_(2)^((l)), W_U_(l=1)^L

Stored in standardized serialization formats (e.g., Hugging Face `safetensors` or PyTorch `.pt` files). Each parameter is represented in 16-bit precision (FP16 or BF16, requiring 2 bytes per parameter). For a 7-billion parameter model (N = 7 × 10⁹):

Memory Footprint = 7 × 10⁹ × 2 bytes = 14 GB

Having open weights enables:

1. Local execution without telemetry or third-party API dependencies.
2. Direct weight inspection, parameter pruning, and Low-Rank Adaptation (LoRA) fine-tuning.
3. Quantization down to 8-bit, 4-bit, or 2-bit representations (W_(quant) = round(W / s) + z).

Enterprises with strict data privacy mandates (healthcare, defense, finance) cannot transmit patient records or proprietary code across external web APIs. They deploy open-weight foundation models (such as Llama 3.1 8B/70B or Mistral NeMo) inside self-hosted, air-gapped on-premise clusters or private VPCs.

For years, state-of-the-art models were accessible only through gated commercial APIs. In February 2023, Meta released LLaMA to academic researchers; its subsequent leak online ignited the modern open-source AI revolution. Projects like Alpaca, Vicuna, and vLLM demonstrated that open weights empower global community optimization, quantization, and architectural breakthroughs.

• Proprietary APIs give you outputs; open weights give you parameters; open-source gives you the training kitchen.
• Weight tensors are numbers in memory, not human code.

Any continuous or structured data modality can be unified into a sequential representation X = [x₁, x₂, ṡ, x_T]:

1. Text: Strings are parsed into subword token IDs via Byte Pair Encoding (x_t ∈ 0, ṡ, |V|-1).
2. Vision (ViT): An image I ∈ R^(H × W × C) is divided into non-overlapping patches of size P × P. The sequence length is T = (HW)/P², each patch flattened to vector x_p ∈ R^(P² C) and projected via linear layer E ∈ R^((P² C) × d).
3. Audio: Continuous audio pressure waves at 24 kHz are transformed via short-time Fourier transforms (STFT) into mel-spectrogram frames or compressed via neural audio codecs into discrete codebook indices.
4. Biology: The 20 standard amino acids are treated as a 20-character discrete vocabulary for evolutionary protein modeling.

Models like OpenAI's Whisper (audio-to-text), Sora (spacetime video patch prediction), Meta's ESM-2 (evolutionary scale protein structure prediction), and DeepMind's AlphaFold all rely on transformer architectures adapted from language sequence modeling.

In 2020, Dosovitskiy et al. published 'An Image is Worth 16x16 Words', demonstrating that standard transformer encoders with zero convolution inductive biases could match or beat ResNets on ImageNet when trained on large datasets. This sparked the universal convergence of deep learning architectures toward transformers.

• If you can tokenize it, the transformer can learn it.
• Pixels, phonemes, and peptides are all just vectors on a sequence timeline.

The complete LLM creation pipeline consists of four distinct engineering phases:

1. Data Curation & Tokenization: Raw web corpora (petabytes of text) are deduplicated via MinHash LSH, toxicity-filtered, and tokenized into integer tensors via Byte Pair Encoding.
2. Pretraining (Self-Supervised): Next-token prediction minimizing cross-entropy loss over trillions of tokens:

L_(pretrain)(θ) = -1/T∑_(t=1)^T log P_θ(x_t | x_(<t))

This phase consumes >95\% of total compute (thousands of GPU days).

1. Supervised Fine-Tuning (SFT): Training on curated (instruction, response) pairs to transition the model from raw text completion to conversational instruction following:

L_(SFT)(θ) = -∑_(t=1)^M log P_θ(y_t | x, y_(<t))

1. Alignment (RLHF / DPO): Optimizing model outputs against human preference labels (chosen y_w vs rejected y_l) using Direct Preference Optimization or PPO:

L_(DPO)(θ) = -E_((x, y_w, y_l))[log σ(β log (π_θ(y_w | x)) / (π_(ref)(y_w | x)) - β log (π_θ(y_l | x)) / (π_(ref)(y_l | x)))]

This exact four-stage pipeline is used by frontier AI labs to produce GPT-4, Llama 3, Claude 3.5 Sonnet, and Gemini. Every open-source foundation model repo follows this progression from pretraining checkpoints to `-Instruct` variants.

Between 2018 and 2022, OpenAI progressed from GPT-1 (unsupervised pretraining + supervised fine-tuning for single tasks) to GPT-2 (pretraining at scale without task fine-tuning), GPT-3 (few-shot prompting), and finally InstructGPT (Ouyang et al., 2022), which formalized the modern SFT + RLHF post-training pipeline.

• Pretraining builds the brain; fine-tuning teaches the manners; alignment sets the guardrails.
• Knowledge is absorbed during pretraining; behavior is sculpted during post-training.

High-level abstractions like Hugging Face's `AutoModelForCausalLM` hide essential mechanics:

• Causal attention masking: upper-triangular matrix filled with -∞ to enforce autoregressive causality.
• Layer normalization positioning (Pre-LN vs Post-LN) and its impact on vanishing gradients during backpropagation:

Pre-LN: x_(l+1) = x_l + Sublayer(LayerNorm(x_l))

• Key-Value caching during generation: avoiding redundant O(T²) recomputation of past token representations by caching K, V ∈ R^(B × h × T × dₖ).
• Matrix memory layouts: contiguous memory, stride ordering, and tensor shapes (B, T, d).

Implementing these line-by-line enables precise memory tracking:

Weight Memory = 2 × N bytes (in FP16)

KV Cache Memory per token = 2 × 2 × L × d_(model) bytes

Production AI engineers rarely use default abstractions out of the box. They write custom CUDA kernels with FlashAttention, implement tensor parallelism (Megatron-LM), and optimize KV-cache memory using PagedAttention (vLLM). You cannot optimize what you do not fundamentally understand.

Sebastian Raschka authored Build a Large Language Model (From Scratch) (2024) to bridge the massive gap between high-level prompt engineering tutorials and deep production systems engineering. Raschka demonstrated that an entire functional GPT-2 (124M parameters) can be written in a few hundred lines of clear, idiomatic Python and PyTorch.

• Abstractions let you move fast; first principles let you fix things when they break.
• If you cannot build it in clean PyTorch, you do not truly understand how it works.

Mathematically, a language model defines a probability distribution over arbitrary token sequences W = (w₁, w₂, ṡ, w_T) drawn from vocabulary V:

P(W) = P(w₁, w₂, ṡ, w_T)

Using the probability chain rule without any Markovian independence assumptions:

P(W) = ∏_(t=1)^T P(w_t | w₁, w₂, ṡ, w_(t-1))

A Large Language Model parameterizes this conditional distribution using deep neural network weights θ:

P_θ(w_t = v | w_(<t)) = (exp(z_t[v] / τ)) / (∑_(j=1)^(|V|) exp(z_t[j] / τ))

where z_t ∈ R^(|V|) is the unnormalized logit vector produced by the final linear projection layer, and τ > 0 is the sampling temperature.

Every generative text platform operates on this exact conditional probability calculation. During text generation, sampling strategies such as Greedy Decoding (v^* = argmax_v z_t[v]), Top-k filtering, or Nucleus (Top-p) sampling select the next token, which is appended to the context window for step t+1.

Claude Shannon laid the mathematical foundations of language modeling in his 1948 landmark paper 'A Mathematical Theory of Communication', calculating n-gram letter and word transition probabilities. It took nearly 70 years to evolve from Shannon's n-gram tables through recurrent neural networks to modern multi-billion-parameter deep Transformers.

• An LLM is not a database of facts; it is a statistical simulator of the next token.
• The chain rule of probability is the engine; the transformer is the vehicle.

Scale in LLMs is governed by three coupled dimensions:

1. Parameters (N): The number of trainable scalar weights in the network (124 × 10⁶ in GPT-2 small up to 405 × 10⁹ in Llama 3.1 405B).
2. Dataset Size (D): The number of tokens consumed during training (10 × 10⁹ in GPT-2 up to 15 × 10^(12) in modern models).
3. Compute Budget (C): Floating-point operations (FLOPs) estimated for standard dense autoregressive transformers as:

C ≈ 6 N D FLOPs

where each parameter undergoes approximately 2 FLOPs per token during the forward pass and 4 FLOPs per token during backpropagation.

Chinchilla Optimal Scaling: Hoffmann et al. (2022) proved that for compute-optimal pretraining, parameters and data should scale in equal proportion:

N_(opt) ∝ C^(0.5), D_(opt) ∝ C^(0.5)

meaning for every doubling of model parameters, training tokens must also double (D ≈ 20 N).

When planning a training run, machine learning engineers calculate GPU cluster requirements based on compute FLOPs:

Training Time (seconds) = (6 N D) / (Num GPUs × Peak FLOPs/GPU × MFU)

where MFU (Model FLOPs Utilization) typically ranges between 35% and 55% on modern H100 clusters.

In 2020, Kaplan et al. at OpenAI published empirical power-law scaling laws (L(N) ∝ N^(-α_N)), establishing that test cross-entropy loss drops predictably as compute, parameters, and data increase over multiple orders of magnitude. In 2022, DeepMind's Chinchilla paper corrected Kaplan's ratios, proving models had been severely undertrained relative to their size.

• C ≈ 6ND: Compute is six times parameters times data.
• A model is only as large as its compute-optimal training diet.

In an RNN, hidden state h_t is computed via a strictly sequential recurrence relation:

h_t = \tanh(W_(hh) h_(t-1) + W_(xh) x_t)

This imposes two fundamental limitations:

1. Sequential Compute Bottleneck: Computing h_t strictly requires h_(t-1). A training sequence of length T requires T consecutive sequential operations (O(T) sequential depth), preventing parallel execution on modern GPU architectures.
2. Gradient Dissipation: Gradients flowing backward across T time steps involve continuous matrix multiplications by W_(hh)^T:

(∂ h_T) / (∂ h₁) = ∏_(k=2)^T (∂ hₖ) / (∂ h_(k-1)) = ∏_(k=2)^T diag(1 - hₖ²) W_(hh)^T

causing exponential vanishing or explosion of gradients over long context windows.

In a Transformer, self-attention connects all token pairs with a maximum path length of O(1):

Attention(Q, K, V) = softmax((QK^T) / (√(dₖ)))V

During training, the complete sequence matrix X ∈ R^(T × d) is processed in parallel matrix multiplications across all tokens simultaneously.

Enabling massive parallelism on GPUs is the sole reason modern models could scale from millions to hundreds of billions of parameters. Training an 8-billion parameter model on 15 trillion tokens with an RNN would take centuries; on a GPU cluster using parallel Transformer self-attention, it takes weeks.

In 2017, a team of researchers at Google Brain and Google Research (Vaswani, Shazeer, Parmar, Uszkoreit, Jones, Gomez, Kaiser, Polosukhin) published 'Attention Is All You Need'. Originally designed for English-to-German machine translation, the architecture rapidly replaced RNNs, LSTMs, and GRUs across all language and sequence modeling domains.

• RNNs think step-by-step in time; Transformers look everywhere at once in space.
• O(1) sequential dependency path enables massive GPU parallelization.

The formal taxonomy of intelligent systems forms a strict hierarchy:

AI \supset ML \supset DL \supset GenAI \supset LLMs

1. Artificial Intelligence (AI): The broad discipline of engineering systems that perform tasks requiring human-like intelligence (symbolic systems, heuristic search, logic solvers).
2. Machine Learning (ML): Algorithms whose performance P on task T improves with experience E without explicit procedural programming: min_θ L(θ; D).
3. Deep Learning (DL): Representation learning using compositions of non-linear parameter layers:

f(x) = σ(W_L σ(W_(L-1) ṡ σ(W₁ x + b₁) ṡ + b_(L-1)) + b_L)

1. Generative AI (GenAI): Models estimating data distributions P(X) to sample novel data instances x̃ ~ P_θ(X), distinct from discriminative models estimating conditional boundaries P(Y | X).
2. Large Language Models (LLMs): Massive autoregressive transformers modeling token sequences over discrete vocabularies: P(X) = ∏_(t=1)^T P(x_t | x_(<t)).

Engineering teams select tools across this spectrum based on problem constraints: tabular fraud detection often uses ML (XGBoost), defect detection uses DL (CNNs/YOLO), while code generation and unstructured text analysis leverage LLMs.

The field originated at the 1956 Dartmouth Summer Research Project on Artificial Intelligence. Over seven decades, the dominant paradigm shifted from symbolic expert systems (1950s–1980s) to statistical learning (1990s–2000s), deep representation learning (2010s), and generative foundation models (2020s).

• AI is the goal; ML is the method; DL is the architecture; GenAI is the mode; LLM is the sequence specialist.
• Nested dolls: each inner layer inherits the principles of its outer parent.

The two-stage paradigm leverages transfer learning at planetary scale:

1. Pretraining Stage (Base Model θ_(base)):

• Objective: Self-supervised next-token prediction over raw, unlabelled text corpora D_(raw):

L_(NLL)(θ) = -∑_(i=1)^(|D|) ∑_(t=1)^(Tᵢ) log P_θ(x_(i,t) | x_(i,<t))

• Scale: 10^(11) to 10^(13) tokens; millions of GPU hours.
• Outcome: Broad linguistic syntax, world facts, commonsense reasoning, and code patterns stored in weights θ_(base).

1. Supervised Fine-Tuning Stage (Instruct Model θ_(instruct)):

• Objective: Supervised learning on prompt-response pairs (x, y) ∈ D_(SFT):

L_(SFT)(θ) = -∑_((x, y)) ∑_(t=1)^(|y|) log P_θ(y_t | x, y_(<t))

• Scale: 10⁴ to 10⁶ curated examples; a fraction (<1\%) of pretraining compute.
• Outcome: Transforms an unstructured text completer into an interactive, question-answering assistant.

Open-source model releases reflect this exact split: Meta releases `Llama-3-8B` (the raw base model that simply completes arbitrary text) alongside `Llama-3-8B-Instruct` (the fine-tuned conversational model ready for user-facing applications).

The pretraining-then-finetuning paradigm was popularized in NLP by ULMFiT (Howard & Ruder, 2018), GPT-1 (Radford et al., 2018), and BERT (Devlin et al., 2018). It replaced the earlier practice of training task-specific models from scratch with random initializations on small labeled datasets.

• Pretraining fills the library; fine-tuning hires the librarian.
• 99% of compute builds the base; 1% of compute steers the behavior.

Data characteristics vary by orders of magnitude across stages:

1. Pretraining Data:

• Volume: 1 × 10^(12) to 15 × 10^(12) tokens.
• Format: Unstructured plain text documents delimited by end-of-text markers: `[Document text...] <|endoftext|>`.
• Filtering Pipeline:
• FastText language identification (stripping non-target languages).
• MinHash / Locality Sensitive Hashing (LSH) removing duplicated documents and spam clusters.
• Perplexity filtering using a lightweight reference model to eliminate low-quality garbled text.

1. Instruction Fine-Tuning (SFT) Data:

• Volume: 10⁴ to 10⁶ dialogues (10⁷ to 10⁸ tokens).
• Format: Structured turn-based schemas:

`"messages": ["role": "user", "content": "...", "role": "assistant", "content": "..."]`

• Quality Standard: Verified correctness, tone consistency, step-by-step reasoning explanations (Chain-of-Thought).

1. Alignment (Preference) Data:

• Volume: 5 × 10⁴ to 2 × 10⁵ comparison pairs.
• Format: Triplet tuples: `(Prompt x, Chosen Completion y_w, Rejected Completion y_l)`.

Teams developing proprietary models spend over 70% of engineering time curating, synthesizing, and auditing datasets. Modern frontier models increasingly utilize high-quality synthetic data generated by stronger models and verified with symbolic compilers or unit tests.

Early LLMs like GPT-2 trained on WebText (40 GB of outbound Reddit links with ≥ 3 karma). By 2024, Hugging Face released FineWeb (15 trillion tokens derived from 96 Common Crawl snapshots), standardizing open-source pretraining pipelines and demonstrating that data curation matters more than model tweaks.

• Pretraining eats petabytes of filtered text; fine-tuning sips megabytes of pure gold.
• Data quality at the base sets the ceiling for all downstream performance.

A production LLM application separates the core probabilistic inference engine from external software layers:

1. Core Inference Engine: The transformer model executing forward passes and returning next-token logits z_t ∈ R^(|V|).
2. Decoding & Sampler: Controls randomness and determinism via temperature τ, Top-p (nucleus) truncation, repetition penalties, and stop token triggers:

pᵢ = (exp(zᵢ / τ)) / (∑_(j ∈ V^((p))) exp(zⱼ / τ))

1. Retrieval-Augmented Generation (RAG): Overcomes knowledge cutoffs and hallucinations by dynamically querying external vector stores:

x_(augmented) = [System Prompt] + [Retrieved Chunks c₁, ṡ, cₖ] + [User Query]

1. Guardrails & Content Moderation: Classifiers detecting prompt injections, jailbreaks, PII leakage, and toxic outputs before rendering to the client.
2. Inference Serving Layer: High-throughput GPU execution engines (e.g. vLLM, TensorRT-LLM) providing dynamic continuous batching and PagedAttention memory management.

Enterprise deployments of models like Claude or ChatGPT embed thousands of lines of orchestration code around the weights: real-time web search integrations, Python code interpreters, citation generators, and token-rate-limiting middleware.

When ChatGPT launched in November 2022, its underlying engine (GPT-3.5) had existed for months. What revolutionized the world was the full-stack product packaging: low-latency streaming websocket responses, an intuitive chat interface, conversation session management, and robust moderation endpoints.

• The model is the engine; the application is the car.
• Without guardrails, retrieval, and decoding management, an engine is just noise on a test bench.