Input Output Pairs
How does continuous text turn into supervised training targets for LLM pretraining?
How this lesson unfolds.
Begin with How does continuous text turn into supervised training targets for LLM pretraining?. Follow the visual diagrams to build an intuitive mental model of input output pairs.
What is the mathematical foundation and geometric intuition of Autoregressive Dataset Construction: Slice Boundary Bounds?
Understand Autoregressive Dataset Construction: Slice Boundary Bounds visually: Target tokens are shifted by exactly +1 position from input context tokens.. Every transformation preserves core structural invariants while mapping coordinate spaces.
- Learning goalMaster the algebraic and geometric derivation of Autoregressive Dataset Construction: Slice Boundary Bounds.
- Helpful foundationsLinear Algebra Fundamentals · Matrix Transformations · Vector Spaces
- Common trapAssuming Autoregressive Dataset Construction: Slice Boundary Bounds is merely a computational trick rather than an invariant spatial law.
- Next connectionConnects to upstream spectral theory and downstream tensor operations.
More context & mechanics
A tensor indexing diagram on natural warm linen (#f5f4e9) with a 1px slate border.
Top section: Title in bold sans-serif: "Autoregressive Dataset Construction: Slice Boundary Bounds". Subtitle: "Target tokens are shifted by exactly +1 position from input context tokens."
Middle section: A tokenized integer sequence S of total length N.
Draw a sliding window of context length T = 4 at window index i:
Input Tensor Xᵢ = S[i : i + T] = [sᵢ, s_(i+1), s_(i+2), s_(i+3)].
Target Tensor Yᵢ = S[i + 1 : i + T + 1] = [s_(i+1), s_(i+2), s_(i+3), s_(i+4)].
Show pairwise alignment:
Token sᵢ predicts s_(i+1); s_(i+1) predicts s_(i+2); s_(i+2) predicts s_(i+3); s_(i+3) predicts s_(i+4).
Bottom section: Indexing Boundary Card: "Index Formula: Yᵢ[k] = S[i + k + 1] for k ∈ 0, ..., T-1; \ Total possible windows in corpus: N - T."
Spacious horizontal tensor slots, clean brackets, uncluttered design.
This principle is foundational across deep learning optimization, high-dimensional representation, and modern LLM architectures.
"Autoregressive Dataset Construction: Slice Boundary Bounds: Invariance and structure determine stability under spatial and numerical transformations."
How do multidimensional tensor shapes encode batch size, sequence length, and embedding dimensions across the LLM pipeline?
Think of tensor shapes like reading shipping container dimensions. A 1D tensor (T,) is a single grocery receipt with T items listed in order. A 2D tensor (B, T) is a binder holding B separate receipts, each with T items (our batch of token IDs). A 3D tensor (B, T, d) transforms each item on each receipt into an entire catalog page of d=768 continuous physical measurements (height, weight, color). A 4D tensor (B, h, T, dₖ) splits those catalog pages across h=12 specialist inspectors, each examining a dₖ=64 slice of the features.
- Learning goalInterpret, trace, and debug tensor dimension transformations across the model forward pass: (B, T) -> (B, T, d) -> (B, h, T, dₖ) -> (B, T, |V|).
- Helpful foundationsMultidimensional arrays and matrix broadcasting · PyTorch tensor operations (view, transpose, reshape) · Array Programming with NumPy (Harris et al., Nature 2020) · PyTorch Tensor View and Strides Documentation
- Common trapConfusing tensor.view() with tensor.transpose(). Transposition swaps physical axes and may require calling .contiguous() in memory; view() merely reinterprets existing memory strides without reallocating data, which will crash or corrupt values if memory is non-contiguous.
- Next connectionNow that we can read tensor geometries, we implement the PyTorch Dataset and DataLoader to bundle sliding windows into batches.
More context & mechanics
Tensor shapes trace the lifecycle of data through the network:
- Input Token IDs:
`torch.Size([B, T])` ∈ Z^(B × T)
B = Batch size (e.g. 8), T = Context length (e.g. 1024).
- Embedded Representations:
`torch.Size([B, T, d_model])` ∈ R^(B × T × 768)
Each token ID is replaced by a 768-dimensional coordinate vector.
- Multi-Head Attention Projections:
`torch.Size([B, h, T, dₖ])` ∈ R^(B × 12 × T × 64)
Reshaping d_(model) → h × dₖ (where 12 × 64 = 768) to compute h parallel attention heads.
- Attention Scores Matrix:
`torch.Size([B, h, T, T])` ∈ R^(B × 12 × T × T)
Pairwise dot products Q K^T across all token pairs.
- Output Logits:
`torch.Size([B, T, |V|])` ∈ R^(B × T × 50257)
Unembedding projection back to vocabulary probability space.
Over 80% of runtime bugs in PyTorch LLM development stem from dimension mismatches (`RuntimeError: The size of tensor a (1024) must match the size of tensor b (768)`). Experienced engineers print and assert tensor shapes at every layer boundary during debugging.
The tensor abstraction in computer science traces back to Ricci-Curbastro and Levi-Civita's absolute differential calculus (1900), adopted by Einstein for general relativity, and formalized for GPU array computing in NumPy (2006) and PyTorch (2016).
- `(B, T)` is words; `(B, T, d)` is meaning; `(B, h, T, dₖ)` is attention; `(B, T, |V|)` is prediction.
- Check your shapes before you check your weights.
How do PyTorch `Dataset` and `DataLoader` pipelines manage batch collation, shuffling, stride offsets, and GPU memory transfer?
Imagine an automated conveyor belt feeding steel ingots into a blast furnace. You have a mountain of raw ore (a 1-gigabyte text file), but the furnace can only melt 4 ingots at a time (batch size B=4), and each ingot must be exactly 256 centimeters long (sequence length T=256). A PyTorch Dataset cuts the ore into exact 256-token ingots using a stride offset; the DataLoader packs 4 ingots into a heat-resistant pallet, shuffles the pallets so the furnace doesn't melt the same patterns repeatedly, and loads them directly into GPU VRAM using pinned memory.
- Learning goalImplement a custom PyTorch Dataset and DataLoader with configurable context window size T, sliding stride S, batch size B, and memory pinning.
- Helpful foundationsObject-oriented programming in Python (__len__, __getitem__) · PyTorch: An Imperative Style, High-Performance Deep Learning Library (Paszke et al., NeurIPS 2019) · PyTorch DataLoader Design and Architecture
- Common trapSetting the sliding stride equal to 1 during large-scale pretraining. While stride 1 maximizes data augmentation, it creates 99.9% overlapping chunks across adjacent batches, leading to redundant compute and overfitting. Pretraining typically uses non-overlapping stride S = T to maximize unique token exposure.
- Next connectionWith batched integer token tensors of shape (B, T) rolling out of our DataLoader, Lesson 10 takes the next critical step: mapping those discrete integers into continuous token embeddings.
More context & mechanics
A production LLM `Dataset` implementation:
```python
import torch
from torch.utils.data import Dataset, DataLoader
class GPTDatasetV1(Dataset):
def __init__(self, txt, tokenizer, max_length, stride):
self.input_ids = []
self.target_ids = []
token_ids = tokenizer.encode(txt)
for i in range(0, len(token_ids) - max_length, stride):
input_chunk = token_ids[i:i + max_length]
target_chunk = token_ids[i + 1:i + max_length + 1]
self.input_ids.append(torch.tensor(input_chunk))
self.target_ids.append(torch.tensor(target_chunk))
def __len__(self):
return len(self.input_ids)
def __getitem__(self, idx):
return self.input_ids[idx], self.target_ids[idx]
```
The `DataLoader` wraps this dataset:
`dataloader = DataLoader(dataset, batch_size=4, shuffle=True, drop_last=True, pin_memory=True)`
- `pin_memory=True`: Allocates page-locked host memory, enabling fast asynchronous DMA transfers from host RAM to GPU VRAM via `cudaMemcpyAsync`.
- `drop_last=True`: Prevents irregular batch sizes at dataset boundaries that trigger CUDA kernel recompilations.
In distributed multi-GPU training, `DistributedSampler` shards indices across ranks, ensuring each GPU worker receives non-overlapping batches while maintaining deterministic global shuffling across training epochs.
Adam Paszke et al. introduced the `Dataset` and `DataLoader` abstractions in PyTorch (2017), decoupling dataset indexing (`__getitem__`) from multi-process background worker batch prefetching (`num_workers > 0`), which remains the gold standard for deep learning pipelines.
- Dataset slices the tokens; DataLoader packs the pallets.
- Pin memory in the loader to keep GPU tensor cores saturated.
How does Autoregressive Dataset Construction: Slicing Index Boundaries operate mathematically at scale?
A token ID sequence: [502, 104, 301, 882, 1024, 77, 91, 400]:
- Learning goalMaster the algebraic and geometric derivation of Autoregressive Dataset Construction: Slicing Index Boundaries.
- Helpful foundationsFoundations of Build Llm From Scratch · Coordinate Invariance & Geometric Projections · Radford et al. (2019), Language Models are Unsupervised Multitask Learners, OpenAI
- Common trapAssuming Autoregressive Dataset Construction: Slicing Index Boundaries is an arbitrary computational trick rather than an invariant spatial law.
- Next connectionConnects upstream foundational lemmas to modern high-dimensional computational systems.
More context & mechanics
A code and tensor boundary diagram in square 1:1 format on dark slate (#0f172a) with clean borders.
Top section: Title in white sans-serif: "Autoregressive Dataset Construction: Slicing Boundaries". Subtitle in muted cyan: "Generating input-target token pairs via sliding context window of length L."
Middle section: A token ID sequence: [502, 104, 301, 882, 1024, 77, 91, 400]:
Show sliding window of context_size = 4:
Window 0:
- Input x: [502, 104, 301, 882] (tokens from index i to i + L)
- Target y: [104, 301, 882, 1024] (tokens from index i + 1 to i + L + 1)
Show target shifted by exactly 1 position to the right!
Annotate the exact indexing bounds in Python:
"x = tokens[i : i + L]"
"y = tokens[i + 1 : i + L + 1]"
Show boundary condition: Max valid starting index is i_(max) = len(tokens) - L - 1.
Bottom section: Causal principle: "At every token position, the model predicts the immediate next token without looking into the future."
Clean monospace code snippets, colored token boxes, clear boundary markers.
Every transformation preserves structural invariants while mapping coordinate spaces. When operating in high dimensions, numerical precision and computational efficiency require adhering strictly to canonical mathematical laws.
Formal derivation and peer-reviewed analysis:
Radford et al. (2019), Language Models are Unsupervised Multitask Learners, OpenAI
"Autoregressive Dataset Construction: Slicing Index Boundaries: Invariance, symmetry, and structure govern mathematical truth across continuous and discrete domains."