Statistics Is the Foundation of Reinforcement Learning
How do repeated rewards become a value estimate and then a decision?
How this lesson unfolds.
Begin with How do stochastic rewards across repeated trajectories transform into an expected value that guides an optimal policy? Use the picture, intuition and explanation together before moving on.
How do stochastic rewards across repeated trajectories transform into an expected value that guides an optimal policy?
Roll a pair of dice repeatedly in a game where odd totals yield +10 points and even totals penalize -5 points. Any single roll is erratic, but over many rounds the average payoff settles reliably at (+10 0.5) + (-5 0.5) = +2.5 points. In reinforcement learning, an agent facing uncertain state transitions computes this exact expected sum of discounted future payoffs, collapsing an unpredictable cloud of possible trajectories into a solid numerical compass that reveals the superior action.
- $$V^\pi(s) = \mathbb{E}_\pi \left[ \sum_{k=0}^\infty \gamma^k R_{t+k+1} \,\middle|\, S_t = s \right] = \sum_a \pi(a|s) \sum_{s', r} p(s', r | s, a) [r + \gamma V^\pi(s')]$$
- Learning goalFormulate the mathematical expectation mapping stochastic trajectory returns to state-value functions and policy improvements.
- Helpful foundationsRandom Variables & Distributions · Mathematical Expectation · Markov Decision Processes · Discounted Cumulative Return · Dynamic Programming (Bellman, 1957) · A Mathematical Theory of Communication (Shannon, 1948)
- Common trapImmediate reward measures single-step sensory feedback, whereas value measures the long-term expected sum of discounted future returns across all potential branches.
- Next connectionFormulate the recursive Bellman expectation equations to propagate value estimates backwards through time.
More context & mechanics
Reinforcement learning grounds sequential decision-making in probability theory. An agent interacting with an environment generates trajectory samples τ = (s₀, a₀, r₁, s₁, a₁, ṡ). Because transitions and policies can be stochastic, the discounted cumulative return = ∑_(k=0)^∞ γ^k R_(t+k+1) is a random variable possessing a probability distribution (G_t | S_t=s). Calculating the mathematical expectation E[G_t | S_t=s] contracts this noisy distribution into a single scalar: the state-value ^π(s). The agent then improves its policy π(a|s) by greedily choosing actions that maximize this expected future payoff.
In Reinforcement Learning from Human Feedback (RLHF) used for training GPT-4 and Llama-3, a reward model scores response tokens, and Proximal Policy Optimization (PPO) estimates the expected advantage function Â_t to adjust policy logits without collapsing the model into repetitive high-reward degeneracies.
Richard Bellman introduced dynamic programming and the principle of optimality in 1953 at RAND Corporation. By formalizing sequential decisions as expectations over Markovian transition matrices, Bellman connected statistical averaging directly to optimal control, providing the mathematical backbone for modern reinforcement learning.
Mnemonic: "Sample, Expect, Decide."
Traversals yield noisy returns (Sample); expectation collapses the variance into value (Expect); greedy argmax picks the winning move (Decide).