Differentiation Does Not Destroy Probability: CDFs, PDFs, and Conservation of Mass
How does differentiating a Cumulative Distribution Function (CDF) preserve total probability while converting cumulative mass into local density?
How this lesson unfolds.
Begin with Why does taking the derivative of a CDF change the mathematical representation from mass to density without losing a single drop of probability? Use the picture, intuition and explanation together before moving on.
Why does taking the derivative of a CDF change the mathematical representation from mass to density without losing a single drop of probability?
Think of a CDF as a snowdrift measuring the total depth of fallen snow up to mile marker x. Differentiation does not shovel away or destroy the snow—it simply measures the steepness (how hard it is currently snowing per foot). By the Fundamental Theorem of Calculus, integrating that snow-rate (PDF) from start to finish reconstructs the exact original total depth: 100% of the snow.
- $$f(x) = F'(x) \implies \int_{-\infty}^{\infty} f(x) \, dx = \lim_{x \to \infty} F(x) - \lim_{x \to -\infty} F(x) = 1 - 0 = 1$$
- $$Differentiating a Cumulative Distribution Function F(x) yields the Probability Density Function f(x) = F'(x); this operation converts cumulative accumulated probability into localized probability density while rigorously conserving total probability mass equal to 1.$$
- Learning goalProve differentiating a CDF yields a valid unit-integrating PDF via calculus.
- Helpful foundationscalculus differentiation · Fundamental Theorem of Calculus · cumulative distribution function · Foundations of the Theory of Probability (Andrey Kolmogorov, 1933) · An Essay towards solving a Problem in the Doctrine of Chances (Thomas Bayes, 1763) · Essence of Calculus (3Blue1Brown)
- Common trapExpecting differentiation to destroy probability; differentiation merely converts cumulative mass to density.
- Next connectionExplore how limits govern differentiation and integration simultaneously.
More context & mechanics
In continuous probability theory, a random variable X is described through two dual representations:
- The Cumulative Distribution Function (CDF, F(x)): Stores probability cumulatively from -∞ up to x:
F(x) = P(X ≤ x) = ∫_(-∞)^x f(t) dt
Properties: F(x) is non-decreasing, lim_(x → -∞) F(x) = 0, and lim_(x → ∞) F(x) = 1.
- The Probability Density Function (PDF, f(x)): Measures the instantaneous concentration of probability at point x:
f(x) = F'(x) = d/dx F(x)
Concrete Example (Waiting Time in a Poisson Process):
Consider an exponential random variable X ~ Exp(λ) representing waiting time:
- CDF: F(x) = 1 - e^(−λ x) for x ≥ 0. Total probability is stored monotonically, reaching F(∞) = 1 - 0 = 1.
- Differentiation: Take the derivative with respect to x:
f(x) = d/dx (1 - e^(−λ x)) = 0 - (-λ e^(−λ x)) = λ e^(−λ x), x ≥ 0
- Total Probability Conservation: By the Fundamental Theorem of Calculus:
∫₀^∞ f(x) dx = ∫₀^∞ λ e^(−λ x) dx = [ -e^(−λ x) ]₀^∞ = 0 - (-1) = 1
Differentiation merely changes the perspective from cumulative volume to local slope; every ounce of probability mass is conserved.
In generative deep learning (Normalizing Flows and Continuous-Time Diffusion Models), models transform simple base probability distributions (like isotropic Gaussians) into complex data distributions (images, audio) via invertible differentiable maps y = g(x). The change of variables formula:
p_Y(y) = p_X(g^(-1)(y)) · | det ( (∂ g^(-1)) / (∂ y) ) |
relies directly on Jacobian differentiation preserving total probability measure ∫ p_Y(y) dy = 1. Without exact density conservation under differentiation, generative models would suffer from mode collapse and probability explosion.
The Fundamental Theorem of Calculus was developed independently by Isaac Newton and Gottfried Wilhelm Leibniz in the late 17th century. In 1904, Henri Lebesgue generalized differentiation and integration to arbitrary measure spaces, proving the Radon-Nikodym theorem: the PDF f(x) is the Radon-Nikodym derivative of the probability measure with respect to the Lebesgue measure: f = (dP) / (dλ).
The Odometer and Speedometer Rule:
The CDF is your odometer (tells you total miles traveled: 0 to 1). The PDF is your speedometer (tells you speed right now: derivative). Taking the derivative changes your dashboard from miles to mph, but you still traveled the exact same journey!