Capstone 2: Continuous Regression Networks & Housing Valuation
How do neural architectures adapt from discrete probability classification to continuous unbounded regression, and how do Mean Squared Error loss and R^2 coefficient quantify performance?
How this lesson unfolds.
Classification networks categorize samples into discrete buckets. Regression networks predict continuous scalar quantities across the real number line. In this grand finale capstone, we adapt our neural engine to continuous valuation on the California Housing dataset.
How does an unconstrained linear output head project hidden representations into continuous scalar targets without non-linear saturation?
Imagine an artist painting a portrait. If you force them to use only a 10-button stamp tool, they can only stamp 10 predefined shapes (classification). In regression, you give them a continuous paintbrush with infinitely smooth stroke control, allowing them to paint any number along the real number line.
- $$\frac{\partial \mathcal{L}}{\partial \hat{y}} = \frac{1}{B}(\hat{y} - y)$$
- $$Regression networks omit the softmax layer, allowing the final affine projection y_hat = h*W + b to predict unbounded continuous real numbers, with Mean Squared Error loss generating residuals y_hat - y directly.$$
- Learning goalFormulate continuous linear output heads and derive the backpropagation gradient of Mean Squared Error loss.
- Helpful foundationsMean squared error definition · Backpropagation chain rule · Deep Learning (Goodfellow, Bengio, Courville, 2016)
- Common trapUsing Softmax for regression targets; Softmax forces outputs to sum to 1.0, making it impossible to predict continuous values like prices or temperatures.
- Next connectionEvaluate regression quality using Residuals and R² score in Screen 2.
More context & mechanics
In regression networks, the final layer is an affine transformation with an identity activation function g(z) = z:
ŷ = h^((L-1)) W^((L)) + b^((L)) ∈ R
The Mean Squared Error objective over batch size B is:
L = 1/2B ∑_(i=1)^B (ŷᵢ - yᵢ)²
Applying the chain rule, the gradient at the linear output is simply the scaled residual:
(∂ L) / (∂ ŷᵢ) = 1/B (ŷᵢ - yᵢ)
This residual propagates directly backward into the preceding hidden layer weights W^((L)).
Suppose true house price y = 2.80 (280,000) and model prediction ŷ = 3.10 (310,000):
- Residual Error: e = 3.10 - 2.80 = +0.30.
- MSE Loss: L = 1/2(0.30)² = 0.045.
- Gradient: (∂ L) / (∂ ŷ) = +0.30.
In gradient descent with α = 0.1:
ŷ_(new) ≈ ŷ - α (∂ L) / (∂ ŷ) = 3.10 - 0.1(0.30) = 3.07
The prediction immediately shifts closer toward the ground truth 2.80.
Foundational architecture for continuous estimation in robotics (joint torque prediction), climate modeling, and valuation systems.
Developed from Carl Friedrich Gauss's (1795) method of least squares and Legendre's (1805) regression principles.
No softmax, no ceiling: regression lets outputs run free across the real number line.
How do residual histograms, Root Mean Squared Error (RMSE), and the R² coefficient diagnose homoscedasticity and quantify regression predictive power?
Imagine shooting arrows at a bullseye. Measuring average distance to the center (RMSE) tells you your accuracy. But examining whether your missed shots are scattered evenly in all directions or consistently biased to the top-right tells you if your bow's sight is misaligned. Residual analysis checks for hidden systemic bias.
- $$R^2 = 1 - \frac{SS_{res}}{SS_{tot}} = 1 - \frac{\sum (y - \hat{y})^2}{\sum (y - \bar{y})^2}$$
- $$Evaluating regression requires auditing residual distributions e_i = y_i - y_hat for zero-mean normal symmetry (homoscedasticity) and computing R^2 = 1 - SS_res / SS_tot to determine the proportion of variance captured.$$
- Learning goalPerform comprehensive regression diagnostics by calculating residual distributions, RMSE, and the R² coefficient of determination.
- Helpful foundationsContinuous regression network architecture · Variance, standard deviation, and normal distributions · Correlation and Causation (Wright, 1921)
- Common trapBelieving R² cannot be negative; if a model performs worse than simply predicting the training mean, SS_(res) > SS_(tot), resulting in a negative R².
- Next connectionCONGRATULATIONS! You have completed all 34 chapters of Neural Networks From Scratch, building the foundational bedrock of AI.
More context & mechanics
Regression evaluation relies on three statistical metrics:
- Residual Error Vector: eᵢ = yᵢ - ŷᵢ. A well-calibrated model exhibits homoscedasticity: residuals are independently and identically distributed with zero mean: e ~ N(0, σ²).
- Root Mean Squared Error (RMSE):
RMSE = √(1/N ∑_(i=1)^N (yᵢ - ŷᵢ)²)
Measures average prediction error in original target units (100,000s of dollars).
- Coefficient of Determination (R²):
R² = 1 - (SS_(res)) / (SS_(tot)) = 1 - (∑_(i=1)^N (yᵢ - ŷᵢ)²) / (∑_(i=1)^N (yᵢ - ȳ)²)
R² measures the percentage of variance explained by the model compared to a baseline predicting the mean ȳ.
Suppose test evaluation across N = 4,000 California houses yields:
- Total Sum of Squares: SS_(tot) = 4,200.0 (Variance around dataset mean price ȳ = \206,000).
- Residual Sum of Squares: SS_(res) = 756.0 (Remaining unexplained variance after neural prediction).
- Compute R² Score:
R² = 1 - (756.0) / (4200.0) = 1 - 0.180 = 0.820
The deep neural network explains 82% of all price fluctuations across California districts!
- Compute RMSE:
RMSE = √((756.0) / (4000)) = √(0.189) = 0.4347 ⟹ \43,470
Average price prediction error is \43,470 on houses averaging \206,000.
Standard metric suite reported in real estate valuation, demand forecasting, and tabular Kaggle competitions.
Sewall Wright (1921) introduced the coefficient of determination in path analysis, connecting regression residuals with explained variance.
One minus unexplained over total: R² measures the percentage of reality your network has mastered.