Coding Neurons

The artificial neuron (Perceptron) is the foundational atomic building block of all neural architectures:

1. Biological Analogy:

• Dendrites: Input channels carrying signals x₁, x₂, ṡ, xₙ.
• Synapses: Trainable connection strengths w₁, w₂, ṡ, wₙ that amplify or attenuate inputs.
• Soma (Cell Body): The central summation junction that computes net accumulated charge.
• Axon: Output channel transmitting activation z to downstream cells.

1. The Affine Summation Junction:

The core computational step is an affine transformation:

z = ∑_(i=1)^n wᵢ xᵢ + b = w^T x + b

Where w ∈ R^n is the weight vector, x ∈ R^n is the input feature vector, and b ∈ R is the scalar bias.

1. The Role of Bias (b):

Without bias (b=0), the decision boundary is forced to pass through the origin (0, 0, ṡ, 0). The bias acts as a learnable threshold offset, shifting the hyperplane anywhere in the input space to separate classes cleanly.

While modern deep learning connects millions of these units into deep multi-layer networks with non-linear activations, the fundamental arithmetic of every single neuron remains the affine sum w^T x + b.

Warren McCulloch and Walter Pitts introduced the first mathematical model of a biological neuron in 1943. In 1957, Frank Rosenblatt created the Perceptron at the Cornell Aeronautical Laboratory, implementing it in hardware on the custom-built Mark I Perceptron computer.

• Weights scale, soma sums, bias shifts: Multiply inputs by importance, add them together, adjust the threshold.
• Affine before active: z = w^T x + b is the universal linear heartbeat of machine learning.

#### 1. Mathematical Formulation

Let input feature vector be x = [x₁, x₂, ṡ, xₙ]^T ∈ R^n.

Let synaptic weight vector be w = [w₁, w₂, ṡ, wₙ]^T ∈ R^n.

Let bias be scalar b ∈ R.

The affine scalar output z is:

z = w · x + b = ∑_(i=1)^n wᵢ xᵢ + b

In vector-matrix notation:

z = w^T x + b

The affine transformation can be expressed as a pure linear dot product by augmenting the input and weight vectors with a dummy coordinate:

x̃ = [x₁, x₂, ṡ, xₙ, 1]^T ∈ R^(n+1)

w̃ = [w₁, w₂, ṡ, wₙ, b]^T ∈ R^(n+1)

Then:

z = w̃^T x̃ = ∑_(i=1)^n wᵢ xᵢ + b · 1 = w^T x + b

This mathematical trick allows hardware accelerators to treat bias addition as part of a single unified matrix multiply.

Given 3 input features:

• Inputs: x = [1.2, -0.5, 2.0]
• Weights: w = [0.4, 0.8, -1.5]
• Bias: b = 0.75

Step 1: Component-wise Multiplications:

• w₁ x₁ = 0.4 × 1.2 = 0.48
• w₂ x₂ = 0.8 × (-0.5) = -0.40
• w₃ x₃ = -1.5 × 2.0 = -3.00

Step 2: Summation:

∑_(i=1)³ wᵢ xᵢ = 0.48 + (-0.40) + (-3.00) = 0.08 - 3.00 = -2.92

Step 3: Bias Addition:

z = -2.92 + 0.75 = -2.17

The pre-activation output of the neuron is exactly z = -2.17.

In NumPy and PyTorch, this calculation is executed via optimized BLAS dot-product calls:

```python

z = np.dot(w, x) + b # CPU / GPU vector instruction

```

On modern CPUs, SIMD instructions (AVX-512) perform all multiplications and the addition across 16 channels in a single clock cycle.

The algebraic dot product w · x was formalized by Josiah Willard Gibbs in the 1880s as the scalar projection of one vector onto another. In neural networks, Frank Rosenblatt's 1958 Perceptron formalized the forward arithmetic trace.

• Multiply pairs, sum, add bias: The three-beat arithmetic of the neuron forward pass.
• z = w^T x + b: Simple high school algebra powers the most advanced frontier models.

Scaling from an isolated neuron to a neural layer represents the transition from vector-vector arithmetic to matrix-vector linear algebra:

1. The Parallel Committee Architecture:

Consider an input feature vector x ∈ R^N (e.g. N=4 input features).

Rather than training a single neuron, we instantiate a layer of M independent neurons (e.g. M=3 neurons).

• Neuron 1 has its own private weight vector w^((1)) and bias b₁.
• Neuron 2 has its own private weight vector w^((2)) and bias b₂.
• Neuron M has its own private weight vector w^((M)) and bias b_M.

1. Dense All-to-All Connectivity:

Every input connects to every neuron. The total number of synaptic connections is N × M.

1. Geometric Dimensionality Transformation:

The layer acts as a linear projection operator that maps vectors from space R^N to space R^M:

f: R^N → R^M

If M > N, the layer expands features into a higher-dimensional space (enriching capacity); if M < N, it compresses features into a bottleneck space (dimensionality reduction).

Dense fully-connected layers (`nn.Linear` in PyTorch) are the backbone of classification heads, projection adapters, and transformer feed-forward blocks. Modern foundation models stack dozens of dense layers with dimensions scaling from d=4096 to d=14336.

Multi-layer neural networks were proposed in the 1960s, but lacked efficient training methods until the popularization of Backpropagation by Rumelhart, Hinton, and Williams in 1986. Organizing neurons into parallel matrix layers made neural networks natively compatible with digital matrix coprocessors.

• Rows are neurons; columns are inputs: The weight matrix stacks the neurons vertically.
• N inputs in, M outputs out: A layer transforms coordinates from N-dimensional space to M-dimensional space.

#### 1. Mathematical Formulation

Let input vector be x = [x₁, x₂, ṡ, x_N]^T ∈ R^N.

Let a layer have M neurons. Stacking each neuron's weight vector w^((j)T) as row j forms weight matrix W ∈ R^(M × N):

W =

w_(1, 1) & w_(1, 2) & ṡ & w_(1, N) \

w_(2, 1) & w_(2, 2) & ṡ & w_(2, N) \

… & … & ⋱ & … \

w_(M, 1) & w_(M, 2) & ṡ & w_(M, N)

, b = b₁ \ b₂ \ … \ b_M

The complete layer forward pass is the matrix-vector equation:

z = W x + b ∈ R^M

Where the output of neuron j is the dot product of row j with input x, plus bias bⱼ:

zⱼ = (W)_(j, :) · x + bⱼ = ∑_(i=1)^N W_(j, i) xᵢ + bⱼ

Let N = 4 inputs and M = 3 neurons.

Input vector: x = [1.0, 2.0, 3.0, 2.5]^T.

Weight matrix W (3 × 4) and bias vector b (3 × 1):

W =

0.2 & 0.8 & -0.5 & 1.0 \

0.5 & -0.91 & 0.26 & -0.5 \

-0.26 & -0.27 & 0.17 & 0.87

, b = 2.0 \ 3.0 \ 0.5

• Neuron 1 Output (z₁):

z₁ = (0.2)(1.0) + (0.8)(2.0) + (-0.5)(3.0) + (1.0)(2.5) + 2.0

z₁ = 0.2 + 1.6 - 1.5 + 2.5 + 2.0 = 4.8

• Neuron 2 Output (z₂):

z₂ = (0.5)(1.0) + (-0.91)(2.0) + (0.26)(3.0) + (-0.5)(2.5) + 3.0

z₂ = 0.5 - 1.82 + 0.78 - 1.25 + 3.0 = 1.21

• Neuron 3 Output (z₃):

z₃ = (-0.26)(1.0) + (-0.27)(2.0) + (0.17)(3.0) + (0.87)(2.5) + 0.5

z₃ = -0.26 - 0.54 + 0.51 + 2.175 + 0.5 = 2.385

Final output vector:

z = 4.8 \ 1.21 \ 2.385 ∈ R³

In deep learning frameworks, weights are stored as `[M, N]` matrices. When processing a batch of B samples (X ∈ R^(B × N)), the formulation transposes to Z = X W^T + b, allowing hardware GEMM units to process all samples and neurons in parallel.

Matrix-vector multiplication was formalized in Arthur Cayley's 1858 memoir on the theory of matrices. Modern GPUs execute these operations using systolic arrays and tensor cores designed specifically for dense matrix multiply-accumulate (MAC) throughput.

• Row times column plus bias: Each row of W produces one element of output z.
• [M × N] × [N × 1] = [M × 1]: Inner dimensions must match; outer dimensions define the output.

Let batch data be organized as a matrix X ∈ R^B × d_(in), where each row is an independent sample. Let a neural layer contain d_(out) neurons, each defined by weight vector wⱼ ∈ R^d_(in). Stacking neuron weights row-wise forms the parameter matrix:

W = w₁^T \ w₂^T \ … \ w_d_(out)^T ∈ R^d_(out) × d_(in)

By the rules of matrix algebra, matrix product A B is defined only when the number of columns in A equals the number of rows in B. Here, X has d_(in) columns, but W has d_(out) rows. Therefore, computing X W is undefined.

Transposing W yields W^T ∈ R^d_(in) × d_(out), aligning the inner contraction dimension:

Z = X W^T + b ⟹ (B × d_(in)) × (d_(in) × d_(out)) + (1 × d_(out)) = (B × d_(out))

Each output entry Z_(ij) = ∑_(k=1)^d_(in) X_(ik) W_(jk) + bⱼ = xᵢ^T wⱼ + bⱼ represents the j-th neuron evaluated on the i-th batch sample.

Modern hardware accelerators (NVIDIA H100 Tensor Cores, Google TPUs) execute batched linear transformations via highly optimized cuBLAS/CUTLASS GEMM routines. Formatting batches as row-major matrices (B, d_(in)) allows SIMD memory controllers to burst consecutive feature dimensions across memory lanes with optimal coalesced memory access.

Matrix multiplication and transposition were formalized by British mathematician Arthur Cayley in 1857. In 2012, Alex Krizhevsky's cuda-convnet demonstrated that mapping neural network layers directly onto GPU GEMM routines provided a 50x speedup over CPU implementations, sparking the modern deep learning revolution.

• Golden Axiom: "Rows meet columns, inner shapes agree: (B, d_(in)) W^T unlocks batch efficiency."
• Remember: Store weights as (d_(out), d_(in)); transpose to (d_(in), d_(out)) so batch dimension B passes untouched.

1. The Linear Perceptron Decision Boundary:

A single neuron classifies inputs using the sign of an affine combination: f(x₁, x₂) = step(w₁ x₁ + w₂ x₂ + b). The boundary separating positive from negative predictions is the straight line w₁ x₁ + w₂ x₂ + b = 0.

1. The XOR Contradiction Proof:

The XOR truth table demands four simultaneous conditions:

• Input (0, 0) ⟹ 0: w₁(0) + w₂(0) + b < 0 ⟹ b < 0.
• Input (1, 0) ⟹ 1: w₁(1) + w₂(0) + b ≥ 0 ⟹ w₁ + b ≥ 0.
• Input (0, 1) ⟹ 1: w₁(0) + w₂(1) + b ≥ 0 ⟹ w₂ + b ≥ 0.
• Input (1, 1) ⟹ 0: w₁(1) + w₂(1) + b < 0 ⟹ w₁ + w₂ + b < 0.

Adding the second and third inequalities yields: (w₁ + b) + (w₂ + b) ≥ 0 ⟹ w₁ + w₂ + 2b ≥ 0.

Subtracting the fourth inequality (w₁ + w₂ + b < 0) gives: b > 0.

This contradicts the first condition b < 0! No real coefficients (w₁, w₂, b) can satisfy all four conditions simultaneously.

1. The Multi-Layer Solution:

By placing two neurons in a hidden layer equipped with non-linear activations (e.g. ReLU or Step), neuron h₁ computes x₁ ∨ x₂ (OR) and neuron h₂ computes \neg(x₁ ∧ x₂) (NAND). The output neuron computes h₁ ∧ h₂. The hidden layer transforms the original non-separable 2D coordinates into a new feature space where (0, 0) and (1, 1) map to linearly separable positions.

This geometric necessity of non-linearity is why every modern deep network interleaves linear projections (matrix multiplications) with non-linear activation layers (ReLU, GELU, SwiGLU). In Transformer MLP blocks and convolutional residual blocks, non-linear activations allow networks to assemble complex decision manifolds capable of language understanding and visual perception.

Frank Rosenblatt invented the Perceptron in 1957. In 1969, MIT mathematicians Marvin Minsky and Seymour Papert published Perceptrons, rigorously proving this single-layer XOR limitation. Because training multi-layer networks was then unsolved, funding evaporated, beginning the first "AI Winter". The impasse lasted until 1986, when Rumelhart, Hinton, and Williams demonstrated backpropagation for training multi-layer non-linear networks.

• Golden Axiom: "One straight line cannot separate diagonal corners: non-linearity bends space so XOR falls into place."
• Remember: Stacking pure linear layers is just one big linear layer; activation functions are the folding creases that give depth its power.

A single artificial neuron computes a linear combination of input vector x ∈ R^n:

z = ∑_(i=1)^n wᵢ xᵢ + b = w^T x + b

Where:

• w ∈ R^n is the weight vector (hyperplane normal).
• b ∈ R is the scalar bias (orthogonal offset from origin).
• The decision boundary separating positive and negative outputs is the hyperplane:

H = x ∈ R^n | w^T x + b = 0

Passing z through an activation σ(z) introduces non-linearity:

a = σ(w^T x + b)

Every dense linear layer `torch.nn.Linear(in_features, out_features)` is simply an array of m individual neurons computing z = W x + b simultaneously via matrix-vector multiplication.

Warren McCulloch and Walter Pitts introduced the mathematical threshold neuron in 1943. Frank Rosenblatt implemented it physically in 1958 as the Mark I Perceptron at Cornell Aeronautical Laboratory.

• Weights tilt the plane; bias shifts the threshold.
• y = mx + b is neuron 101.

Object-oriented formulation of a fully connected neural layer in pure NumPy:

```python

import numpy as np

class Layer_Dense:

def __init__(self, n_inputs, n_neurons):

# Initialize weights with small random Gaussian values

# Shape: (n_inputs, n_neurons) avoids needing W.T during forward pass!

self.weights = 0.01 * np.random.randn(n_inputs, n_neurons)

# Initialize biases to zeros

# Shape: (1, n_neurons) enables seamless broadcasting across batch rows

self.biases = np.zeros((1, n_neurons))

def forward(self, inputs):

# Cache inputs for backward gradient pass

self.inputs = inputs

# Linear affine transformation: (batch_size, n_neurons)

self.output = np.dot(inputs, self.weights) + self.biases

```

Key architectural design decisions:

1. Weight Shape `(n_inputs, n_neurons)`: Storing weights with input dimensions on rows and neurons on columns allows direct matrix multiplication `inputs @ weights` without transposing `weights` on every forward pass.
2. Bias Shape `(1, n_neurons)`: Storing biases as a 2D row vector triggers NumPy broadcasting across the batch dimension without memory duplication.

This is the foundational object design behind PyTorch's `nn.Linear` and TensorFlow's `tf.keras.layers.Dense`.

Harrison Kinsley and Daniel Kukieła codified this pure-NumPy OOP structure in their 2020 masterwork 'Neural Networks from Scratch in Python' (NNFS).

• Inputs dot Weights plus Biases.
• Shape is (n_inputs, n_neurons) so you don't have to transpose.

A biological and computational neuron diagram in square 1:1 format on warm cream parchment (#fcf0db) with an 8px border.

Top section: Title in navy serif: "The Artificial Neuron: Weighted Sum and Bias". Subtitle: "Frank Rosenblatt (1958): Modeling biological dendrites as a linear dot product."

Middle section: A circular central neuron processing unit:

Inputs arriving from left:

• Input x₁ × Weight w₁
• Input x₂ × Weight w₂
• Input x₃ × Weight w₃

Summation Node: Σ = ∑_(i=1)³ xᵢ wᵢ + b = x^\top w + b.

Add scalar bias b (shifting the activation threshold).

Activation Gate: Passes Σ through non-linear activation σ(Σ) to produce final output y.

Show Python code snippet:

Every transformation preserves structural invariants while mapping coordinate spaces. When operating in high dimensions, numerical precision and computational efficiency require adhering strictly to canonical mathematical laws.

Formal derivation and peer-reviewed analysis:

Goodfellow, Bengio, and Courville (2016), Deep Learning, MIT Press ↗

"Coding Neurons from Scratch: The Perceptron Dot Product: Invariance, symmetry, and structure govern mathematical truth across continuous and discrete domains."