Neural Networks from Scratch

A neural network is a function approximator. It takes an input, passes it through a series of mathematical transformations, and produces an output. The goal is to find transformations that map inputs to the correct outputs — and the entire machinery of deep learning is built around doing that efficiently at scale.

The Building Block: A Single Neuron

A single neuron receives a vector of inputs, multiplies each input by a corresponding weight, sums everything together, adds a bias term, and then passes the result through an activation function:

output = activation(w1*x1 + w2*x2 + ... + wn*xn + b)

The weights determine how much each input contributes. The bias shifts the output up or down independently of the inputs. The activation function introduces non-linearity — without it, stacking layers would be mathematically equivalent to a single linear transformation, no matter how many layers you add.

Activation Functions

The most widely used activation today is ReLU (Rectified Linear Unit):

ReLU(x) = max(0, x)

It is zero for negative inputs and linear for positive ones. Simple, computationally cheap, and it avoids the vanishing gradient problem that plagued earlier activations like sigmoid and tanh. Variants like Leaky ReLU and GELU address the "dying ReLU" problem where neurons output zero for all inputs and stop learning.

For the output layer, the activation depends on the task. Sigmoid squashes output to (0, 1) for binary classification. Softmax converts a vector of raw scores into a probability distribution across multiple classes.

The Forward Pass

A neural network is organized into layers. The forward pass is the computation that flows from input to output:

import numpy as np

def relu(x):
    return np.maximum(0, x)

def softmax(x):
    e = np.exp(x - np.max(x))
    return e / e.sum()

# One hidden layer network
def forward(X, W1, b1, W2, b2):
    Z1 = X @ W1 + b1      # linear transform
    A1 = relu(Z1)          # activation
    Z2 = A1 @ W2 + b2     # output layer
    A2 = softmax(Z2)       # probabilities
    return A2

Each layer takes the previous layer's activations as input, applies a learned linear transformation, then passes the result through an activation function.

The Loss Function

After the forward pass produces a prediction, we need to measure how wrong it is. The loss function does this. For classification, cross-entropy loss is standard:

loss = -sum(y_true * log(y_predicted))

For regression, mean squared error is typical:

loss = mean((y_true - y_predicted) ** 2)

The loss is a single number. The entire goal of training is to minimize it.

Backpropagation

To minimize the loss, we need to know how each weight in the network contributed to it — specifically, how the loss changes as we adjust each weight. This is the gradient. Backpropagation computes it efficiently using the chain rule of calculus.

The chain rule says that the gradient of the loss with respect to an early layer's weights is the product of gradients through all the layers that follow it. Backpropagation works by starting at the output layer, computing the gradient there, and then propagating it backward layer by layer — hence the name.

# Gradient of loss w.r.t. output layer weights
dL_dW2 = A1.T @ dL_dZ2

# Propagate gradient back through activation
dL_dA1 = dL_dZ2 @ W2.T
dL_dZ1 = dL_dA1 * (Z1 > 0)  # ReLU derivative

# Gradient of loss w.r.t. hidden layer weights
dL_dW1 = X.T @ dL_dZ1

Weight Updates

Once we have the gradients, we update the weights in the direction that reduces the loss. This is gradient descent:

W1 = W1 - learning_rate * dL_dW1
W2 = W2 - learning_rate * dL_dW2

The learning rate controls how large each update step is. Too large and training overshoots and diverges. Too small and training is painfully slow. In practice, adaptive optimizers like Adam adjust the learning rate per parameter automatically, which is why they work so much better than vanilla gradient descent in most settings.

Why Depth Matters

The universal approximation theorem says that a single hidden layer network can approximate any continuous function given enough neurons. But in practice, deeper networks learn hierarchical representations that shallower networks cannot — at least not without an impractical number of neurons. Each layer in a deep network learns increasingly abstract features: edges become shapes, shapes become objects, objects become concepts. That compositional structure is what makes deep learning so powerful for tasks like vision and language.