Neural Networks

Neural networks often become intimidating at exactly the wrong moment: forward propagation feels manageable, then backpropagation arrives as a page of symbols and cached tensors. The way through is to follow one small network from its input to a prediction, then walk the same path backward without skipping a shape.

Underneath the name, a neural network is a parameterized function built from linear transformations and nonlinear activations. This article uses a small dense network, and the code implements the same equations shown in the text.

1. The Basic Structure

For a dense layer with input matrix $A^{[l-1]}$, weights $W^{[l]}$, and bias $b^{[l]}$:

Rows usually represent examples. The input layer holds features, hidden layers learn intermediate representations, and the output layer is chosen to match the prediction task.

The activation function matters because a stack of linear layers without nonlinear activations collapses into one linear transformation.

2. Forward Propagation

Consider binary classification with one ReLU hidden layer and a sigmoid output:

The sigmoid keeps $P$ between 0 and 1. ReLU is suitable inside the network but not as the probability-producing output for binary cross-entropy.

For mutually exclusive multiclass classification, the output is normally a vector of logits trained with softmax cross-entropy. For regression, a linear output is a common starting point.

3. Binary Cross-Entropy

For labels $y_i\in\{0,1\}$ and predicted probabilities $p_i$:

In code, calculate cross-entropy from logits with a stable library operation when possible. Directly taking log(p) can produce infinities after rounding pushes a probability to exactly 0 or 1.

An equivalent stable expression for one logit $z$ is:

Libraries implement a numerically stable form of this expression.

4. Backpropagation

Backpropagation applies the chain rule from the output toward the input. For sigmoid plus binary cross-entropy, the derivative with respect to the output logit simplifies to:

The remaining gradients are:

Every layer needs the activation or pre-activation values required by its derivative. Reusing the same input and output error for every layer skips the chain rule and usually causes shape errors.

5. A Working NumPy Example

The following network learns the XOR pattern. It is intentionally small enough to inspect.

import numpy as np

rng = np.random.default_rng(7)

X = np.array([
    [0.0, 0.0],
    [0.0, 1.0],
    [1.0, 0.0],
    [1.0, 1.0],
])
y = np.array([[0.0], [1.0], [1.0], [0.0]])

hidden_units = 8
W1 = rng.normal(0, np.sqrt(2 / X.shape[1]), (X.shape[1], hidden_units))
b1 = np.zeros((1, hidden_units))
W2 = rng.normal(0, np.sqrt(2 / hidden_units), (hidden_units, 1))
b2 = np.zeros((1, 1))

def sigmoid(z):
    # This form is adequate for the small example; production libraries
    # provide carefully tested stable implementations.
    return 1.0 / (1.0 + np.exp(-np.clip(z, -50, 50)))

def forward(x):
    z1 = x @ W1 + b1
    a1 = np.maximum(0.0, z1)
    z2 = a1 @ W2 + b2
    probabilities = sigmoid(z2)
    return probabilities, (x, z1, a1, z2)

learning_rate = 0.1

for step in range(5_000):
    probabilities, (x_batch, z1, a1, logits) = forward(X)
    batch_size = X.shape[0]

    # Stable binary cross-entropy computed from logits.
    loss = np.mean(np.logaddexp(0.0, logits) - y * logits)

    dz2 = (probabilities - y) / batch_size
    dW2 = a1.T @ dz2
    db2 = np.sum(dz2, axis=0, keepdims=True)

    da1 = dz2 @ W2.T
    dz1 = da1 * (z1 > 0)
    dW1 = x_batch.T @ dz1
    db1 = np.sum(dz1, axis=0, keepdims=True)

    W1 -= learning_rate * dW1
    b1 -= learning_rate * db1
    W2 -= learning_rate * dW2
    b2 -= learning_rate * db2

    if step % 1_000 == 0:
        print(step, float(loss))

predictions, _ = forward(X)
print(np.column_stack((X, predictions, predictions >= 0.5)))

This example uses the whole four-row dataset for every update. Real training normally uses shuffled mini-batches, a validation set, and an optimizer implemented by a library.

6. Initialization and Optimization

Weights should not all start at zero because hidden units would receive identical gradients and remain symmetric. The example uses a He-style scale that works well with ReLU. Xavier/Glorot initialization is common with tanh or sigmoid-like activations.

Plain gradient descent is enough to show the math. In larger models, SGD with momentum, Adam, or AdamW are common choices. The optimizer does not repair an incompatible output activation, incorrect loss, or broken gradient calculation.

7. Regularization

Regularization aims to improve performance on unseen data:

• L2 penalty: adds $\lambda\lVert W\rVert_2^2$ to the loss.
• Weight decay: directly shrinks weights during an update. It matches an L2 penalty for some plain SGD formulations but is not generally identical with adaptive optimizers; AdamW makes the distinction explicit.
• Dropout: randomly masks activations during training and is disabled during evaluation.
• Early stopping: stops when validation performance no longer improves.
• Data augmentation: creates plausible training variations without changing the target.

More regularization is not always better. It can prevent the model from fitting even the training data, so compare training and validation curves.

8. Checking an Implementation

A useful debugging order is:

1. verify every tensor shape;
2. overfit a tiny dataset;
3. compare analytical gradients with finite differences on a small network;
4. check that the loss falls after one update;
5. separate training behavior from evaluation behavior;
6. inspect for NaN, infinity, and saturated activations.

Conclusion

A dense neural network is a chain of functions. Forward propagation evaluates the chain; backpropagation differentiates it in reverse. Once those two pieces are correct, optimizers and regularizers can improve training. If the chain rule or output distribution is wrong, no amount of tuning will rescue the model.