Backpropagation, one step at a time

A neural network learns by measuring how a small change to each parameter would change its error. Follow one calculation forward, then trace that sensitivity backward.

Illustrative demo data — educational model, not applicant research.

Static figure and worked values are available below. Interactive controls appear when JavaScript is available.

Static worked example and model values

1. Inputs: hold the two input values, target and parameters fixed for this example.
2. Weighted sums: multiply each input by its weight and add the hidden-unit bias.
3. Hidden activations: tanh bends each weighted sum into a nonlinear response.
4. Prediction: combine both hidden activations with output weights and bias.
5. Loss: compare the prediction with the target using half the squared error.
6. Output derivative: the loss derivative with respect to the prediction is prediction minus target.
7. Hidden derivatives: multiply the output derivative by its weight and the local tanh derivative.
8. Parameter gradients: multiply each hidden derivative by the corresponding input; shared paths add.
9. One proposed update: subtract learning rate times each gradient. The original graph remains fixed for comparison.

Inputs [1, -2]; target 0.6; hidden weighted sums [0.800, -0.800]; hidden activations [0.664, -0.664]; prediction 0.997; loss 0.079.

Parameter	Value	Loss derivative
w11	0.300	0.155
w12	-0.200	-0.311
w21	0.100	-0.111
w22	0.400	0.222
b1	0.100	0.155
b2	-0.100	-0.111
v1	0.700	0.264
v2	-0.500	-0.264
c	0.200	0.397

What this network computes

This multilayer perceptron takes two inputs, builds two hidden responses and produces one real-valued prediction. This is a regression example, so the output is not a probability. The weights are deliberately small illustrative values; this is not a trained model or a neuroscience result.

For hidden unit $j$, input $x$, weights $W$ and biases $b$:

$$z_j=\sum_i W_{ji}x_i+b_j,\qquad h_j=\tanh(z_j).$$

The output uses weights $v$ and bias $c$:

$$\hat y=\sum_jv_jh_j+c,\qquad L=\tfrac12(\hat y-y)^2.$$

The graph displays the same forward values throughout so that you can compare them with the derivatives. The step caption identifies the part currently being calculated. Expand the static example to see every parameter and its derivative together.

Why the backward pass works

Backpropagation applies the chain rule to this calculation. Write $r=\hat y-y$. At the output, $\partial L/\partial\hat y=r$. At each hidden weighted sum:

$$\delta_j=\frac{\partial L}{\partial z_j}=r\,v_j(1-h_j^2).$$

The remaining derivatives are local products:

$$\frac{\partial L}{\partial W_{ji}}=\delta_jx_i,\quad \frac{\partial L}{\partial b_j}=\delta_j,\quad \frac{\partial L}{\partial v_j}=rh_j,\quad \frac{\partial L}{\partial c}=r.$$

An input feeds both hidden units, so its derivative sums both paths: $\partial L/\partial x_i=\sum_j\delta_jW_{ji}$. Forgetting this addition is a common reason a plausible implementation gives incorrect gradients.

Try to predict the next step

Pause at the output derivative. If the prediction exceeds the target, is the derivative positive or negative? Then look at an output weight with a positive hidden activation: increasing that weight increases the prediction, so its loss derivative should have the same sign as the residual.

The last step compares the original loss with a single gradient descent update at the configured learning rate. It does not mutate the graph or run an ongoing training experiment. Repeated optimization, batches, regularization and generalization are outside this example.

How this is checked

Each of the nine parameter derivatives and both input derivatives is compared with a central finite difference. A small update must reduce the example's loss. Numerical agreement tests the implementation at selected values; the chain-rule derivation explains the computation more generally.

References

• Stanford CS231n: backpropagation and computational graphs.
• Karpathy: micrograd, an educational automatic differentiation implementation that informs my learning direction.