Backpropagation

Backpropagation tells us how sensitive an error is to the quantities used to compute it. It reuses local derivatives, working backward from the error to the parameters.

Intuition: follow the dependency

Suppose a parameter changes a hidden activation, which changes a prediction, which changes a loss. Its influence is the product of those local sensitivities. If it influences the loss through several paths, add their contributions. The graph above is a map of these dependencies, not a physical signal travelling through a brain.

The chain rule, made reusable

For an intermediate value $z$ with downstream values $u_j$,

$$\frac{\partial L}{\partial z}=\sum_j\frac{\partial L}{\partial u_j}\frac{\partial u_j}{\partial z}.$$

A forward pass stores the values needed by these local derivatives. A backward pass visits operations in reverse dependency order. In a multilayer perceptron with $h_j=\tanh(z_j)$ and output $\hat y=\sum_jv_jh_j+c$, squared-error loss gives

$$\frac{\partial L}{\partial z_j}=(\hat y-y)v_j(1-h_j^2).$$

Multiplying this by input $x_i$ gives the derivative for weight $W_{ji}$. Notice the three factors: error at the output, output weight, and activation slope. A small slope can make a parameter's gradient small even when the prediction error is large.

A small check you can do yourself

In an illustrative scalar calculation, let $q=wx$, $L=q^2/2$, $w=2$ and $x=3$. Then $q=6$, $L=18$, and $dL/dw=q x=18$. Directly writing $L=9w^2/2$ gives the same answer. A numerical check evaluates $[L(w+\epsilon)-L(w-\epsilon)]/(2\epsilon)$; very small $\epsilon$ can introduce floating-point cancellation.

Backpropagation computes derivatives. Gradient descent chooses an update using them. These are distinct operations; a correct derivative does not guarantee a good learning rate, useful data or generalization.

Where you will see this

Use the step-by-step explorable to inspect all nine parameter gradients. The planned GPT learning capstone will build on the same differentiation idea at a larger scale.

References

• Stanford CS231n: backpropagation.
• Karpathy's micrograd implementation.