Why a shared value needs gradient accumulation

A value can affect an answer more than once. If we want to know how changing that value changes the answer, we have to account for every path. This is the small implementation detail that makes += essential in an autograd engine.

This note develops the shared-node example in my micrograd notebook. For the broader procedure, start with backpropagation.

One value, two paths

Illustrative worked example. Let $x=3$ and compute $q=x\cdot x$, then $y=q+x$. The forward values are $q=9$ and $y=12$.

The direct path from $x$ to $y$ contributes $1$. The path through $q$ contributes $2x$. Together,

$$\frac{dy}{dx}=\frac{dy}{dq}\frac{dq}{dx}+1=1\cdot 2x+1=7.$$

At the multiplication node, the left and right operands happen to be the same object. Each operand position contributes the other operand's value multiplied by the upstream gradient. Both contributions must reach that object.

Accumulate locally; visit nodes once

A backward rule adds its contribution to an input's existing gradient. Assigning a new value would discard contributions from other paths. That includes repeated operands within one operation, not just branches created by different operations.

The traversal has a different responsibility. A topological ordering visits each node once, after all downstream contributions have arrived during the reverse pass. Visiting a shared node's backward function repeatedly would propagate accumulated contributions more than once.

These two rules work together: accumulate contributions along edges, then propagate the total from each node once.

Check the derivative independently

For this illustrative polynomial, a centered finite difference gives

$$\frac{f(3+h)-f(3-h)}{2h}=7,\qquad f(x)=x^2+x.$$

The algebra is exact for any nonzero $h$; floating-point computation introduces rounding error, so making $h$ arbitrarily small is not always helpful. A numerical check is a diagnostic, not a proof that every backward rule is correct.

Reset between training steps

Accumulation within one backward pass is necessary. Accumulation across independent training steps is a separate choice. For ordinary gradient descent, clear the parameter gradients before the next backward pass. A new forward pass creates a new graph; retaining old parameter gradients would mix derivatives from different steps.

Step through a small network and compare its paths with this example. Karpathy's micrograd engine is the implementation reference for gradient accumulation and reverse topological traversal.