Why the softmax gradient is prediction minus target

A classifier assigns probabilities to possible answers. Cross-entropy penalizes it when the correct answer receives little probability. Its derivative has a useful form: the predicted probability minus the target, scaled by how the loss is averaged.

This note develops the manual derivatives in my makemore backpropagation notebook. See backpropagation for the chain-rule foundation.

Derive it for one example

Let $z_j$ be a logit and let the correct class be $y$. Softmax and negative log-likelihood give

$$p_j=\frac{e^{z_j}}{\sum_k e^{z_k}},\qquad \ell=-\log p_y=-z_y+\log\sum_k e^{z_k}.$$

Differentiating with respect to $z_j$ yields

$$\frac{\partial\ell}{\partial z_j}=p_j-\mathbf1[j=y].$$

The indicator is one only at the target class. For mean loss over a batch of $B$ equally weighted examples, divide each row's gradient by $B$. Summed loss, class weighting, ignored targets and other reductions require their own scaling.

An illustrative row

For illustrative probabilities $(0.2,0.5,0.3)$ with the middle class correct, the single-example logit gradient is $(0.2,-0.5,0.3)$. The entries sum to zero. Adding a constant to all logits leaves the probabilities unchanged, so the loss has no slope in that common-shift direction.

A negative target-logit gradient means a gradient-descent update increases that logit. The other logits decrease in this isolated logit-space picture; changes to shared network parameters can couple examples and classes.

Avoid a tensor-to-scalar shortcut

Only selected entries of the log-probability matrix contribute directly to the loss. Their derivative is $-1/B$; the others are zero. Replacing that whole matrix with a scalar $-1/B$ would incorrectly send gradients into every entry. Shapes and indexing are part of the derivative.

For stable computation, subtract the largest logit before exponentiation or use a fused log-softmax/cross-entropy operation. The mathematical derivative is unchanged, but overflow is avoided. A centered finite-difference check on moderate logits helps detect missing batch factors and wrong target indices.

The source exercise is Karpathy's makemore backpropagation lesson; micrograd's implementation illustrates how local derivative rules compose.