Satish BajagainContents

Gradient descent

Gradient descent adjusts parameters in the direction that locally reduces a chosen error. The step size matters as much as the direction.

wL(w) = w²0: 21: 1.22: 0.72
Illustrative quadratic loss: two updates with learning rate 0.2 move from w = 2 to 1.2 to 0.72.

Intuition: slope is local

On a smooth curve, the derivative tells you which direction rises at your current position. Moving the other way by a sufficiently small amount reduces the function. The same idea extends to many parameters, where the gradient collects all partial derivatives. It describes the local landscape, not a guaranteed route to the best possible solution.

The update rule

θt+1=θt−η∇θL(θt),η>0.\theta_{t+1}=\theta_t-\eta\nabla_\theta L(\theta_t),\qquad\eta>0.

The learning rate η\eta scales the update. A first-order expansion helps explain why a small step works:

L(θ−ηg)≈L(θ)−η∥g∥2,g=∇L(θ).L(\theta-\eta g)\approx L(\theta)-\eta\|g\|^2,\qquad g=\nabla L(\theta).

The omitted curvature terms matter for larger steps. This is why simply multiplying a successful learning rate by a large factor can fail.

Work through the figure

The illustrative curve is L(w)=w2L(w)=w^2, whose derivative is 2w2w. With learning rate 0.20.2, each update multiplies ww by 0.60.6. Starting from 22 gives 1.21.2, then 0.720.72, and progressively smaller losses.

For this particular quadratic, convergence occurs when ∣1−2η∣<1|1-2\eta|<1, or 0<η<10<\eta<1. At η=1\eta=1, the sign alternates without reducing magnitude. Above one, magnitudes grow. These bounds belong to this example; different curvature produces different limits.

What changes with data

Full-batch descent differentiates the loss over all chosen training examples. A minibatch estimates that direction from a subset, so successive steps are noisy. A falling training loss does not by itself establish held-out accuracy. Feature scale also changes the geometry of the optimization problem.

The update needs gradients, but does not specify how to obtain them. Backpropagation is an efficient method for differentiable computational graphs. Nondifferentiable points require conventions or suitable generalized derivatives.

Where you will see this

The final step of the backpropagation explorable computes one update and compares losses. The sleep-staging study tracks training and held-out evaluation separately.

References