Batch normalization does not make a distribution Gaussian

Subtracting a mean and dividing by a standard deviation changes a set of values' location and scale. It does not change an arbitrary distribution into a bell curve. That distinction corrects an overstatement in my batch-normalization notebook.

This note concerns the hidden activations of an MLP, not biological neurons' firing rates.

The operation for one feature

For a batch of $m$ values, define

$$\mu_B=\frac1m\sum_i x_i,\qquad \sigma_B^2=\frac1m\sum_i(x_i-\mu_B)^2,$$ $$\hat x_i=\frac{x_i-\mu_B}{\sqrt{\sigma_B^2+\epsilon}},\qquad y_i=\gamma\hat x_i+\beta.$$

Here $\epsilon>0$ stabilizes the denominator; $\gamma$ and $\beta$ are learned scale and offset parameters. The normalization above uses the population-form batch variance, with denominator $m$. Library conventions for running variance estimates can differ from the forward-pass convention.

An illustrative counterexample

Illustrative values: take a batch $(0,0,2,2)$. Its mean and population variance are both $1$. Ignoring $\epsilon$ just for this arithmetic demonstration, its normalized values are $(-1,-1,1,1)$. There are still only two distinct values. Nothing has made their distribution Gaussian.

With nonzero $\epsilon$, the normalized variance is $\sigma_B^2/(\sigma_B^2+\epsilon)$, not exactly one. After the learned affine transform, neither zero mean nor unit variance is guaranteed.

Training and inference are different

During training, the statistics depend on other examples in the batch. At inference, a conventional batch-normalization layer uses stored running estimates so a prediction does not depend on the other examples submitted alongside it. Forgetting evaluation mode can therefore change model behavior.

A running average of standard deviations is also not interchangeable with the square root of a running average of variances. When replacing handwritten code with a library layer, check the precise equations and defaults.

Batch normalization can help optimization, but this note does not claim it always helps or explain its benefits solely through one proposed mechanism. The original method is described by Ioffe and Szegedy in Batch Normalization. The exercise comes from Karpathy's makemore series.