Multilayer perceptron

A multilayer perceptron, or MLP, transforms numbers through layers of weighted sums and nonlinear responses. Its structure specifies a family of functions; learning selects parameters within that family.

Intuition: build intermediate features

A hidden unit combines the inputs in one way; another combines them differently. The output can then use those intermediate responses. “Hidden” means these values are internal to the model, not that they are inaccessible: the figure and explorable show them explicitly.

The math

For input vector $x$, one hidden layer and a scalar output,

$$z=Wx+b,\qquad h=\phi(z),\qquad \hat y=v^Th+c.$$

In the illustrated 2 → 2 → 1 network, $x$ has two entries, $W$ is a 2-by-2 matrix, and $b$, $h$ and $v$ have two entries each. There are nine trainable parameters: four input weights, two hidden biases, two output weights and one output bias. The nonlinear function is applied separately to each hidden coordinate.

Why the nonlinearity matters

If $\phi(z)=z$, then

$$\hat y=v^T(Wx+b)+c=(v^TW)x+(v^Tb+c).$$

This is still just one affine function of $x$. Stacking more affine layers does not by itself make a nonlinear predictor. In our example, $\tanh$ supplies that nonlinearity. It also limits hidden responses to between −1 and 1 and has derivative $1-\tanh^2(z)$, which becomes small for saturated inputs.

The output transformation should match the task. A real-valued regression output need not be bounded. Classification typically uses a probability transformation and a compatible loss. Our illustrative output is deliberately linear; interpreting its value as a probability would be a mistake.

Representation is not evidence

An MLP can fit functions without its hidden units having clean biological interpretations. Input encoding, data coverage, loss and evaluation design determine what its predictions mean. A network diagram alone says little about reliability.

Where you will see this

Follow the numerical calculation in backpropagation step by step, then see how gradient descent changes the parameters. This is foundational learning for the planned GPT capstone.

References

• Stanford CS231n: neural networks.
• Dive into Deep Learning: multilayer perceptrons.