From character IDs to a learned context

A character ID tells us which character we have, but its numerical size has no linguistic meaning. An embedding lookup replaces that arbitrary ID with a trainable vector. A network can then combine several such vectors to predict what comes next.

This is a focused explanation of the lookup-table and shape reasoning in my makemore MLP notebook. An MLP is a multilayer perceptron.

A lookup, not an ordering

Let the vocabulary size be $V$ and the embedding width be $D$. A matrix $C\in\mathbb R^{V\times D}$ stores one vector per token. Looking up token $i$ returns row $C_i$; multiplying the integer $i$ by a weight would impose an unwanted numerical ordering on the tokens.

For a context of $K$ tokens, concatenate the selected rows into $e\in\mathbb R^{KD}$. With hidden width $H$, a simple model is

$$h=\tanh(eW_1+b_1),\qquad z=hW_2+b_2,$$

where $W_1\in\mathbb R^{KD\times H}$ and $W_2\in\mathbb R^{H\times V}$. Softmax turns the $V$ logits in $z$ into a next-token distribution.

Follow an illustrative batch

For an illustrative batch size $B=4$, context $K=3$, embedding width $D=2$ and hidden width $H=5$, the shapes are:

• Token IDs: $(4,3)$.
• Looked-up embeddings: $(4,3,2)$.
• Concatenated context: $(4,6)$.
• Hidden activations: $(4,5)$.
• Output logits: $(4,V)$.

Concatenation preserves the order of the context positions. It is not averaging the three embeddings. The same lookup matrix is shared across positions, so repeated token IDs accumulate gradient contributions into the same row.

What learning an embedding does—and does not—mean

Backpropagation updates the selected rows along with the other parameters. Their geometry becomes useful for the prediction objective. Nearby vectors are not guaranteed to be synonyms, particularly in a character model. The original word-level motivation should not be transferred uncritically to characters.

The model still sees only $K$ preceding tokens. A low training loss alone cannot show that its representation generalizes. Choose hyperparameters using validation data and evaluate the final model on held-out examples.

See Bengio and colleagues' A Neural Probabilistic Language Model for the original word-level model, and Karpathy's makemore for the character-level learning exercise.