A bigram model is a table of conditional probabilities

A bigram language model predicts the next character using only the current one. We can build that model by counting pairs, before introducing a neural network. The important question is what each row of the count table means.

This note develops the counting and loss sections of my first makemore notebook. The example below is illustrative, not a reported training result.

Normalize the right thing

Let $N_{ij}$ count how often character $j$ follows character $i$. Let $V$ be the number of possible next tokens, including any boundary token. With additive smoothing $\alpha>0$,

$$P(j\mid i)=\frac{N_{ij}+\alpha}{\sum_{k=1}^{V}N_{ik}+\alpha V}.$$

Each row sums to one. Dividing by the sum of the entire table would instead describe a joint frequency over pairs; it would not give the conditional distribution needed for next-character sampling.

For an illustrative row with counts $(2,1,0)$ and $\alpha=1$, the smoothed probabilities are $(3/6,2/6,1/6)$. An unseen transition has a nonzero probability. Smoothing prevents infinite loss on that transition, but does not create evidence that it is common.

Turn probabilities into a loss

For $T$ observed transitions, mean negative log-likelihood is

$$\mathcal L=-\frac{1}{T}\sum_{t=1}^{T}\log P(c_{t+1}\mid c_t).$$

Natural logarithms measure this loss in nats per transition. Assigning higher probability to the observed next character lowers its contribution to the loss. Multiplying many probabilities can underflow; summing log probabilities avoids that particular numerical problem.

A uniform predictor over $V$ tokens has loss $\log V$. Be careful with a shortcut from scratch notes: one transition receiving more than $1/V$ probability does not establish that the model has learned useful structure. We need the average loss on held-out data, compared under the same tokenization and boundary conventions.

What this model cannot remember

Two prefixes ending in the same character receive the same next-character distribution. A bigram model cannot distinguish them using earlier context. Adding a longer context window motivates a multilayer perceptron language model.

Split complete names into training, validation and test sets before creating adjacent-character examples. Otherwise, fragments of the same name can appear on both sides of the split. Keep the test set for the final evaluation; do not report the illustrative row above as an experiment.

The learning source is Karpathy's makemore repository, accompanying Zero to Hero.