GPT from scratch

I completed a small GPT-style language model from scratch, following Andrej Karpathy’s Neural Networks: Zero to Hero. It predicts the next character, building from a bigram baseline to a decoder transformer. My learning notebook explains the components, code and derivations along the way.

What I built

The implementation includes character tokenization, token and position embeddings, masked multi-head self-attention, feed-forward layers, residual connections and layer normalization. A PyTorch training loop uses AdamW and next-character cross-entropy; a sampling loop generates text one character at a time.

The accompanying script brings these components together. The notebooks preserve the route to that implementation: autograd, makemore, manual backpropagation, hierarchical context and finally a transformer.

Data and learning objective

The teaching implementation uses the Tiny Shakespeare text distributed with Karpathy’s material, with an initial training portion and a held-out validation portion. For characters $x_1,\ldots,x_T$, the autoregressive model represents:

$$p_\theta(x_1,\ldots,x_T)=\prod_{t=1}^{T}p_\theta(x_t\mid x_{<t}).$$

The training objective is the negative log probability assigned to the next character. Causal masking prevents a position from reading future characters during prediction. Tokenization, context length and the data split affect how a loss should be interpreted; a character-level loss is not directly comparable to a subword-token loss.

What to read

Open the decoder-transformer notebook for the progression from bigrams to attention. The complete collection includes handwritten derivations and recall questions, plus a download containing the original notebooks and implementation scripts.

Attribution and scope

This is a completed learning implementation following Karpathy’s Zero to Hero, with my explanations and exercises. It explores the decoder-transformer ideas behind GPT-2; it does not reproduce the original GPT-2 architecture, tokenizer, training corpus or published performance.

The collection retains intermediate attempts and saved outputs. Completion refers to the learning project, not a claim that every notebook cell executes in order without revision. No independently reproduced benchmark or novel language-model research result is claimed here.

Learning notes · Projects

Project workflow

Figures and evaluation

Evaluation exports are not available on this site.

The completed learning implementation is available in the notebook collection. No benchmark result is reported here.

Project links

• Decoder-transformer learning notebook