Notes
Updated
These are explanations for other learners, developed from my notebooks and study of neural networks and brain signals. Each note focuses on one idea, with the math and limitations alongside it.
For the complete study record, open my learning notebook: seven annotated notebooks with original comments, handwritten derivations, saved outputs, full-text search and recall questions. The shorter notes on this page develop selected ideas from that collection.
I've completed the autograd, makemore and GPT-from-scratch lessons represented in my notebook collection, following Andrej Karpathy's Neural Networks: Zero to Hero. The final lesson includes a character-level decoder-transformer implementation. The lesson list below tracks my learning separately from whether a reviewed note is available; it covers my current notebook set, not every video in the course.
Publication dates refer to these edited notes, not the dates I completed the lessons. Notebook outputs are not presented as research results.
Subscribe via RSS · Interactive explanations · Current work
Reviewed notes and essays
- Why a shared value needs gradient accumulation
Follow the two paths through a tiny computation graph to see why backward rules must add their contributions.
- A bigram model is a table of conditional probabilities
Count adjacent characters, normalize each row, and understand what the loss actually measures.
- From character IDs to a learned context
Track the shapes from an embedding lookup through a multilayer perceptron language model.
- Batch normalization does not make a distribution Gaussian
Separate centering and scaling from distribution shape, and distinguish training from inference.
- Why the softmax gradient is prediction minus target
Derive the logit gradient of mean cross-entropy and keep the batch scaling explicit.
- Growing context by combining neighboring representations
Follow a hierarchy of character groups and distinguish the learning exercise from the original WaveNet.
- What an EEG band can—and cannot—tell us
Frequency bands describe parts of a signal; they do not provide a direct label for a person's thoughts.
Learning progress
- Autograd with micrograd
Learning: completed · Reviewed note
- Makemore: bigrams
Learning: completed · Reviewed note
- Makemore: multilayer perceptron
Learning: completed · Reviewed note
- Makemore: activations and batch normalization
Learning: completed · Reviewed note
- Makemore: manual backpropagation
Learning: completed · Reviewed note
- Makemore: hierarchical context
Learning: completed · Reviewed note
- GPT from scratch
Learning: completed · Note not yet published