Growing context by combining neighboring representations

Instead of flattening an entire context at once, a network can first combine neighboring characters, then combine the resulting groups. Deeper representations can summarize progressively longer pieces of the context.

This note focuses on the tensor reshaping in my final makemore notebook. It is a learning exercise inspired by WaveNet, not a reproduction of its audio-generation system.

Pair, transform, repeat

Suppose an illustrative input contains $T=8$ context tokens, each represented by a $D$-dimensional vector. The batched tensor has shape $(B,8,D)$.

Grouping adjacent pairs changes the shape to $(B,4,2D)$ without changing the values. A learned transformation applied at every group maps the final dimension to a hidden width $H$:

$$h_j=\tanh([e_{2j};e_{2j+1}]W+b),\qquad W\in\mathbb R^{2D\times H}.$$

Repeating the pairing reduces four positions to two, then one. After three grouping stages, that final representation can depend on all eight original positions. Each grouping requires an even number of positions; a different context length needs an explicit padding or grouping policy.

A reshape is not learning

The reshape only arranges which values a subsequent transformation receives together. The weights perform the learned combination. Flattening the wrong axes might preserve the total element count while mixing positions or examples incorrectly, so a shape that runs is not necessarily a shape that means the right thing.

A useful check labels each input position and follows which labels can influence each output position. This establishes the receptive field without interpreting learned weights.

What carries over from WaveNet

The original WaveNet paper uses causal, dilated convolutions for raw audio. The makemore hierarchy illustrates expanding context with shared transformations; its pooling-like reduction, task and architecture differ.

For next-character prediction, the supplied context must contain only preceding characters. Excluding future targets is essential regardless of whether the architecture is an MLP, a hierarchy or a Transformer.

This completes the focused notes for my completed makemore lessons. I have since completed the GPT-from-scratch learning implementation; its full notebook is available in the learning collection. Course material: Neural Networks: Zero to Hero.