The Residual Stream: A River Through the Layers
See the residual stream as a glowing river that every layer reads from and writes to, with attention and MLP tributaries adding features.
Przetłumaczone maszynowo z oryginału w języku angielskim. Otwórz oryginał w języku angielskim
A single glowing river
Each layer of a Transformer does not compute on its own — it reads from and writes to one shared flow, the residual stream. Here we watch it as a glowing river running from input to output. Every time the river passes through a layer, a little more information is added and it grows brighter.
Watch the river flow from input to output, brightening as it passes each layer.
attn: gather the context / mlp: finishing before output. The final layer tidies the whole and finishes the representation handed to next-token prediction.