The Residual Stream: A River Through the Layers
See the residual stream as a glowing river that every layer reads from and writes to, with attention and MLP tributaries adding features.
Machinaal vertaald vanuit het Engelse origineel. Open Engels origineel
A single glowing river
Each layer of a Transformer does not compute on its own — it reads from and writes to one shared flow, the residual stream. Here we watch it as a glowing river running from input to output. Every time the river passes through a layer, a little more information is added and it grows brighter.
Watch the river flow from input to output, brightening as it passes each layer.
Stream contents
Subject1.2
Entity1.0
Tense0.9
Sentiment1.2
attn: gather the context / mlp: finishing before output. The final layer tidies the whole and finishes the representation handed to next-token prediction.