LLM Attention: How Words Look at Words

Visualize self-attention as beams of light between tokens and feel how softmax(QKᵀ/√d_k) decides which word "it" is looking at.

Prologue · 1/5

A constellation of words

Seven words float as arcs of light. Each time the viewpoint moves from word to word, where that word is “looking” in the sentence appears as a bundle of light. First watch the flow and confirm that the shape of the light changes from word to word.

Watch the sweep and find the difference between words where the light concentrates and words where it scatters.

softmax(QKᵀ/ √d_k+ mask)·V

Attention weightsit → cat 96%

The1%

cat96%

sat1%

because1%

it3%

was—

tired—