Next-token Prediction: How Words Are Born
Watch logits become a probability distribution through softmax, reshape with temperature, and collapse to a single sampled word.
Scores for the next word
For the continuation of "Once upon a ___", the model produces a score (a logit) for every word in its vocabulary. Here we line up only the leading candidates as pillars of light. Each pillar's height is its logit. You can see "time" standing out above the rest.
Look at the pillar heights and find which word's logit stands out.
Distribution
time93%
while3%
midnight2%
dream1%
star1%
hill1%
day0%
night0%
Sharpness (entropy) 19%. 0% is a single point, 100% is a uniform distribution.