Every language model, including the one you can summon at the bottom of this page, is doing one thing: predicting the next token from the tokens before it. This page does exactly that with a lookup table. It reads a corpus, records which words followed each pair of words, and then rolls dice — no neural network, no training, no gradient, just counting. The output is fluent for about six words at a time and then wanders off, and the reason why is the most useful thing on this page: its entire memory is two words long.
Contexts with only one possible successor are memorised phrases — the chain has no choice there and simply replays the corpus. Branching is where it invents.
Both continue the same five words. One has a memory of two words; the other has read most of the internet. This is the distance the top of the iceberg travelled.
—
—
The corpus is split into tokens, and the chain walks through them building a dictionary whose keys are every adjacent pair of words and whose values are the list of words that were ever seen next. Generating text means holding a two-word context, looking up its successor list, picking one at random in proportion to how often it occurred, emitting it, and sliding the context forward by one word. Because the random draws come from a seeded generator, the same seed always produces the same passage — a small nod to reproducibility that most generative demos skip. This is a genuine probabilistic language model and it was state of the art for decades; what it cannot do is remember anything beyond its two-word window, so it has no subject, no argument, and no idea that a sentence it started thirty words ago is still open. Scaling that window up — and learning the probabilities in a network instead of counting them in a table — is, in one sentence, the entire journey from this page to the model in the panel beside it.