Predicting the Next Word

Language models guess what comes next. Count word pairs to see how.

🎯 What you will learn

Language models predict the next word
Counting pairs of words gives probabilities
Predictions depend on the text they learned from

👋 Let’s get started!

Large language models do one main job: they predict the next word, over and over. Here is a small version. We count which word follows each word in a text, then use those counts to guess. 🔤

🪜 Step by step

1. ✂️ Split into words

split() breaks a sentence into a list of words.

words = "the cat sat on the mat".split()
print(words)

2. 🔗 Count pairs

For each word, count which words come after it.

nxt = {}
for i in range(len(words) - 1):
    if words[i] == "the":
        nxt[words[i + 1]] = nxt.get(words[i + 1], 0) + 1

3. 🎲 Predict

The most common follower is the best guess.

print(max(nxt, key=nxt.get))

🧪 Try it!

Change the code and press ▶ Run to see what happens. You can’t break anything!

Who follows “the”?

words = "the cat sat on the mat the cat ran".split()
nxt = {}
for i in range(len(words) - 1):
    if words[i] == "the":
        nxt[words[i + 1]] = nxt.get(words[i + 1], 0) + 1
print(nxt)
print("Best guess after the:", max(nxt, key=nxt.get))

🎨 Unplugged activity

🔤 Take a sentence from your favourite book. Split it into words and count how often each word is followed by the same word. What word follows “the” the most?

Count the pairs

words = ["a", "b", "a", "c"]
pairs = {}
for i in range(len(words) - 1):
    key = words[i]
    pairs[key] = pairs.get(key, 0) + 1
print(pairs)

📖 Key words

Tap a card to flip it over!

Language model: Predicts the next word 🔤
Probability: How likely something is 🎲
split(): Breaks text into words ✂️

🏆 Challenge

Use the sentence “one two one three” and print the word that most often follows “one”. Press ▶ Run.

Challenge

words = "one two one three".split()
nxt = {}

Knowledge Check Quiz

What does a language model predict?
What does split() give back?
Why might a model predict a wrong word?