A Tiny Language Model

Use word-pair counts to write new sentences, one predicted word at a time.

🎯 What you will learn

Generate text by repeating a prediction
Ties need a rule to break them
Generated text can sound right but be wrong

👋 Let’s get started!

A large language model writes by predicting one word, adding it to the text, then predicting the next. Our tiny model does the same thing with word-pair counts. It is simple, but you can see the same idea. ✍️

🪜 Step by step

1. 🔗 Count pairs

Build the table of which word follows which.

table = {}
for i in range(len(words) - 1):
    table.setdefault(words[i], []).append(words[i + 1])

2. ✍️ Generate

Start with one word. Add the most likely next word, and repeat.

word = "the"
for step in range(3):
    word = best_next(word)
    print(word)

3. ⚖️ Break ties

If two words are equally likely, sort them and pick the first. This keeps results the same every time.

max(sorted(options))

🧪 Try it!

Change the code and press ▶ Run to see what happens. You can’t break anything!

Write a sentence

words = "the cat sat on the mat the cat ran on the mat".split()
table = {}
for i in range(len(words) - 1):
    table.setdefault(words[i], []).append(words[i + 1])

def best_next(word):
    options = table[word]
    return max(sorted(set(options)), key=options.count)

word = "the"
sentence = word
for step in range(5):
    word = best_next(word)
    sentence = sentence + " " + word
print(sentence)

🎨 Unplugged activity

✍️ Use this tiny model with your own short text (at least 20 words). Generate 5 new words. Are they true? Could a person tell the difference?

Repair the generator

def best_next(word):
    options = table[word]
    return max(sorted(set(options)), key=options.count)
print(best_next("the"))

📖 Key words

Tap a card to flip it over!

Generate: Create new text 📝
Tie: Two choices equally good 🤝
Hallucination: Confident text that is not true 👻

🏆 Challenge

Use the sentence “a b a c” and print the word that most often follows “a”. Press ▶ Run.

Challenge

words = "a b a c".split()
table = {}
for i in range(len(words) - 1):
    table.setdefault(words[i], []).append(words[i + 1])

Knowledge Check Quiz

How does the model write a sentence?
Why sort the options?
What is a hallucination?