Information theory · interactive
From surprise to compression
Five live simulations that take you from the information in a single event to codes that squeeze a message down to its entropy. Every number is computed exactly, in bits, from distributions you control.
▌
The path
Each idea builds on the one before it. Work through them in order, then test yourself in the quiz.
Surprise
How many bits one event carries, and why rare events carry more: I(x) = −log₂ p(x).
Open simulation →interactiveEntropy
Average surprise: the fewest yes/no questions needed, on average, to pin down an outcome.
Open simulation →interactiveCross-entropy & KL
The price of a wrong model. This is the loss that trains classifiers and language models.
Open simulation →interactiveHuffman coding
An optimal prefix code built by merging the two rarest symbols, again and again.
Open simulation →interactiveArithmetic coding
One interval for the whole message, which reaches fractional bits per symbol.
Open simulation →interactiveQuiz
Six questions with worked explanations to check what you've learned.
Start quiz →Surprise: information in one event
Entropy: average surprise
Cross-entropy & KL divergence
True distribution p
Model q
| x | p(x) | q(x) | −log₂ p | −log₂ q | p·log₂(p/q) |
|---|---|---|---|---|---|
| A | 0.500 | 0.150 | 1.000 | 2.737 | 0.868 |
| B | 0.250 | 0.150 | 2.000 | 2.737 | 0.184 |
| C | 0.150 | 0.350 | 2.737 | 1.515 | -0.183 |
| D | 0.100 | 0.350 | 3.322 | 1.515 | -0.181 |
| p-weighted sum | 1.743 | 2.431 | 0.689 | ||
Huffman coding: bits ≈ surprise
| sym | p | code | length vs −log₂ p |
|---|---|---|---|
| a | 0.455 | 0 | 1 bits · ideal 1.14 |
| b | 0.182 | 111 | 3 bits · ideal 2.46 |
| r | 0.182 | 10 | 2 bits · ideal 2.46 |
| c | 0.091 | 1100 | 4 bits · ideal 3.46 |
| d | 0.091 | 1101 | 4 bits · ideal 3.46 |
Building the tree
- 1.c 0.091+d 0.091→cd 0.182
- 2.cd 0.182+b 0.182→cdb 0.364
- 3.r 0.182+cdb 0.364→rcdb 0.545
- 4.a 0.455+rcdb 0.545→arcdb 1.000
Arithmetic coding: fractional bits
Quiz
You roll a fair 8-sided die. How much information does learning the result give you?
A coin lands heads 90% of the time. Its entropy is about…
Your model q is exactly the true distribution p. What is the KL divergence D(p‖q)?
Is KL divergence symmetric, i.e. D(p‖q) = D(q‖p)?
For p = (½, ¼, ⅛, ⅛), the Huffman code's average length is…
A symbol has p = 0.9. Why can arithmetic coding beat Huffman here?