Deeper

Lab · c16.s01.L1

Biased-coin entropy

A biased coin's entropy is the probability-weighted average of the surprises of its two outcomes, and that average is largest, at one bit, when the coin is fair.

Question: How much information is in one symbol?

Biased-coin entropyA slider sets the probability that a coin lands heads. Two vertical bars show the surprise of each outcome: heads, common, short; tails, rare, tall. A shaded part of each bar is its probability-weighted share, and those two shares stack to a point on the entropy curve. At a fair coin the point peaks at one bit; toward either edge the rare outcome grows tall but its share shrinks, and the point falls to nearly zero.Biased-coin entropy: the average surpriseh(p) = log2(1/p) H(p) = p h(p) + (1-p) h(1-p) p = P(heads)Surprise of each outcome, bits1 bitheadstailsAverage surprise H(p)p = P(heads)12345600.000.000.250.250.500.500.750.751.001.00peak 1.000 bit at p = 0.506.6440.014Hp = 0.01h(heads) = 6.644 bitsh(tails) = 0.014 bitsheads share 0.066 bits+tails share 0.014 bits≈H =0.081 bits
Re-simulated Re-simulated: the surprise and entropy values are computed exactly from h(p) = log2(1/p) and H(p) = p h(p) + (1-p) h(1-p) by data/scripts/c16.s01.L1.py. No data are sampled; p = 0.90 reproduces the section's worked example. The frames walk p from 0.01 to 0.99 and end on the loaded coin at p = 0.90, so every bar, share and marker is one stored grid point.
Lab controls
0.9

Arrow keys step 0.01, Page Up/Down step 0.10, Home = 0.01, End = 0.99

Loaded coin

Arrow keys cycle the presets, Home and End jump to the first and last

On

Space toggles when focused, or press W

Frame 1 of 9: One time in a hundred: headsAt p = 0.01 heads is almost never the outcome, but when it comes it is worth 6.644 bits: the tallest bar on the chart. Its share is only 0.066 bits because it happens so seldom, and tails adds 0.014 bits, so the stack reaches 0.081 bits, the smallest average on the grid. The gold line marks one bit of surprise: heads towers above it, yet the average sits below it.

Keyboard: Space plays or pauses, Left and Right step, Home goes to the first frame, End goes to the last frame.

Text description of this lab

A slider sets the probability that a coin lands heads. Two vertical bars show the surprise of each outcome: heads, common, short; tails, rare, tall. A shaded part of each bar is its probability-weighted share, and those two shares stack to a point on the entropy curve. At a fair coin the point peaks at one bit; toward either edge the rare outcome grows tall but its share shrinks, and the point falls to nearly zero.

Return to the section that hosts this lab

Caption status: re-simulated. Every lab declares one of published, re-simulated, schematic, synthetic or interpolated.