Lab · c04.s01.L1

Coin-flip luck and permutation test

One score means little until it is placed against what the same procedure scores on labels that carry no information.

Question: Does one run's score beat what the same procedure scores when the labels are shuffled, so that no real signal remains?

Synthetic Synthetic: 40 items, 20 positive and 20 negative, 2,000 label shuffles per setting, seed 20261108. The reported score is chosen by the reader, not measured. The frames show the null histogram, its mean and 95th percentile, the run marker, the shaded tail of shuffles that did at least as well, and the resulting p-value.

Lab controls

Score your one run reported65 per cent correct

Left/Right: plus or minus 5 points (2 items); Page Up/Down: plus or minus 10 points (4 items); Home: 50; End: 90

Settings tried before reporting the best5 settings

Left/Right: plus or minus 1 setting; Page Up/Down: plus or minus 5; Home: 1; End: 20

Frame 1 of 6: Shuffled labelsAll 2,000 label shuffles were scored the same way as the run, trying 5 settings. Each bar counts how many shuffles reached that many of the 40 correct answers.

Keyboard: Space plays or pauses, Left and Right step, Home goes to the first frame, End goes to the last frame.

Text description of this lab

A histogram of 2,000 runs on shuffled labels. Each bar counts how many shuffles reached that many correct answers out of 40. Bars cluster near 50 per cent with one setting and shift right as more settings are tried. A russet line marks the reported score; gold lines mark the null mean and 95th percentile. Shaded bars to its right are shuffles that did at least as well, and the readout gives the p-value.

Return to the section that hosts this lab

Caption status: synthetic. Every lab declares one of published, re-simulated, schematic, synthetic or interpolated.