Deeper

Lab · c23.s01.L1

Bandit explorer

Exploration buys information and costs reward; the share of random pulls decides which of the two the agent buys.

Question: Why must an agent that learns from reward sometimes choose the lever that currently looks worse?

Bandit explorerA strip of 150 pulls: lever A's ticks sit above the axis, B's below; a filled tick marks a paid pull. Two step lines track the running averages; B climbs toward 0.6 when explored, A stays near 0.2, and at ε = 0.00 B is never pulled. Average reward peaks at 83.2 for ε = 0.10 and falls to 74.1 at 0.50: exploring costs reward but is the only way to learn B is better.Bandit explorerWhy pull the lever that looks worse?150 pulls · 400 runs per εhidden: A pays 0.20 · B pays 0.60 — the agent cannot see thisone tick per pull · filled = paidAB1150pullrunning average payoutA solid · B dashed0.70.60.40.20.0average paid400 runs per εpeak 0.10 · 83.2150750ε 0.000.50always B 90.0After pull 150 of 150 · 70 paidA: 19 pulls · average 0.211B: 131 pulls · average 0.504ε 0.10 · run 1 · 400-run mean 83.2 · 6.8 short of 90.0Final: A 19, B 131 · 70 paid
Synthetic Synthetic: a simulated two-armed bandit with hidden payout probabilities 0.20 and 0.60, chosen as teaching values and not measured from any machine or game. The strip is one seeded run of 150 pulls; the curve averages 400 independent runs per ε. Every tick, running average and curve point is written by data/scripts/c23.s01.L1.py with seed 20261108. The frames show the whole run and pulls 1, 10, 25, 50, 75, 100, 125 and 150 for each of the five runs at every ε stop; the peak at 0.10 belongs to this pair of levers and this 150-pull budget, not to bandits in general.
Lab controls
0.1

Left/Right move one stop (0.05); Page Up/Page Down move three stops; Home sets 0.00; End sets 0.50

1

Left/Right move to the previous or next run; Home selects 1; End selects 5

800 ms per frame

Left/Right move 100 ms; Page Up/Page Down move 400 ms; Home sets 300; End sets 1400

Frame 1 of 9: Whole run: pull 150 of 150The run stops at pull 150: lever A was pulled 19 times for a running average of 0.211, lever B 131 times for 0.504, and the run paid 70. Across 400 runs at ε 0.10 the mean is 83.2, 6.8 below the 90.0 that always pulling B earns. B ended with the larger share of pulls.

Keyboard: Space plays or pauses, Left and Right step, Home goes to the first frame, End goes to the last frame.

Text description of this lab

A strip of 150 pulls: lever A's ticks sit above the axis, B's below; a filled tick marks a paid pull. Two step lines track the running averages; B climbs toward 0.6 when explored, A stays near 0.2, and at ε = 0.00 B is never pulled. Average reward peaks at 83.2 for ε = 0.10 and falls to 74.1 at 0.50: exploring costs reward but is the only way to learn B is better.

Return to the section that hosts this lab

Caption status: synthetic. Every lab declares one of published, re-simulated, schematic, synthetic or interpolated.