Deeper

Lab · c23.s06.L1

Policy-gradient variance

Subtracting a baseline leaves the average policy-gradient update unchanged, because the score averages to zero, and it shrinks the update's spread most when the baseline equals the policy's own average reward.

Question: How do you improve a policy without knowing the world?

Policy-gradient varianceTwo hundred dots per strip share an axis from −0.6 to 0.6. Grey hollow dots show one-episode gradient estimates with no baseline; navy filled dots show the same episodes at the chosen baseline. Both centre on the same mean update, which no baseline changes, and the navy strip narrows as the baseline nears the policy's average reward: 0.50 at p₁ = 0.80, 0.55 at p₁ = 0.90. A U-shaped variance curve below bottoms there, marked with a gold diamond.Policy-gradient varianceA baseline moves the spread of the update, not its average−0.600.6one-episode gradient estimate0.1500.000.501.00circle: current baseline · diamond: best baselineno baseline · sd 0.320mean update 0.150p₁ = 0.80 · baseline b = 0.00 · sd 0.320exact variance of the updatebest 0.50
Synthetic Synthetic: the two-armed bandit is invented, not observed. The policy is held at 50/50, action 1 pays 1 with probability p₁ and action 2 with probability 0.20; the variance and standard-deviation curves are exact for that model, and the dot strips are a seeded simulation of 200 episodes (seed 20261243). The controls set the baseline b and p₁. At p₁ = 0.80 the standard deviation falls from 0.320 at b = 0.00 to 0.200 at b = 0.50, 37.5%, while the mean update stays 0.150.
Lab controls
0

Left and Right move one position (0.05); Shift with Left or Right moves five positions (0.25); Page Up and Page Down move 0.25; Home sets 0.00; End sets 1.00

0.80

Left and Right select the previous or next value; Page Up and Page Down jump two values; Home selects 0.60; End selects 0.90

Frame 1 of 1: No baseline (b = 0.00)Without a baseline the 200 one-episode updates spread with standard deviation 0.320 and variance 0.102 around the mean update 0.150. They are wide because a win pays the full reward. Subtracting the policy's own average reward, 0.50, leaves this mean where it is and shrinks the spread.

Keyboard: Space plays or pauses, Left and Right step, Home goes to the first frame, End goes to the last frame.

Text description of this lab

Two hundred dots per strip share an axis from −0.6 to 0.6. Grey hollow dots show one-episode gradient estimates with no baseline; navy filled dots show the same episodes at the chosen baseline. Both centre on the same mean update, which no baseline changes, and the navy strip narrows as the baseline nears the policy's average reward: 0.50 at p₁ = 0.80, 0.55 at p₁ = 0.90. A U-shaped variance curve below bottoms there, marked with a gold diamond.

Return to the section that hosts this lab

Caption status: synthetic. Every lab declares one of published, re-simulated, schematic, synthetic or interpolated.