Lab · c23.s04.L1
Grid-world values
A state's value is one number that stands in for the whole discounted future from that state: the reward there plus the discounted value of the state the policy moves to, built one sweep at a time.
Question: How does one number per square summarize all the rewards still to come?
Frame 1 of 11: k = 0 · rewards onlySweep 0 at γ = 0.90: No look-ahead yet. Only the reward paid on a square has been counted, so the goal holds +1.00 and the pit holds -1.00; every other square reads 0.00. The two terminals keep these values at every discount, because their reward is here rather than later.
k = 0 · rewards only. Sweep 0 at γ = 0.90: No look-ahead yet. Only the reward paid on a square has been counted, so the goal holds +1.00 and the pit holds -1.00; every other square reads 0.00. The two terminals keep these values at every discount, because their reward is here rather than later.
Keyboard: Space plays or pauses, Left and Right step, Home goes to the first frame, End goes to the last frame.
Text description of this lab
A four-by-four grid shows a number in every cell, an arrow for each square’s fixed successor, a gold goal worth +1.00 and a russet pit worth -1.00. At first only the two terminal numbers are non-zero. Each sweep fills one more ring: a square holds its reward plus the discount times the number of the square its arrow points to, shrinking with distance. A chart traces the start square from 0.000 to its converged value; a dashed line marks it.
Return to the section that hosts this lab
Caption status: synthetic. Every lab declares one of published, re-simulated, schematic, synthetic or interpolated.