How does a value summarize the future?

In a 5-by-5 grid the centre cell has value 0.7, and its four neighbours are 2.3 above, 0.4 to the right, −0.4 below, and 0.7 to the left [S:83:82]. These are values, not rewards: each is the expected return from that cell under a fixed policy. The numbers come from a random policy that picks each of the four moves with equal probability, and from a discount rate of 0.9 [S:83:82].

The section asks how one number can summarize a whole future, and the answer is that the number is the expected return from that cell. The value function is the table of those numbers, one per situation. Averaging over a policy means weighting each action by the chance the policy takes it; averaging over the world means weighting each next state by its transition probability. !The centre cell is worth the average of its four neighbours, 0.75, times the discount 0.9, which gives 0.675, rounded to 0.7. The reward at the centre is zero, so the whole value comes from where the fixed policy can go next. The five numbers are the section's worked example and the figure is schematic. Status: schematic.

A value is not a prediction of the next reward; it is the expected return from now on, following a fixed policy [S:83:80]. In an episodic task that is the discounted sum of rewards up to termination; in a continuing task it is the discounted infinite sum, which is finite when the discount is below one and the rewards are bounded [S:83:77]. That is why a cell that pays nothing now can still be worth a lot, and why the next reward is not the value.

Two versions of the table are useful: the state-value function gives one number per state. The action-value function gives one per state and action, so it can compare the choices inside a state [S:83:80]. The first answers "how good is this situation?" and the second answers "how good is this move here?"

The table is not arbitrary, because each value equals the reward now plus the discounted value of where you land, averaged over the actions and the transitions [S:83:81]. That relation is the Bellman equation, and we write the discount rate as γ and the value of a state as V.

That line says a value is built from its neighbours, one step ahead, with the same discount γ.

The policy is random here, so the same cell can send the agent to different neighbours. In general the world can be random too, and the value averages over both the policy's action choice and the transition probabilities [S:83:81], which is why the number is an expected return and not a forecast of one trip.

Worked example. Take the centre cell and its four neighbours, whose average is 0.75. Moving from the centre pays 0 in this grid, so the value is 0.9 × 0.75 = 0.675, which rounds to 0.7, the number printed in the table; the check uses four neighbours, one average, and one multiplication.

The grid also shows a surprise. One cell, A, pays 10 the moment you enter it, yet its value is below 10, because the move after the payment lands in a state from which the agent is likely to hit the edge [S:83:82]. A value is not the next reward; it is the whole trip.

The Bellman relation turns one global question into many local ones, so instead of simulating every future a solver adjusts each cell until it agrees with its neighbours and then repeats. That is the bridge to the next section, where the adjustment becomes an algorithm.

A policy is optimal when no other policy does better [S:83:84], and a greedy policy reads a value table and takes the action with the best one-step lookahead [S:83:101]. The best value table satisfies the same Bellman relation with a maximum over actions instead of an average [S:83:85].

Once the table exists, acting is a lookup: for each action, find the next state, or average over the possible next states, under the rules of the world, add its discounted value to the reward, and take the largest. The hard part is filling the table, not reading it.

Think of a player's rating: one number summarizes many games and lets you compare two players without replaying every match. The comparison breaks because a rating hides how variable the player is, since two players with the same average can fail in very different ways.

The cost is size, because a table with one entry per state is too large to fill when the states are images or robot positions, since there are far more states than any table can hold [S:83:91]. An image is a state when its pixels carry what the next step depends on, and then the table needs one row per image. The value is also only as good as the policy and the description of the world behind it, so a wrong description gives a confident wrong number.

The formal statement. The state-value function of a policy gives the expected return from each state under that policy [S:83:80]. The action-value function conditions on the first action as well [S:83:80]. The Bellman expectation equation writes each state's value as the policy-weighted average of the reward plus the discounted value of the next state [S:83:81]. The optimal value satisfies the same relation with a maximum over actions in place of the average [S:83:85].

The derivation. Split the return into the first reward and the rest. Take the expectation. The Markov property lets you replace the expected tail by the value of the next state, because the tail depends on the past only through that state. Write the result for every state at once and you have a linear system: a vector of values equals a vector of rewards plus the discount times the transition matrix times the vector of values. The solution inverts the matrix that is the identity minus the discounted transition matrix [S:82:40]. That inverse exists when the discount is below one [S:82:40]. The omitted step is the invertibility argument, which uses Gershgorin's circle theorem: every eigenvalue lies in a disc that misses the origin [S:82:40].

The edge case. L1 computed the centre cell's value as the discount times the average of its four neighbours. That works because the reward there is zero and the four moves are deterministic and equally likely. Change one hypothesis and the arithmetic changes. If the cell paid a reward, the value would be that reward plus the same discounted average. If the move were random with unequal probabilities, the neighbours would be weighted by those probabilities, not by one quarter. L1's average is a special case of an expectation, and the special case hides the weights.

What L1 hid. L1 said a value is only as good as the policy and the model behind it. The sharper statement is that the equation can fail to have a solution. The fixed point is unique when the discount is below one, or when eventual termination is guaranteed from all states under the policy being evaluated [S:83:96]. The condition is policy-relative. In episodic pole-balancing, the policy that balances forever earns 1 at every step and has an infinite return, so no finite fixed point exists for it, even though other policies terminate and the task is formally episodic [S:83:78]. In a continuing task with a discount of one, the identity minus the transition matrix is singular, because every row of the transition matrix sums to one. There is no inverse, and the value may be infinite. Uniqueness in the episodic case comes from the boundary condition at the terminal state, not from the discount.

Where the analogy breaks. A player rating summarises past games. A value summarises future ones, and it is computed from the values of neighbouring states rather than from outcomes. The rating analogy suggests that the number is estimated from data. In the dynamic-programming case it is computed from the model, and no game is ever played.

Original sources. The Bellman equation for a value function appears in §3.5 [S:83:81] and §3.6 [S:83:86] of Sutton and Barto. The optimality equation has a unique solution for a finite MDP [S:83:86]. Zhao derives the contraction property in the maximum norm, gives the proof of the fixed-point theorem, and shows that the iterates converge exponentially fast from any starting table [S:82:57-59]. The same chapter gives the closed-form solution as a matrix inverse [S:82:40].

An open question. The equation is well posed when the discount is below one or eventual termination is guaranteed from every state under the policy being evaluated [S:83:96]. Outside those cases it can have no finite solution, and the standard repairs, discounting, average reward and relative values, give different optimal policies. Discounting is one repair, and it changes the task rather than solving the original one. No general test says which repair matches a given continuing task. Evidence that would settle it: a task with a known optimal behaviour, two repairs, and a proof that one repair's policy is optimal on the task while the other's is not.

A frontier result. AlphaGo Zero trains a value network to predict the winner of games it plays against itself [S:286:2]. That network supplies a value for a board position. In each simulation the search traverses interior nodes using stored statistics and evaluates the leaf it reaches with the network; the value is backed up through the visited nodes, and the policy output becomes the prior probabilities when the leaf is expanded [S:286:26]. The network is not queried at every node. The search uses $f_{{\theta }_i}$ to evaluate leaf positions and prior probabilities [S:286:24]. The Bellman relation is still the object being approximated, now with a deep network in place of a table.

What to read next. Read §3.5 and §3.6 of Sutton and Barto for the two Bellman equations [S:83:81], the greedy policy [S:83:86], and the exercise that checks the equation numerically on the centre cell [S:83:82]. Read Chapter 3 of Zhao for the matrix form [S:82:40] and the contraction proof [S:82:57-59]. Then read the AlphaGo Zero methods for a value function at scale [S:286:2].