Rewards and returns

Two summer jobs promise 14 units of pay over five weeks: the first pays 1, 2, 6, 3, 2, and the second pays 3, 3, 3, 3, 2. If only the total matters the jobs tie, but if money next week is worth more than money next month the second job wins, because it pays earlier.

Comparing two futures needs one number per future, and that number is the return: the rewards ahead, added up with each later reward reduced by a discount rate [S:83:76-77]. The discount rate is written γ and lies between 0 and 1 inclusive [S:83:77]. At γ = 1 there is no discount, and a continuing return may diverge; below 1 the sum is finite when the rewards are bounded [S:83:77]. The endpoint is admitted whenever the sum stays finite, in particular when every episode terminates [S:83:79]. !Both jobs pay 14 over five weeks. At a discount rate of 0.5 the first is worth 4 and the second 5.75, because the second pays earlier. The outlined bars are the raw rewards, the filled bars are the same rewards after discounting, and the two totals are equal while the returns are not. The figure is synthetic: both sequences are the section's worked example, computed for this book. Status: synthetic.

!A reward k steps ahead counts gamma raised to the power k - 1 times as much as a reward now. At gamma = 0.5 the weight is nearly gone within ten steps; at 0.99 it still counts after twenty. A discount rate below 1 turns an endless stream of reward into a finite return: the sums are 2, 10 and 100. The figure is synthetic: the weights are computed from the plan's three rates. Status: synthetic.

That line says each step of delay multiplies a reward's value by γ [S:83:77], so the multiplication turns a list of future numbers into one number that can be compared. A reward is one step's payment, while the return is the whole future squeezed into one number.

A reward is not a hint about how to act: it says what outcome you want, and the agent finds its own route to it [S:83:76]. A chess agent rewarded for taking pieces may take pieces and still lose the game, so the reward must mark the outcome you actually want rather than a step you believe leads there.

Tasks come in two shapes: an episode is one run of interaction that ends, such as one game or one trip through a maze. A task made of episodes is an episodic task whose episodes each end in a terminal state [S:83:76]. A continuing task never ends, such as a controller that runs a factory all day [S:83:76]. Balancing a pole can be either one episode per attempt or one continuing task with a penalty at each failure [S:83:78].

The distinction matters for the sum. In a continuing task with a reward of 1 at every step an undiscounted return is infinite [S:83:76], so a discount rate below 1 makes the sum finite: with a constant reward of 1 the return is 1/(1−γ) [S:83:77]. That closed form is the geometric series: multiply the partial sum by γ, subtract, and all but the first and last terms cancel; then let the last term go to zero. At γ = 0.5 that is 2, and at γ = 0.9 it is 10, and discounting is also a mathematical convenience, because with γ below 1 the infinite sum has a finite value as long as the rewards are bounded [S:83:77]. Bounded means one fixed number is at least as large as every reward in absolute value.

The future is not one fixed sequence: each action can lead to many possible sequences, so the agent maximizes the expected return, the probability-weighted average over futures [S:83:76]. A single lucky episode is therefore not convincing evidence of a good policy, which is why the arithmetic below uses expected values even when the example shows one sequence.

Returns at neighbouring steps are linked: the return now equals the next reward plus the discounted return after that [S:83:77].

That line says the value of a future is the first payment plus a discounted version of the rest.

Worked example. Take γ = 0.5 and rewards 1, 2, 6, 3, 2 [S:83:78], whose discounted rewards from the start are 1, 1, 1.5, 0.375, and 0.125, adding to 4. The second job, 3, 3, 3, 3, 2, gives discounted rewards 3, 1.5, 0.75, 0.375, and 0.125, adding to 5.75.

Both jobs pay 14 in total, so discounted the earlier-paying job is worth 1.75 more. Working backwards confirms the first sum: the last reward counts as 2, the 3 before it becomes 3 + 0.5×2 = 4, the 6 becomes 6 + 0.5×4 = 8, the 2 becomes 2 + 0.5×8 = 6, and the 1 becomes 1 + 0.5×6 = 4.

Think of the discount rate as an interest rate run backwards: a promise of 100 next year is worth less than 100 today, and γ plays the part of one over one plus the interest rate. The analogy breaks because a market sets interest, while γ is a design choice you must defend.

The cost is that one number cannot carry everything, because a discounted return hides how variable the outcomes are and a high γ makes the agent care about rewards far ahead, so credit must travel further back. In a continuing task, γ must stay below 1 or the sum may not exist [S:83:76].

The formal statement. The return from the present step adds up the rewards ahead. Each later reward is multiplied by the discount once more than the one before it, and the discount lies between zero and one inclusive [S:83:77]. The endpoint $\gamma =1$ is admitted when every episode terminates, because the sum then has finitely many nonzero terms [S:83:79]. The sum is finite when the discount is below one and the reward sequence is bounded [S:83:77]. Episodic and continuing tasks share one notation, because episode termination can be treated as entry to an absorbing state that pays zero forever [S:83:79].

The derivation. For a constant reward of 1, the partial sum of $n$ terms is $\text{reward}\times (1-{\gamma }^n)/(1-\gamma )$. Gamma raised to $n$ tends to zero when the discount is below one, so the limit is the reward divided by one minus the discount [S:83:77]. The recurrence follows by factoring one reward out of the infinite sum: the return now is the next reward plus the discounted return after that [S:83:77]. Nothing is omitted except the algebra of the geometric series.

The edge case. L1 says a discount below 1 makes the sum finite when the rewards are bounded. Both conditions do work. Let the reward double at every step and let the discount be one half. Every term of the sum is then 1, and the sum diverges although the discount is below one. Run the other direction. A continuing task that pays 1 at every step has an infinite return when the discount equals one [S:83:76]. Neither condition alone is enough. You need bounded rewards and a discount below one, or a finite episode.

What L1 hid. L1 used the pair of jobs to show that two futures with the same total can rank differently. That hid the sign of the reward. Discounting multiplies a penalty as well as a payment, so an agent with a discount below one prefers to postpone a cost. The book asks whether the signs of the rewards matter or only the intervals between them [S:83:83]. In a continuing task, adding a constant to every reward adds a constant to every value and changes no preference [S:83:83]. In an episodic task it changes preferences, because the constant is collected once per step and episodes differ in length.

Where the analogy breaks. An interest rate is a price set by a market where you can borrow and lend. The discount rate is a design choice with no market behind it. You cannot trade a reward now for a reward later, and nothing forces the rate to match an observable quantity. So the interest analogy explains the arithmetic and not the choice of rate.

Original sources. Sutton and Barto define the discounted return in §3.3 [S:83:77], with the pole-balancing task as the motivating example [S:83:78]. The same task can be read as episodic or as continuing, and the book uses that pair to justify one notation for both [S:83:78]. Episodic and continuing tasks are then unified by adding an absorbing terminal state [S:83:79]. The undiscounted continuing formulation is deferred to a later chapter [S:83:80].

An open question. Discounting is a choice, not a measurement. The book introduces the discount as a parameter of the return, not as a quantity to be measured [S:83:77]. With a discount below one, the optimal policy can differ from the policy that maximises the average reward per step, and no general principle says which objective a given task wants. The field carries both formalisms and no test that selects between them from data. Evidence that would settle a given case: two tasks where the optimal policies differ, plus a measurement showing that the policy from one objective loses on the outcome the designer cares about.

A frontier result. Distributional reinforcement learning keeps the whole distribution of the return instead of its expectation [S:283:1]. The paper's argument concerns the approximate setting, not what a risk-neutral optimal agent should do: in the exact setting the expectation is sufficient for choosing an optimal action, and the paper does not deny it. What the distribution adds is information that changes learning: distributional Bellman backups preserve multimodality, mitigate the effect of a nonstationary policy, and expose an instability of the distributional optimality operator that the expected-value operator does not have [S:283:1]. Risk-sensitive behaviour appears in the paper as one of the earlier uses of the value distribution, not as the reason for studying it [S:283:1]. A distributional agent predicts a catalogue of outcomes rather than a single mean. The return stays the object of study, but one number per future is no longer enough.

What to read next. Read §3.3 of Sutton and Barto for the return [S:83:77] and §3.4 for the absorbing-state convention [S:83:79]. Read the distributional paper's introduction for the case against a scalar return [S:283:1]. Then read the chapter that the book points to for a formulation that is both continuing and undiscounted [S:83:80].