Section 23.3 says a state is the part of the history that changes the future, and that two histories sharing a state must offer the same prospects. Section 23.6 says a policy can be improved from sampled returns without a model of the world. In a real task the agent often has neither a hand-designed state nor a model: it sees pixels, or a few noisy sensors, and must decide what to remember. The field has no agreed answer to how an agent should learn a state summary that is small enough to learn from, rich enough to support good decisions, and stable enough that the value estimates built on it mean something.
Design a comparison between a hand-designed state and a learned state representation on a task where the observation hides a fact that changes the future. State a measurable prediction, say what result would change your mind, and name one thing your measurement cannot establish.
Rubric. Tick each criterion your answer meets, then save the self-assessment. The platform does not grade it.
Worked solution
There is no single correct answer, so this is a model of the shape a complete answer takes, not a key. A level-3 answer begins by fixing a task. One example is a corridor with a door that can be open or closed, where the observation shows the agent's cell but not the door. The hand-designed state is the cell plus the door bit; the learned state is a small vector trained from the observation history with the same data and the same budget. Both states feed the same value learner and the same exploration rule.
The answer then names two measurements. The first is a sufficiency test: collect pairs of histories that the learned state maps to the same point, and compare their one-step reward and next-state distributions. If the distributions differ, the state has merged situations that the future separates. The second is a control test: evaluate the greedy policy from each state on held-out episodes and compare the average return. A state can pass the first test on the training distribution and fail the second when the policy visits rare parts of the corridor.
The prediction must be falsifiable. A reasonable one is that the learned state will approach the hand-designed state on the training distribution but lose more on held-out door configurations, because the door bit is rare in the data. The answer says what would change its mind: if the learned state matches or beats the hand-designed state on the held-out configurations at the same budget, the prediction fails, and the hidden fact was recoverable from the observations the agent already had.
The limit is the hard part. A sufficiency test checks the pairs someone chose to collect; it cannot prove that no pair was missed. A control test measures the policy the state supports under one learner, so a bad result may come from the learner rather than the state. The answer names one of these limits and explains why it matters. It does not claim that a lower value error settles the question, because the state with the lowest error on the training distribution may still be the one that hides the door when the door matters most.