Ablation

Rainbow combined six improvements into one agent and reached a median score of 223 per cent of human performance in the no-ops regime and a hundred and fifty-three per cent in the human-starts regime, across 57 Atari games [S:288:5]. The authors then removed one component at a time and re-ran the entire system at a comparable computational cost. Removing prioritized replay cost a great deal, and removing multi-step learning did too. Removing the dueling network produced a change that the authors could not separate from noise, and they said so in the paper [S:288:6].

This section asks what happens when you remove one part. You learn what that part earns inside this system, with the other parts still present, and you learn nothing beyond that. The measurement is a removal, not an addition, and the direction decides which question the answer belongs to. !Four measurements of a system with three components. The full system scores 80; removing A costs 18, removing B costs 1, and removing C costs nothing. The three effects sum to 19, which is the full score minus the score of the system with all three removed. An additive model with an empty-system baseline of 61 and weights 18, 1 and 0 reproduces every row: 61 + 18 + 1 + 0 = 80, 61 + 1 + 0 = 62, 61 + 18 + 0 = 79, 61 + 18 + 1 = 80. The four rows therefore do not show interaction; they show that the empty-system score is not zero. Interaction needs the missing cell, the system without both A and B. Each removal is a complete retrain, not a subtraction of weights. Status: synthetic.

The idea. An ablation is the removal of one component from a complete system, followed by a full re-measurement with everything else held fixed [S:288:6]. It answers a narrow question that a hostile reader will ask: would this system be worse without that part? It does not answer whether the part would help a different system, because the other parts are not the same. !The six removals in Rainbow, each a complete retrain of about ten days on one graphics card at 200 million frames. The blocks share a time axis and the variants run within 20 per cent of one another in wall-clock time, which is what makes the removals comparable. The row costs about sixty days, and a variant trained at a different budget would not be an ablation of this system. Status: schematic.

Building up from a small base answers a different question, namely whether a part helps some system. Removal answers whether the part earns its place here, and the two answers can disagree without either being wrong. Rainbow reports large drops and honest nulls side by side, and the null about dueling is informative rather than embarrassing: the effect differed by game and cancelled in the median [S:288:6]. The same report shows why a median is a coarse instrument: removing multi-step learning hurt almost uniformly, and the full agent beat that ablation in 53 games out of 57 [S:288:6]. Removing dueling seemed to help some games and hurt others, which is a different kind of finding and deserves a different sentence.

Worked example. A system with three components, measured four times. The scores are constructed for this section.

The removal effects are 80 − 62 = 18 for A, 80 − 79 = 1 for B, and 80 − 80 = 0 for C. The full system scores 80, and the three effects sum to:

$$(80-62)+(80-79)+(80-80)=18+1+0=19$$

The three removal effects sum to 19, which equals the full score minus the score of the configuration with all three components removed. That gap does not show interaction. An additive model with an empty-system baseline and weights 18, 1 and 0 reproduces every row:

$$61+18+1+0=80,61+1+0=62,61+18+0=79,61+18+1=80$$

The table alone therefore shows only that the empty-system score is not zero; it does not attribute the residual to the parts acting together. Interaction requires the missing cell — the system without both A and B — or the empty system, measured rather than assumed. A report that presents three contributions as if they summed to the total has quietly assumed the interaction away. A removal is also a complete retrain rather than a subtraction of weights, so each row of the table costs what the full system costs. Rainbow reports that a run of 200 million frames takes about ten days on one graphics card, which makes six ablations six full runs [S:288:5].

Rainbow shows the same trap with real numbers. Double Q-learning looked useless in the aggregate, and the authors then hypothesized that value clipping was masking its effect, so that one component was hiding another [S:288:6]. The honest reading of that table is not "double Q-learning does nothing" but "this system cannot show what double Q-learning does while the clipping stays".

Equal cost. An ablation at a different budget is not an ablation, because the comparison then mixes the component with the extra training. Rainbow's variants run within 20 per cent of each other in wall-clock time, which is what makes its six removals comparable [S:288:5]. The same report gives the shape of the gain and not only its final size: the full agent matches the DQN baseline's final score after 7 million frames and passes the best baseline after 44 million [S:288:5].

Counting what you tried. Six ablations are six comparisons, and the most impressive one is also the one most likely to be luck. The informal principle in Mining of Massive Datasets says to calculate how many events of the kind you are looking for would appear in random data, and to distrust your findings when that number swamps the real ones [S:57:26]. Removing parts one at a time and reporting the winner is a search, and a search needs its own accounting.

Taking one instrument out of an orchestra and listening for what is missing is the usual picture of an ablation. An orchestra keeps its score fixed, so the players are additive and one instrument's removal is a genuine subtraction. A model retrains, so the remaining parts change what they play, and a removal is a new performance rather than a missing voice.

The limitation is scope. An ablation is evidence about one system at one budget, and it cannot tell you why a part helped or how two parts would behave in a different combination. It also cannot rescue a comparison that was never fair, which the remaining sections of this chapter try to prevent.

The formal statement. Write the score of a configuration as a function of the set of components it contains. Write $S$ for a score where higher is better, the convention of the L1 table. An ablation of one component reports

$$\text{effect}=S(\text{full})-S(\text{full without }c)$$

which is $80-62=18$ for A in the L1 table. Read it as: the fall in score when the component is removed and everything else retrains. On a risk $R$ where lower is better, the rise in risk is $R(\text{full without }c)-R(\text{full})$, the negative of that number. The two conventions must not be mixed. Two hypotheses belong to the definition. The retrain is complete rather than a subtraction of weights, and the budget is held fixed [S:288:5].

The derivation. Take two components, A and B. Ask what A adds when B is present, then ask what A adds when B is absent. The difference between the two answers is the interaction:

$$\text{interaction}=[S(\text{full})-S(\text{without }A)]-[S(\text{without }B)-S(\text{without }A\text{ and }B)]$$

Read it as: the effect of A with B present, minus the effect of A with B absent. Zero pairwise interaction in this sense is not sufficient for additivity when there are three or more components. Take $f(\varnothing )=0$, $f$ of any single component $=0$, $f$ of any pair $=5$, and $f(ABC)=10$: every pairwise term is zero, yet the three removal effects are 5 each and sum to 15 against a full score of 10. Additivity requires every interaction of order two and higher to vanish, which for $k$ components needs the full $2^k$ factorial table. The two-component case is the only non-trivial one in which the printed implication holds.

L1's four rows do not settle which case they are in. An additive model with an empty-system baseline reproduces them, so the residual is not evidence of interaction until the empty system or a two-way cell is measured.

The edge case. Rainbow's double Q-learning is the case that breaks the naive reading. Removing it changed the median little, and the authors traced the reason to value clipping: the clipped range hides the overestimation that double Q-learning corrects [S:288:6]. A near-zero effect therefore has two readings, "the component does nothing here" and "another component masks it". The ablation table cannot separate them. Only an experiment with the mask removed can.

What L1 hid, and why it was safe. L1 treated each row as one number. Each row is a random variable, and removing a component changes the seed-to-score function, so a single run per configuration confounds the effect with seed luck. Rainbow reports one training run per variant [S:288:5] and summarises each by its median over the 57 games; it states that the aggregate difference for dueling was not significant but reports no seed variation and no test, so the null is a statement about one run's game-level median rather than a seed-controlled estimate [S:288:6]. L1's equal-cost paragraph covered training time; the run count is the other half of the same requirement.

Where the analogy breaks. An orchestra holds its score fixed, so the players are additive by construction and one instrument's removal is a genuine subtraction. A model retrains, so the remaining parts change what they play. The removal is a new performance rather than a missing voice, and that is why a removal is not a subtraction. The difference is a property of learned systems, not a flaw in the method.

Original sources. The ablation as a published discipline is Rainbow's: six components combined, then removed one at a time, with every variant retrained at the same cost [S:288:6]. The report is unusual for printing its nulls, and the null about the dueling network is the one to read first. There was no significant difference in aggregate, with gains on some games and losses on others [S:288:6]. The informal accounting rule for a search over many comparisons is Bonferroni's principle, stated in Mining of Massive Datasets [S:57:26].

An open question. How to attribute credit when components interact is unsettled. Rainbow supplies the clean example. Double Q-learning looked useless until the authors noticed that value clipping was masking its effect, and they could offer only a hypothesis about the mechanism [S:288:6]. A removal effect in one context is not a contribution in general. Evidence that would settle it: the full factorial table, every subset measured at equal cost and with several seeds, from which each component's marginal contribution could be read in every context where it appears. The corpus contains no agent paper that reports the full factorial table for more than a handful of components; Rainbow reports the six single removals [S:288:6]. The cost is $2^k$ configurations at equal cost, so the claim is "none that we found", not "no published paper".

A frontier method. Where the factorial table is too expensive, the field reports a distribution over contexts instead. SAC's ablation compares a stochastic policy against a deterministic variant, five individual runs for each, initialised with different seeds [S:291:7]. That is the move of 4.1 applied to a component rather than to a method. The comparison still sits in one context, and the seeds do not widen it.

What to read next. Read [S:288:6] beside [S:291:7]. The first is an ablation that publishes its nulls, and the second is an ablation whose object is a component's mechanism rather than its score. Then read [S:57:26] for the arithmetic of how many comparisons you have actually run.