Interpretability

A YouTube comment reads "Christmas Song visit my channel! ;)". A spam classifier, a model that labels messages as unwanted or not, gives it a probability of 0.99. Remove the word "channel!" and the probability drops to 0.17 [S:110:266]. The other six words do not carry the decision. That drop is an explanation: the model's decision rests on one piece of the input.

Interpretability is the degree to which a person can understand the cause of a decision [S:110:7]. An explanation is the statement that does that. This section's question has two answers, because there are two kinds of explanation. A local explanation accounts for one prediction. A mechanistic explanation accounts for the computation that produced it [S:446:4]. !Attribution by removal, and the check that makes it more than a story. The full review scores 0.8; removing great drops it to 0.3, an attribution of 0.5, and removing film drops it to 0.7, an attribution of 0.1. Removing both leaves 0.2, and $0.5+0.1=0.6=0.8-0.2$, so the parts account for the whole drop in this example. The scores are synthetic, from the section's worked example. Status: synthetic.

Local methods treat the model as a box you may query. Feature attribution asks how much each part of the input contributed to the output. The simplest version removes a part and measures the change. LIME does this at scale: it makes perturbed copies of the input, records the model's answers, and fits a small interpretable model to those answers near the original point [S:110:262]. SHAP does the same job with a rule from cooperative game theory that divides the prediction among the features [S:110:208]. !Two kinds of explanation. A local explanation stays outside the box: perturb the input, record the output, and fit a small model near the original point. A mechanistic explanation goes inside: read the activations, isolate a direction or a sparse feature, and test it by intervention. The first accounts for one prediction; the second accounts for the computation. The figure is schematic. Status: schematic.

The small model only has to match the black box near the original input, not everywhere; that is called local fidelity [S:110:262]. SHAP adds a guarantee that the attributions sum to the gap between the prediction and the average prediction, which LIME does not provide [S:110:208]. The guarantee makes SHAP easier to audit, but it does not make the explanation true of the model's internals. An explanation is a claim about which parts of the input matter, and it can be wrong even when the prediction is right.

Worked example. A review gets a spam probability of 0.8. Removing "great" drops it to 0.3, so the attribution for "great" is 0.5. Removing "film" drops it to 0.7, so its attribution is 0.1. Removing both drops it to 0.2.

The two attributions add to 0.6, which equals the fall from 0.8 to 0.2. The check passes, and that is what makes the table more than a story. A negative attribution would mean the part pushed the prediction down. The size of the drop is the size of the effect. This is an intervention, and it is the only kind of check that separates a faithful explanation from a fluent one.

Mechanistic work goes inside. A sparse autoencoder learns a wide layer that reconstructs a model's internal activations, with the constraint that few units are active at once, so that each active unit can be read as a candidate feature [S:439:1]. Erasing a single direction from thirteen chat models removed their refusal behaviour, and adding it back produced refusal on harmless requests [S:438:1]. A direction is not a full explanation, but it is a causal handle.

A feature found this way is a hypothesis, not a fact. It earns belief when intervening on it changes the output in the predicted direction, as the refusal direction did [S:438:1]. Without that test, a readable label can be a story attached to a number.

The idea breaks on stability. LIME's explanations can change when the sampling is repeated, and they can be manipulated to hide a bias [S:110:269]. Mechanistic methods face the same problem in another form: current decomposition methods, which break an activation into parts, lack solid theoretical foundations, so a clean-looking feature may not be the unit the model uses [S:446:3]. An explanation you cannot falsify is decoration.

The formal statement. Fix one input and a set of features $F$. Let $v(S)$ be the model's output when only the features in $S$ are present. The Shapley value of feature $i$ is its average marginal contribution over every order in which the features can arrive: $${\varphi }_i=\sum _{S\subseteq F\setminus \{i\}}\frac{|S|!(|F|-|S|-1)!}{|F|!}\left(v(S\cup \{i\})-v(S)\right).$$ The weight in front counts the orderings that reach the subset $S$ before feature $i$ arrives.

Four axioms pin the value down: efficiency, symmetry, dummy and additivity [S:110:208]. No other assignment satisfies all four.

The derivation. Take two features. There are two orderings.

In the order one then two, feature one contributes $v(\{1\})-v(\varnothing )$ and feature two contributes $v(\{1,2\})-v(\{1\})$. In the reverse order the two contributions swap. The Shapley value averages the orderings, so each feature receives half of each marginal. Adding the two values cancels the middle terms and leaves $v(\{1,2\})-v(\varnothing )$. That is efficiency: the parts sum to the whole gap between the prediction and the empty input.

The general formula runs the same argument over all orderings, and the step I omit is the counting argument that produces the factorial weight.

An edge case. L1 checks that two occlusion attributions add to the fall from 0.8 to 0.2, and the check passes. That is a property of the example, not of occlusion. When two features interact, removing one changes what the other is worth, and the separate drops need not add to the joint drop. The Shapley value makes the parts add to the whole by averaging over orderings, and it pays for that with evaluations on feature subsets that never occur in the data [S:110:209].

With correlated features, those artificial inputs can pull the value away from every real input.

What L1 hid, and why that was safe there. L1 defined an attribution by removal, which is easy to compute and easy to check against a second removal. The axiom set is what separates a method with a guarantee from a method with a habit. For a first pass the removal picture is the right one, because the worked example was additive and the guarantee coincides with the arithmetic there.

Where the analogy breaks. "A smaller model of a larger model" suggests that the explanation is a model you can query. A Shapley value is not a model. It returns one number per feature and cannot say what happens if a feature changes by a stated amount [S:110:209]. Faithfulness is a claim about interventions, and a table of numbers is not a simulator.

Original sources. LIME is Ribeiro, Singh and Guestrin's local surrogate. It perturbs the input, records the model's answers, and fits a small interpretable model near the original point [S:110:262]. The example in L1 comes from Molnar's treatment, where removing "channel!" moves the spam probability from 0.99 to 0.17 [S:110:266]. The Shapley axioms come from cooperative game theory, and SHAP applies them to a model's output [S:110:208].

An open question. The field has not settled how to show that a recovered feature is the unit the model uses. Sparse-autoencoder reconstructions lose performance, so the dictionary may not preserve the computation it claims to decompose [S:446:10]. A probe can also succeed on activations with no causal link to the output [S:446:20]. A benchmark with a known ground-truth mechanism supplies the other half of the test, because it scores a method's output against an answer rather than against plausibility [S:446:19].

The evidence that would settle one case is an intervention that moves the output in the direction the feature predicts, replicated across seeds and models, and compared against the effect of a matched random direction.

A frontier method. The refusal direction is the cleanest current example. Erasing one direction from the residual stream removed refusal across thirteen chat models, and adding the direction back produced refusal on harmless requests [S:438:1]. The method depends on this section because it treats a direction as a causal handle and tests it by intervention. That is the faithfulness test which a table of local weights does not provide.

What to read next. Read the LIME chapter for the method and its stated limits [S:110:262], then the Shapley chapter for the axioms and the correlated-feature problem [S:110:209]. Read the open-problems review for reconstruction error and for probing [S:446:10]. Read the refusal paper for what a causal test looks like in practice [S:438:1].