Independence and base rates

A forecast gives a 10% chance of rain in each hour from noon to midnight. If the twelve hours were independent, meaning that the belief in rain in one hour told you nothing about the next, the chance of a dry day would be $0.9^{12}$, which is about 0.28. The forecast does not mean that, because rain persists: a wet five o'clock makes a wet six o'clock more likely, not less [S:5:723].

When does one event tell you nothing about another? When learning the first leaves your belief in the second unchanged. For an event $B$ of positive probability, that is the statement $P(A\mid B)=P(A)$. In general, $A$ and $B$ are independent when $P(A\text{ and }B)=P(A)P(B)$ [S:5:722]. !In the left table the events cannot occur together, so the overlap is 0 and learning one rules out the other. In the right table the beliefs are 0.5 and 0.5, the overlap is 0.25, and learning one leaves the other unchanged. Disjoint is not independent. Status: schematic.

An event of probability zero is defined to be independent of every event, including itself. Independence is an assumption about the world rather than a fact you can read off a single observation [S:5:723]. !The test keeps a sensitivity of 0.90 and a false-positive rate of 0.05; only the prevalence moves along the horizontal axis. A prevalence of 0.01 gives a posterior of 0.154, and 0.20 gives 0.818. As the disease becomes common the curve approaches the sensitivity, because false positives stop dominating. Status: synthetic.

Independence has a product form. Two events are independent exactly when the belief in both together equals the product of their separate beliefs [S:5:723]. The form is convenient because it turns a question about information into a multiplication, and it is dangerous because the multiplication is easy to perform on events that do not satisfy it.

$$P(\text{heads on both flips})=P(\text{heads on the first})\cdot P(\text{heads on the second})$$

That line says: for two independent flips, multiply the belief in one by the belief in the other. [F01]

Worked example. Flip two fair coins on opposite sides of a room. The belief in heads on one coin, written P, is 0.5, and the two results do not influence each other [S:5:722]. The belief in two heads is 0.5 × 0.5 = 0.25, and the same product gives the belief in two tails. The four ordered results each carry 0.25, and they add to one.

Disjoint events are the opposite case, and confusing the two is a standard error. If two events cannot happen together and both have positive probability, learning that one occurred sets the belief in the other to zero, so they are dependent, not independent [S:5:723]. Independence requires that one event leaves the other's belief alone, and disjoint events fail that test because learning one rules the other out.

Independence can be checked against data, at least in principle. Compare the product of the two separate rates with the rate observed when both events occur. If the product is 0.2 and the observed joint rate is 0.3, the assumption is false for this population.

Base rates, the prevalence of a condition before any test, enter through the same door. A test is informative exactly when its sensitivity differs from its false-positive rate. If the two were equal, a positive result would leave the belief in disease unchanged, and the test would tell you nothing. In the clinic example 0.90 differs from 0.05, and that difference is what makes the test useful. The prevalence still controls how much a positive result tells you, because positive results from healthy people grow with the number of healthy people [S:5:715].

The table holds the test fixed and moves only the base rate, which is the prevalence of the condition before the test. In clinic A, 0.90 × 0.01 = 0.009 and 0.05 × 0.99 = 0.0495, so the posterior is 0.009/0.0585, which is 0.154. In clinic B, 0.90 × 0.20 = 0.18 and 0.05 × 0.80 = 0.04, so the posterior is 0.18/0.22, which is 0.818.

The same positive result therefore means something different in the two clinics, although the test and its two error rates are identical. Nothing about the test changed between the rows; only the company the test keeps changed. A report that quotes the posterior without the prevalence has quoted a number that belongs to one clinic only.

A smoke alarm is a useful comparison. An alarm that fired independently of smoke would tell you nothing, and an alarm that fires only with smoke tells you a great deal. Independence is symmetric, but the pair of conditional beliefs is not. An alarm is built so that the belief in an alarm given a fire is high, while the belief in a fire given an alarm can be small when fires are rare. The causal direction runs from fire to alarm.

Independence assumptions also fail in ways that are hard to see. The DNA calculation in the O. J. Simpson murder trial [S:5:725] multiplied five marker frequencies, the rates at which each genetic marker appears in a population, to obtain a probability of one in a hundred and seventy million [S:5:726]. The multiplication assumed that the markers appear independently.

The defence challenged that assumption, and the frequencies themselves came from a few hundred blood samples, which leaves the product resting on uncertain inputs as well as on an assumption [S:5:726].

Two habits follow from the section. State an independence assumption out loud when you multiply, and state the base rate when you quote a posterior. Each habit costs one sentence and guards against one of the two errors described above.

The idea has two costs. The assumption is easy to state and hard to justify, and it fails loudly in weather, in DNA evidence and in any process where one event makes another more likely. The product form also invites arithmetic on events that were never independent, and the result looks as precise as a correct one. A model that assumes independence where none holds does not become less confident; it becomes confidently wrong.

The formal statement. Two events are independent when

$$P(\text{both})=P(\text{one})P(\text{other}).$$

That line says that the belief in the joint event is the product of the two beliefs [S:5:723]. For a list of events, mutual independence requires the product condition on every sublist of size two or more, not on pairs alone [S:5:724]. Conditional independence given a third event requires the same product inside the conditional measure [S:62:42].

The derivation. Begin with the conditional definition and suppose the condition has positive probability. Independence is equivalent to the statement that conditioning leaves the event's belief unchanged. Multiply by the denominator and the product form follows. For three events, mutual independence needs four equations: three for the pairs and one for the triple. Pairwise independence supplies the first three and says nothing about the fourth.

The edge case. Flip three fair coins. Let the events say that coin one matches coin two, that coin two matches coin three, and that coin three matches coin one. The source computes one half for each event and one quarter for each pair, so every pair passes the product test [S:5:727]. All three together have probability one quarter rather than one eighth, so the triple fails [S:5:727]. The core sentence "learning one leaves the other unchanged" passes every pairwise check and still misleads here. Pairwise independence is weaker than the property you need in order to multiply three numbers.

A second edge case. An event of probability zero is defined to be independent of every event, including itself [S:5:722]. The "learning changes nothing" story cannot explain that convention, because learning a null event lies outside the story. The definition is chosen at the boundary so that the product rule stays true.

What the core hid. The core said independence means that one event tells you nothing about another. That is the two-event case with positive probability. It hides the strength of the multi-event condition and the convention at zero. It was safe because the core multiplied two coins and never three.

Where the analogy breaks. A smoke alarm is useful because it depends on fire. The analogy treats dependence as a property of a pair. Independence for a list is a property of every sublist, and a model can pass every pairwise check while failing the joint one. The alarm picture has no room for that distinction.

Original sources. The definition, and the warning that independence is a modelling assumption, are in Mathematics for Computer Science [S:5:723]. The same text gives the three-coin counterexample and the DNA multiplication error [S:5:726]. Conditional independence appears in Murphy [S:62:42].

An open question. How much data makes a mutual-independence assumption credible. The text is blunt: assuming that the DNA markers are mutually independent may well not be reasonable unless you have examined hundreds of millions of blood samples [S:5:728]. The field has no general rule for deciding which sublists to test. Evidence that would settle a particular case: direct estimates of each joint frequency, with enough samples to bound the error, or a causal model that predicts the dependence before you measure it.

A frontier result. Exchangeability does much of the work that independence used to do in prediction. A sequence is exchangeable when its joint distribution is unchanged by permuting the indices, and de Finetti's theorem shows that an infinitely exchangeable sequence is a mixture of independent sequences given a hidden parameter [S:62:106]. Conformal prediction uses that weaker assumption to attach a coverage guarantee to each prediction set, without claiming that the observations are independent [S:48:4].

The cost of the weaker assumption is visible in the DNA case. If only pairwise independence holds, the source bounds the match probability by the product of the two rarest markers, which is far larger than the five-way product [S:5:727]. Testing mutual independence therefore requires one test per sublist, and the number of sublists grows exponentially with the number of events [S:5:724]. The practical rule is to state the assumption and to test the cheapest implication you can measure.

What to read next. Read Mathematics for Computer Science Sections 17.7 and 17.8 for the definition and the counterexample [S:5:722], Murphy Section 3.2.5 for exchangeability [S:62:106], and the conformal prediction introduction for the predictive use [S:48:4].