Conditional probability

Two hundred emails sit in a folder. Forty of them are spam, and the word free appears in thirty of those forty. The same word appears in sixteen of the other hundred and sixty messages. A filter that has now seen the word free faces forty-six messages rather than two hundred [S:5:705].

How does new information change the count? It replaces the sample space with the part of the sample space that the information permits. The numerator counts the messages that are both spam and contain free, while the denominator counts every message that contains free. Nothing else in the folder contributes to the answer, and the other hundred and fifty-four messages drop out of the calculation entirely. !One unit of area is one message: the free column is 46 wide and the no-free column 154, and each column is split by spam. In the free column 30 of 46 messages are spam, so the gold block fills 0.652 of the column; across the whole table those same 30 messages are 30 of the 40 in the spam row, or 0.75. Both margins are drawn to the same scale, and the two denominators differ. Status: schematic.

Write $A$ and $B$ for two events. A conditional probability is the belief in $A$ after learning that $B$ occurred [S:5:706]. The vertical bar is read as given, so P(spam | free) is the belief in spam given that the message contains the word free. The definition divides the belief in the overlap by the belief in the condition, and the condition must have nonzero probability or the ratio has no meaning [S:5:706].

$$P(\text{spam}\mid \text{free})=\frac{P(\text{spam and free})}{P(\text{free})}$$

That line says: restrict attention to the messages containing free, then ask what share of them are spam. [F01]

Worked example. Put the folder into a table with two rows and two columns. Thirty messages are spam and contain free, and ten are spam without it. Sixteen are ordinary messages containing free, and a hundred and forty-four are ordinary messages without it.

Every margin, meaning the row and column totals, agrees with the story: forty spam, a hundred and sixty ordinary, forty-six containing free, two hundred in all. The belief in spam given free is 30/46, which is approximately 0.652. The belief in free given spam is 30/40, which equals 0.75. The belief in spam before any word is seen is 40/200, which equals 0.2.

Counts and beliefs are two forms of one statement. Dividing every entry of the table by two hundred converts counts into beliefs, and the ratio 30/46 survives the conversion because the common factor cancels. A conditional belief can therefore be read from a table of raw counts before any division is performed, which is why the table is the more useful object to draw first.

The same arithmetic appears in a tree diagram, where each branch records a conditional belief and the belief in a complete path is the product of its branches [S:5:711]. The table is that tree drawn flat, with the first fork choosing the row and the second fork choosing the column.

The two conditional beliefs answer different questions, and the difference is not a rounding error. The first asks how many free messages are spam, and the second asks how many spam messages are free. A filter that confused them would sort the folder by the wrong ratio. The same table supports both questions because conditioning selects a row or a column, and rows and columns carry different totals.

The definition also runs backwards, and the reversed form is worth memorising. The belief in the overlap equals the belief in the condition multiplied by the belief in the event given the condition, which is the product rule for two events [S:5:711]. In the folder, the condition's share is 46/200 = 0.23, and 0.23 times 0.652 is approximately 0.15, which equals the overlap 30/200. The two routes reach the same interior cell, which is what consistency requires.

This identity gives you a second way to check a conditional belief. Multiply the condition's share by the conditional share and compare the product with the overlap counted directly. A mismatch means that one of the three numbers is wrong, and the size of the mismatch often points to the faulty margin.

Conditioning resembles a spotlight that narrows to one part of a stage. The share of the lit area taken by one actor can be large while the share of that actor's area that is lit stays small. The resemblance stops at the edge of the beam, since a spotlight does not change which comparisons are worth making and information does.

Information can also be used badly. The best-known case is the Monty Hall problem, named for the host of the game show Let's Make a Deal. A contestant picks one of three doors, a prize sits behind one, goats behind the other two, and the host, who knows where the prize is, opens a losing door and offers a switch. A famous analysis conditioned on "the contestant picks door A and a goat is behind door B", and that condition includes the outcome in which the host opens door C [S:5:707].

The contestant sees which door the host opens, so the correct condition is "the contestant picks A and the host opens B". The arithmetic was right for the wrong condition. The definition of conditional probability cannot detect that mistake, because it begins after the choice of condition has been made. The choice itself is a modelling decision, and no formula performs it for you.

The idea has two costs. The condition must be possible, since a belief cannot be divided by zero, and the condition must be the information you actually possess, since a narrower or wider condition yields a correct answer to the wrong question. The next section turns the asymmetry between the two conditional beliefs into a rule.

The formal statement. For two events, where the condition has positive probability, the conditional probability is defined by

$$P(\text{event}\mid \text{condition})=\frac{P(\text{event and condition})}{P(\text{condition})}.$$

That line says: keep only the outcomes inside the condition, then ask what share of them also lie in the event [S:5:706]. The product rule is the same equation rearranged:

$$P(\text{event and condition})=P(\text{event}\mid \text{condition})P(\text{condition}).$$

That line says: the belief in both equals the belief in the condition times the belief in the event given the condition [S:5:711].

The derivation. The definition is chosen, not proved. The choice is forced by one requirement: conditioning must preserve the relative beliefs of events inside the condition. For two events $A$ and $A^{\prime }$ inside $B$, the ratio $P(A\mid B)/P(A^{\prime }\mid B)$ must equal $P(A)/P(A^{\prime })$. The displayed ratio is the only renormalisation that preserves those relative weights. Non-negativity, $P(B\mid B)=1$ and additivity then follow. Those axioms alone do not select the ratio: the uniform measure on $B$ is also a probability measure. The product rule then carries no extra content, since it is the definition multiplied by the denominator. The chain rule for several events follows by applying that product rule repeatedly [S:5:728].

The edge case. A condition of probability zero. The ratio is undefined, and the core says the condition must be possible. That is right for the ratio and incomplete as a statement about the theory. Conditioning extends to a $\sigma$-algebra or a random variable through conditional expectation, and that object is unique only up to sets of measure zero [S:81:61]. A regular version, which returns a distribution for every value, need not exist [S:81:61]. It does exist when the variable takes values in a Borel space [S:81:62]. The honest statement is narrower: the ratio needs a positive denominator, and the general notion needs a regularity hypothesis.

What the core hid. The core described conditioning as narrowing the sample space. That picture is exact for finite spaces with positive-probability conditions. It hides that conditioning defines a new measure by renormalisation, and that renormalising is where the asymmetry between the two directions enters. It also hides that the product rule is the definition in disguise.

Where the analogy breaks. A spotlight narrows attention and leaves relative areas unchanged. Conditioning renormalises, so the total becomes one, and a small slice can dominate the answer. A spotlight also has a sharp edge, while the condition may be uncertain, and then you must average over the conditions you consider possible.

Original sources. The standard treatments state the definition as the ratio of the joint probability to the condition [S:62:41]. The tree-diagram version and the product rule appear in Mathematics for Computer Science [S:5:711]. The Monty Hall analysis isolates the failure that matters: the arithmetic was correct and the condition was wrong [S:5:707].

An open question. Conditioning on an event of probability zero has no elementary definition, and the general theory does not select a unique answer. Conditional expectation is unique only almost surely, and a regular conditional distribution may fail to exist [S:81:61]. Existence is settled for variables that take values in a Borel space [S:81:62]. What remains open is the modelling question: two regular versions can disagree on a null set, and no formula chooses between them. Evidence that would settle a dispute: a specified generating process for the condition, which selects one version and makes the disagreement testable.

A frontier result. Probabilistic graphical models describe a joint distribution over many variables and answer conditional queries about it, such as the probability of a label given the inputs [S:61:2]. That architecture makes conditional probability the object a learning system stores and queries, rather than a step in a hand calculation.

The chain rule explains the storage cost. A joint distribution over many events factors into one conditional per step, and the factorisation is exact for every ordering of the events [S:5:728]. The order changes the factors, not the product, which is why a model can choose the ordering that makes each conditional easy to estimate.

What to read next. Read Mathematics for Computer Science Sections 17.1 to 17.4 for the definition and the Monty Hall error [S:5:705]. Read Downey Chapter 1 for the product rule in use [S:40:21], and the notes in Lattimore and Szepesvari for the measure-theoretic caveat [S:81:61].