Expectation and variance

Three of the pumpkins in the garden weigh one pound, two weigh three pounds, and one Atlantic Giant weighs 591 pounds. The mean of those six weights is 100 pounds [S:41:46]. "The average pumpkin in my garden is 100 pounds" is then a true sentence about six pumpkins and a misleading sentence about pumpkins, because not one of the six comes close to weighing 100 pounds [S:41:46].

There is no typical pumpkin in that garden, and the repair is not a better average but a second number: the centre says where the weight sits, while the spread says how far individual pumpkins stray from it. The centre is the expectation, also called the long-term average, because it is what you would collect per trial if the trial ran many times over [S:135:242]. !Six pumpkins weigh 1, 1, 1, 3, 3 and 591 pounds, so their mean is 100 pounds. Five sit at the left edge and one sits alone at the right, and the mean line falls in the break where no pumpkin can be. The nearest pumpkin is 97 pounds from the mean. The centre and the spread are two different facts, and here the centre alone describes none of the six. Status: synthetic.

Read it in pieces: E(X) is the expectation of the rule X, x_i is one value it takes, P(x_i) is that value's probability, and the sigma says add over every value. Then the instruction: multiply each value by its probability, and add the products. For six equally likely pumpkins each probability is one sixth, so the sum returns the ordinary mean, and 600 pounds of pumpkin divided among six pumpkins gives 100 pounds apiece.

Every average you have computed so far is an expectation in disguise. Give each of the n rows in a column, where n is the number of rows, the same probability of one over n, and the weighted sum becomes the ordinary mean. That is why the mean of a column and the expectation of a random variable obey the same algebra, and why a single formula covers both cases.

The spread needs one more step, because deviations point both ways and would cancel if you added them raw. Square them first, weight each square by its probability, and add; that quantity is the variance, the mean squared deviation from the centre [S:41:46]. Its square root returns to the original units and is the standard deviation [S:41:46].

That formula says: read Var(X) as the variance of the rule X, then measure each value's distance from the centre, square that distance, count it with the value's probability, and sum the results.

Worked example. A soccer team plays on zero, one or two days of a week, with probabilities 0.2, 0.5 and 0.3 [S:135:243]. Multiply each count by its probability and add.

The third column adds to 1.1 days, and the fourth adds to 0.49 square days, so the standard deviation is 0.7 days. Check one row by hand: the value 2 sits 0.9 above the centre of 1.1, and 0.9 squared is 0.81, and 0.81 times 0.3 is 0.243.

The units carry information, because days multiplied by probability stays in days, so 1.1 is a number of days, while squared deviations are in square days and 0.49 needs a square root before you can compare it to a calendar. That is the whole reason the standard deviation is reported more often than the variance, even though the variance is the quantity that adds up cleanly when you combine independent pieces, whose errors do not move together.

A team that plays 1.1 days per week is not a thing you will ever observe, since it plays none, one or two days. The expectation is a statement about a long run of weeks rather than about any week you can point at [S:135:242]. The standard deviation is read the same way: 0.7 days describes the typical distance from the centre over many weeks, not a distance that any single week must show.

The pair costs something, and the pumpkin garden shows the price: two numbers cannot tell you that the distribution is lopsided, that one value is enormous, or that a gap separates small cases from large ones. Pregnancy length has mean 38.6 weeks and standard deviation 2.7 weeks, so deviations of two or three weeks are ordinary [S:41:47]. Its variance is 7.3, which is 2.7 squared, and that number is hard to interpret because its units are square weeks [S:41:47]. The mean is also pulled by rare large values, which is why the 591-pound pumpkin moved the centre from under two pounds to 100 and left the other five pumpkins exactly as they were.

The formal statement. For a discrete variable taking values v with probabilities that sum to one, the expectation of a variable f is the sum of each value times its probability. That sum is defined only when the sum of the absolute values is finite. Linearity is the first theorem: the expectation of a times f plus b times g is a times the expectation of f plus b times the expectation of g, for constants a and b, and no independence is required [S:5:771]. The variance is the expectation of the squared deviation from the mean, and the standard deviation is its square root [S:41:46].

The derivation. Expand the square with the mean written as c. The square of f minus c is f squared minus two c f plus c squared. Take expectations term by term and use linearity. The middle term becomes minus two c times the expectation of f, which is minus two c squared, and the last term is c squared. The two cancel against each other and leave the expectation of f squared minus c squared:

That identity is the form you compute with, because it needs one pass over the values for f and one for f squared, and no pass to collect the deviations first.

The edge case. The core divides by n, which is right when the n values are the whole population, the entire set you are describing. A sample drawn from a larger population gives every row probability one over n and still needs n minus one, because the sample mean was computed from the same rows and sits closer to them than the population mean does. When the n values are independent draws from one common distribution with finite variance, dividing by n underestimates that variance on average, and the standard repair divides by n minus one [S:41:128]. The estimator with n in the denominator is biased low, and the one with n minus one is unbiased under those two conditions together [S:41:128]. The gap matters most at small n, which is exactly where you are most likely to meet the formula.

The edge case that breaks the formula. The sum can fail to converge. A Cauchy variable has no mean and no variance, so the weighted sum returns nothing rather than a large number [S:36:97]. The expectation exists only when the sum of the absolute values is finite, and a symmetric distribution can have no centre of mass at all.

What the core hid. Three things. The formula is the discrete case; a continuous variable needs an integral, and the same linearity holds there [S:81:42]. The mean is not the only usable centre, and the choice depends on what an error costs [S:41:127]. And two numbers say nothing about shape, which is why the pumpkins could have a true mean of 100 pounds and no typical pumpkin.

Where the analogy breaks. "Centre of mass" suggests a balance point that always exists. The Cauchy distribution is symmetric and has no mean, so the balance point is not there to find [S:36:97].

Original sources. OpenStax defines the expected value as the long-term average and gives the probability-weighted sum [S:135:242]. Downey gives the variance as the mean squared deviation and the standard deviation as its square root [S:41:46]. The linearity theorem is Corollary 18.5.3 of Mathematics for Computer Science, with the remark that no independence is required [S:5:771]. The integral form, and the rule for transporting an expectation through a function, are in section 2.5 of Lattimore and Szepesvári [S:81:42].

An open question. Which centre to report is unsettled once the cost of error is asymmetric. The mean minimises squared error, and a different summary can win when overestimating and underestimating cost different amounts [S:41:127]. One number cannot serve both purposes, and the field has no default that fits every decision. Evidence that would settle a case: the actual cost curve of the decision, applied to held-out rows, with the mean, the median and a trimmed mean compared on the same data. Until the cost is written down, the choice of centre is a guess about the loss.

A frontier result. The bias–variance decomposition splits expected squared prediction error into irreducible noise, squared bias and variance [S:64:242]. It is 12.2 applied to an estimator rather than to a variable: the centre of the estimator's sampling distribution is its bias, and the spread around that centre is its variance. Every complexity curve in the later chapters is this decomposition drawn as a graph, which is why the two numbers in this section keep returning.

What to read next. Read section 2.8 of Downey for the two summaries and the pumpkin garden [S:41:46], then section 18.5 of Mathematics for Computer Science for linearity [S:5:771]. Read section 7.3 of The Elements of Statistical Learning for the decomposition [S:64:242]. The three readings answer what the centre is, why it adds, and what its error costs.