Maximum likelihood

You toss a coin ten times and watch the sequence H T H H T H H T H H, where H is heads and T is tails: seven heads and three tails. Is the coin fair? The data are compatible with a fair coin, but they sit closer to a coin that lands heads 70 percent of the time. You want the value of the head probability that makes this particular result least surprising.

That value comes from the likelihood. For data you can count, the likelihood is the probability of the observed data at each candidate value of the parameter, the unknown number the model is built around. For continuous readings it is a density instead, which is not a probability and need not stay below 1 [S:40:75]. The rule is to choose the parameter that maximizes it, and the resulting number is the maximum likelihood estimate [S:96:500]. Only the way the likelihood depends on the parameter matters, so any positive factor that does not contain the parameter can be dropped [S:96:500]. !Seven heads in ten tosses. The curve in (a) is L(p) = p to the seventh times (1 - p) cubed, the chance of this sequence read as a function of the head probability. Its height is 0.000977 at p = 0.5, 0.00222 at p = 0.7 and 0.000478 at p = 0.9, so the top sits at the observed fraction 7/10. The curve is not a statement about which p is true; it is the chance of the data under each p. Panel (b) is the logarithm of the same curve on the same axis: the heights change and the peak stays at 0.7, which is why the sum of logs can stand in for the product of chances. Only the middle of the log axis is shown; the curve falls below it toward both ends. Status: synthetic.

The likelihood is not the probability that the parameter is right. It is the probability of the data under each candidate, and the two are easy to confuse. The distinction matters when you compare models rather than parameters: the likelihood ranks two candidate coins by how well each predicts this data, not by how likely each is to be the true one. A likelihood of 0.0022 is not a 0.22 percent chance that the head probability is 0.7; it is the chance of this sequence if that probability is 0.7. Change the model, for example by allowing tosses to depend on each other, and the same data produce a different likelihood and a different estimate.

For ten independent tosses the likelihood of this sequence is the chance of each head multiplied by the chance of each tail. Every sequence with seven heads and three tails has the same chance, so the order does not change the product; it only decides which of several equally likely sequences you saw. Writing p for the head probability and $L(p)$ for the likelihood of this sequence, the product is:

The formula says: raise the head probability to the number of heads, raise the tail probability to the number of tails, and multiply [S:96:500]. If you record only the count, seven heads, the likelihood picks up a factor that counts how many sequences carry seven heads, the binomial coefficient. That factor does not contain p, so it plays no part in choosing p [S:96:500].

Evaluate the formula at three candidates.

The fair coin gives about 0.00098. The 0.7 coin gives about 0.0022, more than twice as much. The 0.9 coin gives about 0.00048, because it makes three tails unlikely. The largest value is at 0.7, and the maximum likelihood estimate is therefore 0.7.

Multiplying many small probabilities underflows a computer, which means the product becomes too small for the machine to hold [S:96:500]. Taking a logarithm turns the product into a sum and does not move the maximum, because the logarithm rises with its argument. The logarithm of a number is the power to which a fixed base must be raised to give that number, so it converts multiplication into addition. Maximizing the log-likelihood is therefore the same as maximizing the likelihood, and the derivative, the slope of the curve, is easier to handle. For the coin the slope vanishes at the observed fraction of heads, where the curve is flat at its peak.

This is not a coincidence of the coin. For n independent trials with x heads, the maximum likelihood estimate is x/n, the observed fraction [S:96:501]. Ten tosses with seven heads give 0.7. The estimate is the relative frequency you would have guessed, now derived from a principle rather than assumed.

The same principle covers continuous readings, where the data are measurements on a scale rather than counts and the likelihood is a density. For Gaussian data the likelihood is a product of bell curves, one per observation. Maximizing it gives the sample average for the centre, and a variance estimate that divides by n rather than n−1 [S:63:190]. The maximum likelihood rule thus reproduces the estimators from the previous section and explains why one of them is biased.

The table also shows how to compare two candidates without drawing a full curve. The ratio of their likelihoods says how much better one explains the data. Here 0.0022 divided by 0.00098 is about 2.3, so the data favour 0.7 over 0.5 by a little more than two to one. A ratio near one means the data cannot separate the candidates, which is the honest way to say that the experiment was not decisive.

The logarithm also explains why the estimate is often a simple average. Each observation contributes one term to the log-likelihood, and the terms add. For the coin the total is $x\log p+(n-x)\log (1-p)$, and its slope is $x/p-(n-x)/(1-p)$. Setting the slope to zero gives $x(1-p)=(n-x)p$, hence $x=np$ and $\hat{p}=x/n$. The same pattern returns for every model in which observations are independent.

Every step assumed the tosses are independent. If the coin has memory, or if the observations arrive in clusters, the product of probabilities is the wrong likelihood and the estimate inherits the error. The principle is only as good as the model that supplies the probabilities. This is the first place where a statistical guarantee depends on an assumption you cannot check from the data alone.

The method has a sharp edge. Toss a coin once, see heads, and the estimate is 1: the rule says the coin is certain to land heads again [S:96:501]. Nobody believes that, which shows that the estimate carries no measure of its own uncertainty. A single number cannot tell you whether it came from ten tosses or ten thousand. The next section supplies the missing measure, and it also explains why one observation is close to worthless.

The formal statement. Let $\{p_{\theta }:\theta \in \Theta \}$ be a parametric family and let $x=(x_1,\dots ,x_n)$ be the observed data. The likelihood is $L(\theta )=p_{\theta }(x)$, read as a function of $\theta$ with $x$ held fixed. The maximum likelihood estimate is any $\hat{\theta }$ in $\arg {\max }_{\theta \in \Theta }L(\theta )$ [S:96:500]. The definition needs only that the maximum exists. The derivation below adds three assumptions: the observations are independent given $\theta$, the parameter lies in the interior of $\Theta$, and the log-likelihood is differentiable there.

The derivation. For $n$ independent Bernoulli trials with $x$ heads, the likelihood is $p^x(1-p{)}^{n-x}$ times a factor that does not contain $p$. The log-likelihood is $\ell (p)=x\log p+(n-x)\log (1-p)$ [S:96:501]. Differentiate and set the derivative to zero: $x/p-(n-x)/(1-p)=0$. Multiply by $p(1-p)$ to get $x(1-p)=(n-x)p$, hence $x=np$ and $\hat{p}=x/n$. The second derivative is negative on the interior, so the stationary point is the maximum. No step is omitted.

The edge case. The maximum need not exist. For a Gaussian mixture, the likelihood is unbounded: a component can place its mean on one observation and send its variance to zero [S:64:237]. The likelihood then rises without limit, and there is no maximizer. L1's coin cannot show this because its likelihood is bounded on the closed interval from 0 to 1. A reader who treats "the maximum likelihood estimate" as always defined will meet this case in any mixture model.

What L1 hid. L1 said the estimate is the observed fraction, which is correct. It hid that the derivation assumes an interior maximum. When $x=0$ or $x=n$, the maximizer sits on the boundary at 0 or 1, and the derivative argument does not apply. L1's remark that nobody believes the estimate after one head is this boundary case in disguise. The remark is also the reason the next section needs a measure of uncertainty.

Where the analogy breaks. "Least surprising" suggests that the parameter is a hypothesis and that the likelihood ranks hypotheses by how probable they are. The likelihood is not a probability over $\theta$. It is one probability over the data for each candidate $\theta$. The ratio in L1 compares two candidate models, not two degrees of belief, and it says nothing about either model being true.

Original sources. The maximum likelihood principle was studied by Ronald Fisher at the beginning of the twentieth century [S:79:355]. Eisenstein states the principle, writes the Bernoulli likelihood, and derives the estimate by setting the derivative of the log-likelihood to zero [S:96:500] [S:96:501]. The corpus states the attribution but does not contain Fisher's paper, so L3 should not quote it.

An open question. The consistency of maximum likelihood needs the true distribution to lie inside the model family [S:66:120]. When it does not, maximum likelihood still targets the member of the family closest in KL divergence to the empirical distribution, which need not be the truth. The finite-sample behaviour in that regime is not settled in general. Evidence that would settle a given case: a finite-sample bound under a named form of misspecification, checked by simulation at the sample sizes the bound claims.

A frontier result. Most supervised training is maximum likelihood. Minimizing the negative log-likelihood is minimizing the cross-entropy between the empirical distribution and the model, and for a Gaussian model it is least squares [S:66:118] [S:66:119]. That identity connects the coin estimate in L1 to the objective used to train a language model. The same principle also explains why a model trained to fit the training data can still be far from the data-generating process.

What to read next. Read Eisenstein A.6 for the derivation and the maximum a posteriori extension [S:96:500] [S:96:501]. Read Goodfellow 5.5 for the KL and cross-entropy view, and 5.5.2 for the conditions under which the estimator is consistent [S:66:118] [S:66:120]. Read Shalev-Shwartz and Ben-David 24.7 for the historical note and the pointer to the Bayesian alternative [S:79:355].