The multivariate Gaussian

Measure two things about every person in a room, height in centimetres and weight in kilograms, and plot one point per person. The result is a cloud, and the cloud is not a round blob. It leans, because tall people tend to be heavy, and that lean is as much a part of the description as the width of the blob.

A multivariate Gaussian is the bell curve for several measurements at once. Its centre is a list of means, one per measurement, and its shape is carried by a covariance matrix, a square table with one row and one column per measurement [S:62:56]. The matrix is symmetric, meaning the entry in row i and column j equals the entry in row j and column i. It must be positive definite, which is the condition that the ellipse, the oval traced by one level of the density, encloses an area instead of collapsing onto a line [S:32:81]. !Three covariance matrices and the clouds they describe, drawn at one and two standard deviations around 170 centimetres and 70 kilograms. The 42 in the corners tilts the ellipse upward to the right, a zero there stands it upright, and equal diagonal entries turn it into a circle. The determinants are 3,136, 4,900 and 2,401, all positive, so each matrix describes a real cloud. Status: synthetic.

Each entry off the diagonal is a covariance, a measure of the tendency of two variables to vary together [S:41:117]. When both measurements sit above their means at the same time, the two deviations share a sign, their product is positive, and the covariance is pulled upward [S:41:117]. Write the two measurements as x and z, with one row per person, and the covariance is the average of the paired products.

That sentence says: pair up the two deviations row by row, multiply each pair, and average the products. Here x-bar is the average of the x column and z-bar the average of the z column.

A covariance carries the units of both measurements multiplied together, so it is often divided by the two standard deviations to give a correlation between minus one and one [S:41:119]. The correlation is the lean measured on a scale that does not depend on whether you weighed in kilograms or in pounds.

Worked example. Suppose heights have a standard deviation of 7 centimetres, weights have a standard deviation of 10 kilograms, and their correlation is 0.6. The covariance is the correlation times the two standard deviations, so it is 0.6 times 7 times 10, which is 42 kilogram-centimetres [S:41:119].

The diagonal, the entries from top left to bottom right, holds the two variances, 49 square centimetres and 100 square kilograms. The off-diagonal entries hold the covariance twice, because the matrix is symmetric. The determinant of a two-by-two matrix is the product of its diagonal entries minus the product of its off-diagonal entries: 49 times 100 is 4,900, 42 squared is 1,764, and the difference is 3,136.

The square root of 3,136 is 56, and that 56 appears in the denominator of the density. The normalising constant, the factor in front that makes the total area under the density one, is $(2\pi {)}^{-1}$ times the inverse square root of the determinant, so here it is $1/(2\pi \cdot 56)$ [S:62:56].

Positive definiteness is a separate test, and for a symmetric two-by-two matrix it is exactly two conditions: the top-left entry is positive and the determinant is positive [S:25:219]. The determinant alone is the product of the two eigenvalues, so a positive value only says they share a sign; the matrix whose entries are minus 49, minus 42, minus 42 and minus 100 has determinant 3,136 and both eigenvalues negative. A negative determinant does rule out a density, because the eigenvalues then have opposite signs and the quadratic form is unbounded below.

Read the ellipse from the numbers. The positive 42 tilts it upward to the right, because height and weight rise together. Set the off-diagonal entries to zero and the ellipse stands upright with its axes along the two measurements, which is what a diagonal covariance matrix produces [S:62:57]. Make the two diagonal entries equal as well and the ellipse becomes a circle [S:62:57].

This object is the default model of a cloud for the rest of the book. A classifier assigns each new point to one of a fixed set of classes. One that assumes every class is a Gaussian replaces the hard problem of estimating one density per class with the simpler problem of estimating a mean vector and a covariance matrix for each class [S:49:167]. That density is one bell for the points in the class.

The idea costs something, and the price is in what a covariance can see. It records straight-line co-variation only [S:41:120], and its units are a product of two units, which is why it is hard to report directly [S:41:118]. The Gaussian is also an assumption about shape: a cloud can be curved, clustered or heavy-tailed, and two means and a matrix will then describe something you do not have. In many dimensions the surprise is sharper still, because most of a Gaussian's mass sits in a thin shell far from its centre rather than at the peak [S:62:57].

The formal statement. A random vector takes values in a list of d real numbers. Its mean is the list of the means of the measurements, and its covariance matrix puts the covariance of measurements i and j in row i and column j, with the variances on the diagonal [S:62:56]. The multivariate Gaussian has a density that is a constant times the exponential of minus one half of the squared Mahalanobis distance from the mean, where that distance is measured in units of the covariance matrix. The constant in front carries the inverse square root of the determinant of the matrix [S:62:56]. The matrix must be symmetric and positive definite [S:32:81].

The derivation of the ellipse. Because the covariance matrix is real and symmetric, it has real eigenvalues and an orthonormal set of eigenvectors [S:58:100]. Change coordinates to those eigenvectors. The quadratic form becomes a weighted sum of squares, with the eigenvalues as weights, so a level set is an ellipsoid whose axes point along the eigenvectors and whose half-widths scale with the square root of the eigenvalues [S:58:101]. The determinant is the product of the eigenvalues, and the volume of the ellipsoid scales with the square root of that product, which is why its inverse sits in the normalising constant [S:58:102].

The edge case. Positive definite is not decoration. If one eigenvalue is zero, the matrix is only positive semidefinite, the determinant is zero, the inverse does not exist, and the distribution is confined to a subspace of lower dimension with no density there [S:58:101]. In data terms, if one measurement is an exact linear combination of the others, the sample covariance is singular, and a classifier that inverts it fails.

The edge case of reading a covariance. Zero covariance means no straight-line co-variation, and it does not mean independence. Pearson's correlation measures linear relationships only [S:41:120]. Two measurements can be perfectly dependent through a curve and still have a correlation of zero. The ellipse summarises the linear part of the cloud, and nothing beyond it.

What the core hid. That the axes of the ellipse are the eigenvectors and that the widths are the square roots of the eigenvalues. The core read the tilt from the off-diagonal entry, which is the two-dimensional case of the same fact. It also hid a surprise of dimension. In a standard Gaussian in d dimensions the squared distance from the centre is a sum of d squared standard normals, which is a chi-square variable with d degrees of freedom [S:134:474]. That variable has mean d and variance 2d, so the distance itself is typically near the square root of d with a spread of order one. The relative spread of the squared distance is the square root of 2/d, which shrinks as d grows, and most of the mass sits in a thin shell rather than at the peak [S:62:57].

Where the analogy breaks. "A cloud" suggests a blob with a dense middle. In high dimensions the middle is nearly empty, and a sample from the model looks nothing like the mean vector [S:62:58]. The cloud is a shell.

Original sources. The density is equation 2.71 in Murphy's advanced text, with the mean vector and the covariance matrix named in the line beneath it [S:62:56]. Murphy also gives the reason the Gaussian is hard to avoid: among distributions with a given mean and covariance, it has the largest entropy [S:62:56]. The Bayesian multivariate normal model states the positive-definiteness requirement explicitly [S:32:81]. The geometry of the ellipse, with axes along the eigenvectors, is in section 2.3 of Bishop [S:58:100].

An open question. The Gaussian is the default cloud, and high-dimensional data are not Gaussian. Murphy shows that samples from a high-dimensional Gaussian sit in a thin shell, so a typical sample is far from the mean and looks like noise rather than like a prototype [S:62:58]. What remains unsettled is which departures matter for a given task: a model can have the right mean and the right covariance and still be wrong in the tails. Evidence that would settle a case: score the same data under a Gaussian and under a non-Gaussian alternative, then check which one predicts held-out events far from the centre.

A frontier result. The multivariate Gaussian generalises to the Gaussian process, a distribution over functions whose every finite collection of values is multivariate Gaussian. Vershynin states that the distribution of a mean-zero Gaussian process is completely determined by its covariance function [S:19:168]. The object you met as a matrix becomes a function of two indices, and the ellipse becomes a prior over curves.

What to read next. Read section 2.3 of Murphy for the density and its marginals [S:62:56], then section 2.3 of Bishop for the eigenvector picture [S:58:100]. Read section 7.1 of Vershynin for the Gaussian process [S:19:168]. The three readings go from a matrix, to its geometry, to a distribution over functions.