How do you learn a hidden cause?

A colour photograph 64 pixels wide and 64 tall contains 12,288 numbers, while the causes behind it are few: the angle of the light, the pose, the colour of a wall.

How do you learn a hidden cause? You invent a variable you never observe, specify how it is distributed, specify how pictures depend on it, and tune both parts until the observed pictures become probable under the model. !Two causes for one pixel, with means 0 and 4 and equal priors. At x = 3 the bright cause is 55 times denser per unit, and the posterior for bright is 0.982. Give the dim cause prior 0.99 and the same pixel makes bright 0.356: a weak signal loses to a strong expectation. The left panel draws the prior-weighted densities; the right panel stacks the two posteriors. Status: synthetic.

Worked example. One pixel, two causes. A dim lamp and a bright lamp both illuminate a wall, and the pixel value is the only measurement you possess. !A latent model has three parts. The prior says which causes are plausible before any picture arrives, the likelihood says what a cause would look like, and the posterior is read off the picture afterwards. The solid arrows are the generative direction; the dashed return arrow is the posterior. Training tunes the prior and the likelihood, while the posterior is what the model concludes, not a parameter. Status: schematic.

Cause 1 puts the pixel at mean 0 with spread 1, while cause 2 puts it at mean 4 with the same spread. The two causes are equally likely beforehand, so each weight is 1/2.

The densities are Gaussian, and at x = 3 the bright cause is about 55 times more likely per unit than the dim one. Adding the last column gives 0.1232, and dividing the bright row by that total gives 0.1210 / 0.1232 = 0.9820.

That number is the posterior, the probability of a cause after the picture has been seen [S:74:348]. The pixel is almost certainly from the bright lamp.

Now name the three pieces against the numbers. The 1/2 above each lamp is the prior, the probability of each cause before any picture arrives. The Gaussian curve around each mean is the likelihood, the probability of a picture given a cause [S:74:341]. The 0.9820 is the posterior, what the model concludes after the picture.

The prior is not decoration. Give the dim lamp prior 0.99 and the bright lamp 0.01, and the same pixel becomes more likely dim, with the bright posterior falling to 0.3555. A weak signal loses to a strong expectation, which is the arithmetic behind every argument about base rates.

Call the picture x and the hidden cause z. A latent variable is a variable the model does not observe but uses to explain the data [S:74:341].

Write P for probability, and the model adds over causes.

$$P(x)=\int P(x\mid z)P(z)dz$$

The expression says that the probability of a picture is the likelihood of that picture under each cause, weighted by how likely the cause was, and accumulated over every possible cause.

The integral conceals the difficult part of the problem. For a latent-variable model with a neural likelihood the marginal has no closed form, so P(x) cannot be evaluated directly [S:74:345]. That is a property of the sum over causes, not of network likelihoods in general: a normalizing flow defines an exact density [S:74:363], and an autoregressive decoder an exact factorisation [S:74:284], so both evaluate the probability of a picture directly.

The model is an assertion about how the data was made. One paper states the assumption plainly, that the data come from a random process involving an unobserved continuous variable [S:192:2]. A cause can be a label, a short vector of numbers, or a whole hierarchy, and the choice is a modelling decision rather than a fact you read off the pixels.

The true cause never appears in the data. That same paper says the true parameters and the latent values are unknown to us [S:192:2]. Consequently the model is scored on the pictures it explains, not on whether its causes match the world.

With a discrete cause the sum is a finite list; with a continuous code it becomes an integral, and that difference decides what you can compute. A mixture of a hundred Gaussians is a sum of a hundred terms, which a machine evaluates directly. An integral over a hundred-dimensional code has no such list, which is the gap between the toy example and a real model.

Learning adjusts the means, the spreads and the priors, while the posterior is not a parameter at all. It is what the model concludes after each picture, and the parameters are theta, the numbers training changes.

Nothing here required a neural network, and that is the point. The same three pieces scale up: a prior over a short code, a network that turns the code into a picture, and a posterior that says which codes could have made the picture you hold [S:74:343].

A short code is the payoff, because the 12,288 pixel values are replaced by 100 numbers and the decoder rebuilds the remainder [S:74:343]. The reconstruction is not exact, since the code keeps the part that varies with the cause and discards the noise.

The cost is that the quantity you most want is precisely the one you cannot compute. The sum over causes has no closed form for a neural likelihood, and the posterior requires that same sum in its denominator [S:74:345]. The next two sections are attempts to avoid computing it; the last two show how to draw from the model and how to judge it.

The formal statement. Let $x$ be observed and $z$ unobserved. A latent variable model states a joint density that factorises into a prior and a likelihood:

$$P(x,z)=P(x\mid z)P(z)$$

The line says that the chance of a cause and a picture together is the chance of the picture given the cause, multiplied by the chance of the cause. The observed density is the marginal, obtained by summing or integrating $z$ out. The posterior is the conditional, obtained by Bayes' rule. Four hypotheses hold the construction together. A cause is drawn from the prior before any picture arrives. A picture is drawn from the likelihood once the cause is fixed. Both spaces carry stated measures, so the two densities are defined. The marginal integral exists and is finite.

The derivation. One identity gives Bayes' rule. The joint probability equals $P(z\mid x)P(x)$ and also equals $P(x\mid z)P(z)$. Set the two expressions equal and divide by $P(x)$. The denominator is the marginal, so the posterior is proportional to likelihood times prior. This account omits the evaluation of that denominator. For a latent-variable model with a neural likelihood the integral has no closed form [S:74:345]. The proportionality is enough to compare two causes at a fixed picture, which is why the core could compute a posterior for two lamps and could not compute a density for a picture.

The edge case. Take a code with two dimensions and a likelihood with two separated bumps. The posterior then has two peaks, and no single Gaussian sits on both. Prince states that when the posterior is multimodal a Gaussian approximation is poor [S:74:351]. The core's phrase about which codes could have made the picture invites you to read a posterior as a location. It is a distribution, and a summary such as its mean can fall in a region the model considers unlikely.

What the core hid. The core separated a finite sum from an integral and said the difference decides what you can compute. That is right, and it hides a second difference. In the mixture the sum runs over causes the model already contains, so the posterior weights are exact. In the continuous model the integral runs over a continuum, and every numerical method replaces it with a finite set of points. A hundred-term sum and a hundred-sample estimate look alike and carry different errors. The first is exact. The second has variance.

Where the analogy breaks. The core calls $z$ a hidden cause. In this construction $z$ is a variable that makes the data likely, not a cause you could intervene on. Two models with different codes can fit the same pixels equally well, because exchanging latent variables gives an equivalent model [S:66:257]. The fit is therefore judged on pictures, not on the recovered causes, and that is a property of the family rather than a defect of one member.

Original sources. Kingma and Welling state the generative process in two steps, a latent value drawn from a prior and an observation drawn from a conditional [S:192:2]. They also state the negative fact that the true parameters and the latent values are unknown to us [S:192:2]. The marginal that follows is intractable in their setting, so the posterior has no closed form either [S:192:2-3]. The mixture of Gaussians is the textbook case where the marginal is a finite sum over causes [S:74:342].

An open question. The field has not settled when a latent variable can be read as the cause that generated the data. Models with latent variables are often not identifiable, because exchanging two latent variables gives an equivalent model [S:66:257]. Weight-space symmetry is the easy case, and it is not the only one [S:66:257]. A model can fit held-out pictures well while its code means something else entirely [S:66:257]. Evidence that would settle a claimed recovery: plant a known code in synthetic data that the model family can represent, fit the model, and show that the recovered code matches the planted one up to a stated transformation.

A frontier result. Diffusion models keep this template and change one choice. The code becomes a chain of noisy copies, and the encoder is fixed rather than learned [S:74:363]. The approximate posterior is then known in advance, so training fits the reverse chain instead of fitting an encoder. The model still carries a hidden variable at every step, and the posterior over the whole chain remains intractable [S:208:2]. The template survived, and the hard part moved.

What to read next. Read Kingma and Welling sections 2.1 and 2.2 for the generative process and the bound [S:192:2-3]. Read Prince 17.1 for the mixture and 17.4 for the posterior [S:74:342-349]. Then read Goodfellow, Bengio and Courville on non-identifiability before you interpret any latent direction as a cause [S:66:257].