What a generative model is

A grey picture 28 pixels wide and 28 tall is 784 numbers. Change one of those numbers by one step and you have a different picture. The set of possible pictures is vast, and a model that invents one has to know something about that whole set.

Learning a distribution means deciding where the probability goes: training raises the number under each observed picture while keeping mass for the pictures you have not seen. Every placement favours some pictures and starves others. A model that puts all its mass on the training list is a lookup table; one that spreads it evenly is a shrug. The rest of the chapter makes that placement precise. !Sixteen pictures and three ways to place probability on them. Each column is one two-by-two picture and the strip under it is the probability the model assigns to that picture. The memoriser fills four columns with 1/4, the bars-only model six with 1/6, and the uniform model all sixteen with 1/16; a grey tick marks a picture that receives nothing. Coverage is the count at the right of each strip. Status: synthetic.

A generative model synthesises new examples with statistics similar to the training data [S:74:284]. A probabilistic generative model additionally defines a distribution over the data and assigns a probability to each example [S:74:284]. The distribution is the object of interest, and a picture is one draw from it. A GAN is a generative model of the first kind only: it produces samples but defines no distribution over the data [S:74:290].

For a 784-pixel picture the table of probabilities has more entries than you could ever store. So the model stores a rule with a few parameters, and the rule produces the numbers on demand [S:74:284]. A small rule that fits a large table is the whole ambition of the subject. A rule with too many knobs fits the training list and fails everywhere else [S:74:286].

Training pushes probability onto the pictures you have. The standard objective maximises the probability of the observed data, which is the same as minimising the sum of negative log-probabilities [S:74:284]. Both sentences describe one action: raise the numbers under the training pictures.

Training data is a finite list. The distribution is a claim about everything that is not on the list. A model that gives probability zero to an unseen picture can never produce it, however good its copies of the training pictures are [S:74:285].

Some generators skip the distribution altogether. A GAN maps random noise to a picture and never writes down a probability [S:74:290]. That choice is legitimate, and it changes what you can ask the model afterwards.

Worked example. A two-by-two black-and-white picture has four pixels and two colours, so there are 16 possible pictures. Suppose the training set holds four of them: a top bar, a bottom bar, a left bar and a right bar.

Three models claim to have learned the set. The table gives the probability each one assigns to a picture.

Multiply the probabilities of the four training pictures. The memoriser scores 1/256, about 0.0039. The bars-only model scores 1/1296, about 0.00077. The uniform model scores 1/65536, about 0.0000153. On the data it saw, the memoriser wins by a wide margin.

Now take a picture outside the training set. The memoriser gives it probability 0. So does the bars-only model, unless the picture is one of the six pictures with exactly two pixels lit: four bars and two diagonals. The uniform model gives every picture 1/16, about 0.0625.

Coverage is the property that the samples represent the entire training distribution [S:74:285]. All three models produce every training picture, so all three cover the training set: coverage is 4/4 for each. What separates them is the support share, the fraction of the 16-picture space each can produce: the memoriser 4/16, the bars-only model 6/16, the uniform model 16/16. That is the table's last column, and it is not coverage.

The two measures coincide only when every picture in the space is a training picture. The uniform model reaches every picture and still produces grey mush.

Two uses follow from one distribution. You can ask it for the probability of a picture, or you can ask it for a sample. The first use scores and filters. The second use invents. A model can be good at one and poor at the other, which is why the checklist below separates coverage from sample quality.

The scoring use has a practical form. The probability a model gives to a picture can be thresholded, so an unusual picture is flagged as an outlier [S:74:286]. A model trained on digits can call a photograph of a cat unlikely without ever having seen a cat. That number is a judgement about the whole distribution, not about one pixel.

A checklist from one textbook names six properties of a good generative model: efficient sampling, high-quality samples, coverage, a well-behaved latent space, a disentangled latent space, and efficient likelihood computation [S:74:285]. No model in that book has all six [S:74:286]. The list is useful because it forces you to say which property you are buying and which you are giving up.

The cost is that probability and quality are different things. A diffusion model can produce sharper pictures than its likelihood score suggests, because much of its probability goes to fine details the eye ignores [S:208:2]. A model can also score well by spreading probability thin over everything, which is the uniform model again with better manners. The next two sections build a distribution out of hidden causes, the one after that builds one out of noise, and the last two show how to sample from it and how to judge whether it worked.

The formal statement. Fix a data space and a family of distributions $P_{\theta }$ indexed by a parameter vector $\theta$. A generative model is that family, together with a rule that turns a parameter setting into a sampler. Training chooses the setting that maximises the average log-probability of the observed examples. Three hypotheses carry the weight. The examples are independent draws from one distribution. Every member of the family assigns mass to every example, not only to the ones you hold. For continuous data the object is a density with respect to a stated measure, so its value is not a probability.

The derivation. Under independence, the probability of the whole training set is the product of the individual probabilities. A product of many numbers below one underflows, and its logarithm is a sum. Maximising the sum is therefore the same action as maximising the product, and dividing by the count does not move the maximiser. This account omits the existence of a maximiser. A neural family need not attain one, and training stops at a local optimum rather than a global one.

The edge case. The core counts how many outcomes have nonzero probability and calls that the support share. Support and mass are different properties. Give the memoriser a small floor on each unseen picture. Its support share jumps to the whole space while its samples remain the training pictures with probability close to one. Coverage of the training set was already full and does not move; what changes is the set of pictures the model can produce at all. A model can pass a support test and fail every practical one. This is why a held-out likelihood is a stronger test than a coverage count: it charges for the floor on every example the model meets [S:74:286].

What the core hid. The core said a small rule fits a large table. That is true of a factorised model and false of a raw table. A binary picture has as many possible values as its pixels have subsets, and no parameter count touches that number. The rule wins by sharing structure across pixels, not by compressing the table after the fact. The core also treated the probability of a picture as one number. For grey pixels the model returns a density, and its value depends on the units and on the preprocessing. A likelihood is comparable across models only when both saw the same representation.

Where the analogy breaks. The core called the distribution a table of probabilities that a rule produces on demand. A table has independent entries. A factorised rule ties entries together, which is why it generalises and why it cannot represent an arbitrary distribution. The analogy also suggests that producing an entry is free. For a diffusion model the entry is the endpoint of a chain, and the chain can take a thousand network calls [S:74:378]. Producing one entry costs the whole chain.

Original sources. The distribution-first definition is the textbook one [S:74:284]. The adversarial branch starts with Goodfellow et al. (2014), who frame the task as estimating a generative model through a two-player game [S:193:1]. That paper is the reason a generator can exist without a density, the point the core makes in its GAN paragraph [S:74:290].

An open question. The field has not settled how to certify that a model learned the distribution rather than a flattering subset of it. Held-out likelihood and sample quality rank models differently. The DDPM paper reports state-of-the-art samples and states that its log likelihoods are not competitive with other likelihood-based models [S:208:2]. The same paper finds that most of its codelength goes to image details the eye cannot see [S:208:2]. Evidence that would settle the question for a benchmark: score the same models by exact held-out likelihood and by a quality judgement made blind to those scores. Do this across several datasets, and report the disagreement instead of averaging it into one rank.

A frontier result. Diffusion models are the case that forced the split. They produce samples that exceed the realism of those produced by GANs, and they define a distribution while doing it [S:74:363]. If you take high likelihood as the definition of a good model, you have to explain that result. The explanation is that likelihood charges for imperceptible detail, and that charge is large [S:208:2].

What to read next. Read Prince chapter 14 for the six properties and the metrics that test them [S:74:284-286]. Read the first two pages of the DDPM paper for the likelihood admission and the Markov-chain definition [S:208:1-2]. Then read Prince chapter 15 for the GAN, which is the cleanest example of a generator with no distribution to score [S:74:290].