Flow matching and evaluation

One generator, two numbers. On CIFAR-10, one diffusion model reports an Inception score of 9.46 and a Fréchet inception distance of 3.17 [S:208:1], and neither number is a picture.

How do you compare two generators? You compare the distributions they produce, not the pictures they happen to make. A single sample is an anecdote, while a thousand samples are evidence about coverage, fidelity and the shape of the distribution. !Precision is the share of generated samples that fall inside the data manifold, and recall is the share of real examples that fall inside the model's. In the drawing, five of ten model points sit inside the data region and six of eight data points sit inside the model's. One perfect sample gives high precision and almost no recall; a wide vague cloud gives the reverse. A single distance blends the two failures into one number. Status: schematic.

Worked example. Three categories, 100 real pictures, and two generators. The real split is 40, 35 and 25. !Three categories and 100 pictures. Generator B matches the real split within 0.08 in total variation; generator A is three times further at 0.25. Half of A's distance comes from one empty category, the rest from two moderate errors, so the average hides where the failure sits. The distance reports the average, the bars report the failure. Status: synthetic.

The distance in the last column is the total variation distance: half the sum of the absolute differences, divided by 100. For A that is half of 10 plus 15 plus 25, divided by 100, which is 0.25. For B it is half of 6 plus 2 plus 8, divided by 100, which is 0.08. B is the better match overall.

The table hides what A does well. A gets the first two categories nearly right and produces nothing in the third, so its precision on the first two is high and its recall over the whole set is poor. A single distance number averages that failure into a modest penalty, while a person looking at the samples would notice the missing category at once.

The precision and recall pair names that difference. Precision is the fraction of generated samples that fall inside the data manifold, the region where real pictures sit, and recall is the fraction of real examples that fall inside the model manifold [S:74:288]. A generator that makes one perfect picture has high precision and terrible recall.

A model can win on precision and lose on recall, or the reverse. Neither failure is visible in a single distance number, which is why careful comparisons report the pair alongside the distance.

The cleanest framing for the comparison comes from statistics. Deciding whether two sets of samples came from the same distribution is a two-sample test [S:69:920]. A generator is good when a two-sample test cannot separate its output from the real data. FID and precision-recall are statistics of that comparison. Held-out likelihood scores the model instead, and the Inception score uses only generated samples and a pretrained classifier, so it answers a different question and is not a two-sample test.

The most direct criterion is likelihood. Measure the probability the model gives to held-out pictures [S:74:286]. That number rewards coverage, because a model that ignores part of the data must assign low probability to the ignored part [S:74:286]. It fails for a GAN, which assigns no probability to anything [S:74:286], and it is expensive for a diffusion model.

Likelihood also disagrees with the eye. A model can score well on held-out likelihood and still produce samples the eye judges poor. Most of its codelength is spent on detail the eye cannot see: in DDPM more than half of the lossless codelength describes imperceptible distortion [S:208:2]. A good likelihood therefore does not guarantee a good-looking sample, without any claim that the model prefers a blurry average.

The fine print matters here. A likelihood for a variational autoencoder or a diffusion model is usually a bound rather than an exact value. Among the four families in Prince's table, only normalizing flows compute it exactly and cheaply: the GAN has none, and the VAE and diffusion model have only a bound [S:74:286]. Autoregressive models also compute it exactly, through the factorisation of a picture into one conditional per pixel, so the restriction is about that table rather than about generative models in general [S:62:857]. A comparison between a GAN and a flow has to fall back on sample metrics.

Image metrics take a different route. The Inception score feeds generated pictures to a pretrained classifier and asks two things: each picture should be confidently one class, and the classes should be spread out across the batch [S:74:287]. The score is the exponential of the average KL divergence between the per-image class distribution and the marginal class distribution, $IS=\exp (\mathbb{E}[D_{\mathrm{KL}}(Pr(y\mid x^{\ast })\Vert Pr(y))])$ [S:74:287].

It is built from a divergence rather than a distance, and the exponential is what turns a modest average divergence into a number near ten. The score needs a classifier whose label set matches the data. The ImageNet score uses a classifier trained on ImageNet's 1000 classes; a CIFAR-10 score needs a classifier whose label set is CIFAR-10's ten, and a reported 9.46 should name which one it used [S:74:287].

The Fréchet inception distance compares real and generated pictures in the feature space of the same kind of network [S:74:288]. It fits a multivariate Gaussian to each set of features and measures the distance between the two Gaussians [S:74:288]. Smaller is closer, and zero means the two fitted Gaussians coincide. That is a statement about the approximation, not about the feature distributions: two different distributions with the same mean and covariance receive the same score, so a zero can hide any difference the Gaussian fit cannot represent.

FID is a distance between the two fitted Gaussians, which is its strength and its weakness. It reacts to both realism and variety, but it cannot tell you which one failed [S:74:288].

Numbers also move when the measuring stick moves. The Inception score is sensitive to the classifier it uses, and retraining that classifier gives quite different values [S:74:287]. Two reported scores are comparable only when the same network, the same preprocessing and the same sample count produced them [S:74:287].

The cost is that every metric carries the assumptions of its measuring stick. FID fits Gaussians to features from a pretrained network, so it ignores anything that network discarded, and some of that discarded detail matters for realism [S:74:288]. The Inception score does not reward variety inside a class, so a model that produces one perfect example per class scores well [S:74:287]. A benchmark is an instrument with a known bias, and reading it without the bias is the mistake.

The formal statement. A two-sample test takes a batch of draws from one distribution and a batch from another, and decides between the hypothesis that the two are equal and the hypothesis that they differ [S:69:920]. The test has a level, the probability of rejecting a true hypothesis, and a power, the probability of rejecting a false one. A generator passes when the test fails to reject at the sample size you can afford. Every metric in the core is either a statistic for such a test or a score built for a different question.

The derivation for one metric. For two distributions on the same finite set, the total variation distance is half the sum of the absolute differences of their probabilities. It equals the largest gap the two distributions assign to any single event, which is why the core calls it a distance between distributions rather than between samples. The worked example applies the definition directly: pair the categories, subtract, take absolute values, add, halve, divide by the count. This account omits the proof that the largest gap is attained by the set where the first distribution exceeds the second. The proof takes two lines and needs only finite additivity.

The edge case. The Fréchet distance fits a Gaussian to each set of features and compares means and covariances [S:74:288]. That fit has a sample-size cost the core's table hides. A covariance in $d$ dimensions has on the order of $d$ squared entries, and with fewer than $d$ samples the estimate is singular. Two small batches then produce a distance that reflects the batch size as much as the generators. The repair is either many more samples or a metric that does not estimate a full covariance, and a reported number should name its sample count.

What the core hid. The core treated likelihood as a test that rewards coverage. That is true of held-out likelihood and false of training likelihood, which a memoriser maximises [S:74:286]. The core also treated precision and recall as exact fractions. Both are estimated against an approximate manifold built from hyperspheres whose radius is the distance to the nearest few neighbours [S:74:288]. Change the neighbour count and both numbers move. A precision of one means one thing at five neighbours and another at fifty.

Where the analogy breaks. The core calls a benchmark an instrument with a known bias. An instrument with a known bias can be corrected. The bias of the Fréchet distance depends on the feature network, the sample count and the preprocessing, and no calibration standard fixes it. The Inception score does not look at real data at all, so its bias is not an offset from a true value. It answers a different question, and a correction would need a second measurement [S:74:287].

Original sources. The two-sample framing comes from statistics and is stated in the GAN chapter of Dive into Deep Learning [S:69:920]. The Inception score and the Fréchet inception distance are Salimans et al. (2016) and Heusel et al. (2017), as Prince's notes record [S:74:289]. FID compares the Pool-3 feature statistics of Inception V3, while the Inception score is computed from the network's class-probability output over the 1000 ImageNet classes [S:74:287]. Prince's note says both are based on Pool-3, which conflicts with the definition two pages earlier; only FID reads those features [S:74:289]. Manifold precision and recall are Kynkäänniemi et al. (2019) [S:74:289].

An open question. The field has not settled a metric that separates realism from coverage. The Fréchet distance reacts to both and cannot say which one failed [S:74:288]. Precision and recall separate them, and both depend on an estimated manifold [S:74:288]. Prince's notes point to a detailed critique of the Inception score and to a survey of the strengths and weaknesses of the alternatives [S:74:289]. Evidence that would settle it: a benchmark with two controlled corruptions. One removes modes and leaves each remaining sample intact. The other blurs every sample and leaves the modes in place. A metric that orders the two cases correctly at a stated sample count would settle the question for that benchmark.

A frontier result. Flow matching changes the generator and leaves the evaluation intact. It trains a continuous normalizing flow by regressing a velocity field along a chosen probability path, and it reports better likelihood and sample quality than diffusion baselines, with fast sampling from standard ODE solvers [S:213:1]. The numbers that judge it include the ones in this section, which is why the measurement problem outlives any single generator.

What to read next. Read Prince 14.3 for the four metrics and their failure modes [S:74:286-288]. Read Ho, Jain and Abbeel section 1 for a reported pair of numbers beside the likelihood admission [S:208:1-2]. Then read Lipman and coauthors for the generator that the current metrics were not designed to describe [S:213:1].