Diffusion

Set the noise so that each step keeps 80 per cent of the current pixel value and adds a random number with spread 0.6, and after ten steps only 10.7 per cent of the original value survives while the remainder is noise. A diffusion model learns to invert that process.

How does removing noise create an image? You fix a process that destroys a picture in small random steps, then train a network to undo one step at a time. Starting from pure noise and applying the network many times, a picture condenses out. !One network for the whole chain. The noisy value and the step number enter together, and the network returns its guess at the noise that was added. The same weights serve every step, so the parameter count does not grow with the length of the schedule. Status: schematic.

The corrupting half is called the forward process, and it contains no parameters at all. It gradually blends the input with samples of white noise until nothing of the input remains [S:74:363]. The restoring half is the reverse process, and it carries every parameter the model learns [S:74:363]. !The signal survives as 0.8 multiplied by itself once per step, and the noise spread climbs toward 1. At ten steps the signal is 0.107 and the noise spread is 0.994. Past that point every starting picture lands in the same cloud, which is why generation can begin from a fresh draw. Status: re-simulated.

A diffusion model is the pair: a fixed noising chain and a learned chain that reverses it [S:208:2]. The rates at each step form the noise schedule, and the schedule is more consequential than it first appears. A step that removes too much at once makes the reverse question ambiguous, because many different pictures could have produced the same noisy value. Small steps keep each reverse move close to a Gaussian, which is the form the decoder assumes [S:74:371].

The forward transition is one equation.

$$z_t=\sqrt{1-{\beta }_t}{z}_{t-1}+\sqrt{{\beta }_t}{\epsilon }_t$$

Here z is the noisy picture at step t, beta is the amount of noise assigned to that step, and epsilon is a fresh draw from a standard normal. The first term shrinks the old value, the second adds new noise, and the shrinkage keeps the variance from growing.

Training does not require the network to output a clean picture. It asks for the noise that was added, and the loss compares that guess with the noise actually used [S:74:375]. Each reverse step is a small denoising operation: the step subtracts most of the guessed noise, and repeating the operation removes the rest [S:74:364].

The algebra collapses to something plain. The derivation starts from a bound on the probability of the data. The practical loss, however, discards the per-step weights and becomes a least-squares comparison between the predicted noise and the true noise [S:74:377], which is two vectors, one subtraction and one squared length.

Prediction is considerably easier than generation. Given a noisy picture and the step number, the network faces a well-posed question, namely which noise vector was mixed in. The answer is one vector, and the network can be trained on as many pairs as it likes, because fresh noise turns one picture into a new example at every step [S:74:367].

Worked example. One pixel, value 1, three steps, with beta fixed at 0.36. The shrink factor is then 0.8 and the noise spread is 0.6, and every number below can be checked on paper.

After three steps the pixel still resembles its start, because three steps is not many. The signal after ten steps is 0.8 raised to the tenth power, which is 0.107, while the noise has spread 0.994, so a value near 1 is now buried.

Run the schedule for enough steps and the original value carries no information at all, because every starting picture lands in the same standard normal cloud. That is exactly why generation can begin from a fresh draw of noise.

The objective is not arbitrary. The exact variational bound carries a weight at each step, and the practical loss replaces those weights with a constant; that replacement is a reweighting, not a third objective [S:208:5]. DDPM report that the true bound gives better codelengths while the constant-weighted version gives the best sample quality, which is why the scores in their table belong to the practical loss [S:208:6]. The connection to denoising score matching explains why that reweighting works, and the weighting decides which noise levels receive attention during training [S:208:1].

The reverse step is trained, not derived. At each step the network estimates the noise, the step subtracts it, and a small amount of fresh noise goes back in so the chain stays stochastic [S:74:364]. The result is that noise is gradually removed until a realistic-looking picture remains.

One network serves every step. The step number enters as an extra input, usually encoded as a vector and added to the network's internal channels, so the model knows whether it is cleaning a barely touched picture or a nearly destroyed one [S:74:378]. A single set of weights therefore covers the whole trajectory, which keeps the parameter count manageable.

The cost is computational time. Sampling is a lengthy sequential procedure, and a good image can require a thousand network calls [S:74:378]. The same slowness buys quality, since diffusion models produce samples that exceed the realism of those from a GAN [S:74:363]. Their latent space also has no clean meaning, so you cannot read a direction in it the way you can with a variational autoencoder [S:74:384].

The formal statement. The forward chain is Markov with Gaussian steps:

$$q(z_t\mid z_{t-1})=\mathcal{N}\left(\sqrt{1-{\beta }_t}{z}_{t-1},{\beta }_{t}I\right)$$

The line says that each step shrinks the old value by a factor and adds fresh noise whose variance is set by the same number. The reverse chain is Gaussian with a learned mean. Four hypotheses hold. Each ${\beta }_t$ lies between zero and one. Each step depends on the previous value and on nothing earlier. The schedule drives the final marginal to a standard normal. The reverse conditional is close enough to Gaussian, which holds when the ${\beta }_t$ are small and the number of steps is large [S:74:371].

The derivation. Unroll two steps and collect the coefficients. The old value is multiplied by the square root of the product of the two shrink factors, and the two noise terms are independent Gaussians, so their variances add. Repeating the substitution gives the diffusion kernel: a single draw at step $t$ equals the shrunk original plus one Gaussian term whose variance is the remainder [S:74:367]. The kernel says that you can jump to any step in one draw, without running the chain. This account omits the induction. The induction is the same substitution repeated, and each repetition adds one independent variance.

The edge case. The core runs three steps by hand and concludes that the picture still resembles its start. The conclusion is right and the reasoning is incomplete. What matters is the ratio of signal to noise, not the surviving signal. After ten steps the signal is small and the noise spread is near one, so the ratio has collapsed even though the value still sits near its start. Two schedules with the same surviving signal at step ten can differ in the noise they added, and the ratio separates them. Workers in this area index the schedule by a log signal-to-noise ratio, which makes the comparison direct [S:211:2].

What the core hid. The core said the practical loss discards per-step weights and becomes a least-squares comparison. The discarded weights are not decoration. The exact bound carries a factor that depends on the noise amount, on the reverse variance and on the cumulative shrink [S:74:377]. Dropping the factor changes which noise levels the model attends to during training. The constant-weighted version is what produced the best reported samples, while the exact bound gave the better codelength; the connection to denoising score matching explains the choice, and it is a reweighting of the bound rather than a third objective [S:208:5-6]. The objective that is run and the objective that was derived are not the same object, and the run objective is still a bound.

Where the analogy breaks. The core says the model learns to run the film backwards. A film run backwards returns the exact frame. The forward process destroys information, so the reverse step cannot recover the start. It draws from a set of plausible starts, and fresh noise goes back in at every step [S:74:364]. The reverse chain is a sampler, not an inverse.

Original sources. Sohl-Dickstein and coauthors introduce the idea as destroying structure with an iterative forward diffusion process, then learning a reverse process that restores it [S:195:1]. Ho, Jain and Abbeel turn that idea into a competitive image model and report the CIFAR-10 scores the core quotes [S:208:1]. Prince's chapter states the chain in the form used here, derives the kernel in closed form, and is the account this section follows [S:74:363-367].

An open question. The field has not settled how to weight the per-step terms. The exact variational bound carries the weights ${\lambda }_t$, and the practical loss replaces them with a constant; that replacement is a reweighting, not a third objective [S:208:5]. DDPM report the constant-weighted objective as giving the best sample quality and the exact bound as giving the better codelength, so the choice between them is a choice of which quantity to optimise rather than a search over three weightings [S:208:6]. No derivation selects the constant weighting from first principles, and the reported choice is made by experiment [S:208:1]. Evidence that would settle it for a model family: sweep the weighting on one trained network, and measure held-out likelihood and sample quality for each setting. Show whether one weighting dominates or whether the best setting depends on the data.

A frontier result. Deterministic sampling cuts the cost. Denoising diffusion implicit models keep the training objective and replace the stochastic chain with a deterministic one, reporting sampling ten to fifty times faster in wall-clock time [S:207:1]. Prince reports good samples with fifty steps once the forward process stops being stochastic [S:74:379]. Flow matching goes further and regresses a velocity field along a chosen probability path, which contains the diffusion paths as special cases [S:213:1].

What to read next. Read Ho, Jain and Abbeel sections 1 and 2 for the chain and the bound [S:208:1-2]. Read Prince 18.2 for the kernel and 18.5 for the reparameterisation [S:74:367-377]. Then read Song, Meng and Ermon for the sampler that removes the noise from the chain [S:207:1].