Derive: Making Pictures Out of Noise

The two densities are $$q(z)=\frac{1}{\sqrt{2\pi v}}\exp \left(-\frac{(z-m{)}^2}{2v}\right),p(z)=\frac{1}{\sqrt{2\pi }}\exp \left(-\frac{z^2}{2}\right).$$ Their logarithms are $$\log q(z)=-\frac{1}{2}\log (2\pi v)-\frac{(z-m{)}^2}{2v},\log p(z)=-\frac{1}{2}\log (2\pi )-\frac{z^2}{2}.$$ Take the expectation under $q$. The first term of $\log q$ is constant, and ${\mathbb{E}}_q[(z-m{)}^2]=v$ because $v$ is the variance of $q$. So $${\mathbb{E}}_q[\log q]=-\frac{1}{2}\log (2\pi v)-\frac{1}{2}.$$ For $\log p$, the variance identity gives ${\mathbb{E}}_q[z^2]={\mathrm{Var}}_q(z)+({\mathbb{E}}_{q}z{)}^2=v+m^2$. So $${\mathbb{E}}_q[\log p]=-\frac{1}{2}\log (2\pi )-\frac{m^2+v}{2}.$$ Subtract: $$D_{\mathrm{KL}}=-\frac{1}{2}\log (2\pi v)-\frac{1}{2}+\frac{1}{2}\log (2\pi )+\frac{m^2+v}{2}.$$ The two logarithms combine as $-\frac{1}{2}\log (2\pi v)+\frac{1}{2}\log (2\pi )=-\frac{1}{2}\log v$. Collecting the remaining terms, $$D_{\mathrm{KL}}=\frac{1}{2}\left(m^2+v-1-\log v\right).$$ That is the stated formula. At $(m,v)=(2,1)$ the logarithm is zero, so $$D_{\mathrm{KL}}=\frac{1}{2}(4+1-1-0)=2.$$ At $(m,v)=(1,1/4)$, $$D_{\mathrm{KL}}=\frac{1}{2}\left(1+\frac{1}{4}-1-\log \frac{1}{4}\right)=\frac{1}{2}\left(\frac{1}{4}+\log 4\right)=\frac{1}{2}(0.25+1.386294361)=0.818147180\dots ,$$ which is $0.8181$ to four decimal places. The expression is a sum of two non-negative pieces, $\frac{1}{2}(m^2)$ and $\frac{1}{2}(v-1-\log v)$, and the second is non-negative because $\log v\le v-1$ for every $v>0$ with equality at $v=1$. Both pieces vanish exactly when $m=0$ and $v=1$, which is the standard normal. So the distance is zero at the prior and positive at every other Gaussian, as a divergence must be.

Let $d_i=p_i-q_i$. Because $p$ and $q$ are both distributions, $$\sum _i{d}_i=\sum _i{p}_i-\sum _i{q}_i=1-1=0.$$ Split the indices into $P=\{i:d_i>0\}$ and $N=\{i:d_i<0\}$. The sum over $P$ and the sum over $N$ are equal in size and opposite in sign, so each has size $$\sum _{i\in P}{d}_i=-\sum _{i\in N}{d}_i=\frac{1}{2}\sum _i|d_i|.$$ For any subset $A$, $$p(A)-q(A)=\sum _{i\in A}{d}_i\le \sum _{i\in A\cap P}{d}_i\le \sum _{i\in P}{d}_i=\frac{1}{2}\sum _i|d_i|.$$ The same bound applies to $q(A)-p(A)$ by the same argument with the roles swapped, so $$|p(A)-q(A)|\le \frac{1}{2}\sum _i|d_i|$$ for every $A$. Choosing $A=P$ gives $p(P)-q(P)=\frac{1}{2}{\sum }_i|d_i|$, so the bound is attained and $$\underset{A}{\max }|p(A)-q(A)|=\frac{1}{2}\sum _i|d_i|=\mathrm{TV}(p,q).$$ Now let $U$ be a set with $q(U)=0$. Applying the identity to the event $U$, $$\mathrm{TV}(p,q)\ge |p(U)-q(U)|=p(U)-0=p(U).$$ So the real probability of any region the model cannot produce is a lower bound on the distance. Coverage and distance are not separate stories: an outcome the model never produces contributes its full real probability to the distance.

For the given vectors, the differences are $$0.45-0.40=0.05,0.35-0.60=-0.25,0.20-0=-\mathrm{0.20.}$$ The absolute values sum to $0.50$, so $$\mathrm{TV}=\frac{1}{2}(0.50)=\mathrm{0.25.}$$ The model gives zero to category 3, so $U=\{3\}$ and $p(U)=0.20$. The bound reads $0.25\ge 0.20$, and it is strict here because the model also puts too little mass on category 1 and too much on category 2. The uncovered mass sets the floor; the misallocation between the other categories adds the rest. This is why section 22.6 says a single distance number averages a missing category into a modest penalty: the missing category contributes $0.20$, and the remaining $0.05$ comes from errors that a reader would not call missing coverage at all.

Put $a=\sqrt{1-\beta }$. Unrolling the recursion gives $$z_t=a^t{z}_0+\sqrt{\beta }\sum _{k=0}^{t-1}{a}^k{\epsilon }_{t-k}.$$ The signal is the first term, so its fraction is $$a^t=(1-\beta {)}^{t/2}.$$ The noise terms are independent standard normals with coefficients $\sqrt{\beta }{a}^k$, so their variances add: $$v_t=\beta \sum _{k=0}^{t-1}{a}^{2k}=\beta \frac{1-a^{2t}}{1-a^2}.$$ Since $1-a^2=1-(1-\beta )=\beta$, this is $$v_t=1-a^{2t}=1-(1-\beta {)}^t.$$ That is the closed form. At $\beta =0.36$ we have $a=0.8$, so after five steps the signal is $0.8^5=0.32768$ and the noise variance is $1-0.64^5=0.8926258176$, whose square root is $0.9448$. After ten steps the signal is $0.8^{10}=0.1073741824$ and the variance is $1-0.64^{10}=0.9884707849$, whose square root is $0.9942$, matching the value section 22.4 quotes. As $t$ grows, $(1-\beta {)}^t\to 0$ for every $0<\beta <1$, so the signal vanishes and the noise variance tends to one. The terminal distribution is standard normal whatever the starting value, which is what lets section 22.5 start sampling from a fresh standard normal draw.

Now the edges. At $\beta =0$ the shrink factor is $a=1$, so the signal stays at $(1-0{)}^t=1$ and the variance is $1-1^t=0$ for every $t$. The forward process does nothing: no step changes the picture, and the "pure noise" starting point of the reverse chain is the wrong distribution. The reverse process would have to undo an identity map, and it would return its input unchanged, so it could not create a picture from noise. At $\beta =1$ the shrink factor is $a=0$, so after one step the signal is $0$ and the variance is $1$. Every starting picture lands in the same standard normal cloud after a single step. The reverse step receives a noise draw that contains no trace of the picture that produced it, so no decoder can tell which picture to restore; the information was destroyed in one move, and the reverse question has no answer. The useful schedule sits between the edges, with $0<\beta <1$, where each step destroys some information but leaves the reverse question answerable, and enough steps together reach the standard normal.