Practise: Making Pictures Out of Noise

The chosen list holds 64 of the 512 pictures, so the coverage is $$\frac{64}{512}=\frac{1}{8}=\mathrm{0.125.}$$ The model gives zero to every picture outside the list, so it can never produce one of the other 448. For two distinct chosen pictures the probability of the pair is the product $$\frac{1}{64}\cdot \frac{1}{64}=\frac{1}{4096}=0.000244140625,$$ which is $0.000244$ to six decimal places. The listed probabilities sum to $64/64=1$, so the table is a distribution. This is the memoriser from section 22.1 with a longer list: it scores well on the pictures it stored and covers one eighth of the space. A model with full coverage would give every picture a positive number; this one spends all its mass on 64 pictures and leaves the rest unreachable.

Bayes' rule gives $$P(\text{cause }1\mid x)=\frac{P(x\mid 1)P(1)}{P(x\mid 1)P(1)+P(x\mid 2)P(2)}.$$ With the stated priors the numerator is $(2/3)(1/4)=1/6$ and the second term in the denominator is $(1/3)(3/4)=1/4$. The denominator is $$\frac{1}{6}+\frac{1}{4}=\frac{2}{12}+\frac{3}{12}=\frac{5}{12},$$ so the posterior is $$\frac{1/6}{5/12}=\frac{2}{12}\cdot \frac{12}{5}=\frac{2}{5}=\mathrm{0.4.}$$ The observed picture favours cause 2 by a factor of three in likelihood, and the prior for cause 1 narrows that gap but does not close it. With equal priors the prior cancels top and bottom: $$\frac{1/4}{1/4+3/4}=\frac{1/4}{1}=\frac{1}{4}=\mathrm{0.25.}$$ So the same picture now points to cause 2 with posterior $0.75$. The prior mattered: it moved the answer from $0.25$ to $0.4$, which is the base-rate effect section 22.2 describes.

The reconstruction cost is $$(1-0.9{)}^2=0.1^2=\mathrm{0.01.}$$ The distance term for a one-dimensional Gaussian code against a standard normal prior is $$D_{\mathrm{KL}}=\frac{1}{2}\left(m^2+v-1-\log v\right).$$ With $m=1.5$ and $v=1$, the logarithm is $\log 1=0$, so $$D_{\mathrm{KL}}=\frac{1}{2}\left(2.25+1-1-0\right)=\frac{1}{2}(2.25)=\mathrm{1.125.}$$ The total loss is the sum, $$0.01+1.125=\mathrm{1.135.}$$ The distance term dominates because the code mean sits at $1.5$, far from the prior's centre at zero. A code mean of zero with variance one would make the distance term zero and leave only the reconstruction cost. The example shows the regulariser at work: it charges for a code that reconstruction alone would accept.

The variance is $0.25$, so the standard deviation is $\sqrt{0.25}=0.5$. The reparameterization trick writes the code as $$z=m+\sigma \epsilon ,$$ where $\epsilon$ comes from a fixed standard normal. For $\epsilon =1.2$, $$z=2+0.5(1.2)=2+0.6=\mathrm{2.6.}$$ For $\epsilon =-0.4$, $$z=2+0.5(-0.4)=2-0.2=\mathrm{1.8.}$$ Both codes lie near the encoder mean, one above and one below. The randomness enters through $\epsilon$, which is a separate input, so the gradient with respect to $m$ and $\sigma$ passes around the draw rather than through it. That is the whole point of the manoeuvre: the sampler is random, and the path used to train it is not.

The transition is $$z_t=\sqrt{1-\beta }{z}_{t-1}+\sqrt{\beta }{\epsilon }_t.$$ With $\beta =0.19$, the shrink factor is $\sqrt{0.81}=0.9$ and the noise scale is $\sqrt{0.19}=0.435889894\dots$. Step one: $$z_1=0.9(2)+0.435889894(0.5)=1.8+0.217944947=\mathrm{2.017944947.}$$ Step two: $$z_2=0.9(2.017944947)+0.435889894(-1)=1.816150452-0.435889894=\mathrm{1.380260558.}$$ Step three: $$z_3=0.9(1.380260558)+0.435889894(0.25)=1.242234502+0.108972474=\mathrm{1.351206976.}$$ Rounded to four decimal places, $z_3=1.3512$. The value has moved from $2$ toward zero and the two noise draws have pushed it in opposite directions, so the final value sits below the start. The shrink factor is the part that survives; the noise term is the part that accumulates.

The signal is multiplied by $0.8$ at every step, so after five steps its fraction is $$0.8^5=\mathrm{0.32768.}$$ The noise variance obeys $v_t=0.64v_{t-1}+0.36$ with $v_0=0$, whose solution is $$v_t=1-0.64^t.$$ At $t=5$, $$v_5=1-0.64^5=1-0.1073741824=0.8926258176,$$ and the standard deviation is $$\sqrt{0.8926258176}=0.944788769\dots$$ So the signal fraction is $0.3277$ and the noise spread is $0.9448$ to four decimal places. The signal has lost about two thirds of its size, while the noise has grown to nearly its terminal spread of one. Five steps is enough to blur the value but not enough to erase it; the next few steps finish the job.

The intervals are $$[0,0.1),[0.1,0.5),[0.5,0.75),[0.75,1].$$ The draw $0.62$ falls in the third interval, because $0.5\le 0.62<0.75$, so the sampler returns outcome 3. The draw $0.95$ falls in the fourth interval, because $0.75\le 0.95\le 1$, so the sampler returns outcome 4. The answer is (b). The procedure is the one section 22.5 describes: draw a uniform number, walk the cumulative probabilities, and return the outcome attached to the interval where the number lands. Outcome 4 has the same probability as outcome 3, and each owns an interval of width $0.25$; the widths, not the order, carry the probabilities.

The proportions are $p=(0.50,0.30,0.20)$ for the real data and $q=(0.80,0.20,0)$ for the generator. The total variation distance is $$\mathrm{TV}(p,q)=\frac{1}{2}\sum _i|p_i-q_i|.$$ The three absolute differences are $$|0.50-0.80|=0.30,|0.30-0.20|=0.10,|0.20-0|=\mathrm{0.20.}$$ Their sum is $0.60$, so $$\mathrm{TV}(p,q)=\frac{1}{2}(0.60)=\mathrm{0.30.}$$ The generator never produces category 3, which holds $20$ per cent of the real pictures, so its recall over the full set cannot exceed $0.80$. The distance of $0.30$ records that failure along with the excess mass in categories 1 and 2. One number hides which of the two failures dominates, which is why section 22.6 asks for precision and recall beside the distance.