Derive: Estimating From Data

Write $Y-T=(Y-\theta )+(\theta -T)$. Expanding the square gives $$(Y-T{)}^2=(Y-\theta {)}^2+2(Y-\theta )(\theta -T)+(\theta -T{)}^2.$$ Take expectations. The first term is $\mathbb{E}[(Y-\theta {)}^2]=\mathrm{Var}(\epsilon )={\sigma }^2$. The last term is $\mathbb{E}[(\theta -T{)}^2]$. Because $\mathbb{E}[T]=\theta +b$, this is $\mathrm{Var}(T)+(\mathbb{E}[T]-\theta {)}^2=v+b^2$. For the cross term, $Y-\theta =\epsilon$ has mean 0 and is independent of $T$, so $\mathbb{E}[\epsilon (\theta -T)]=\mathbb{E}[\epsilon ]\mathbb{E}[\theta -T]=0$. Therefore $$\mathbb{E}[(Y-T{)}^2]={\sigma }^2+v+b^2.$$ The term ${\sigma }^2$ is the irreducible noise, the term $v$ is the variance of the fit, and $b^2$ is the squared bias, exactly the three terms of section 13.6. Section 13.2's formula $\mathrm{MSE}(T)=v+b^2$ is the special case with no new observation: take ${\sigma }^2=0$ and $Y=\theta$. The prediction problem adds the noise floor that no estimator can remove. Now put $b=-1.5$, $v=4$ and ${\sigma }^2=2.5$. The squared bias is $(-1.5{)}^2=2.25$, so the total is $2.5+4+2.25=8.75$.

For $T_0=X$, the variance is 1 and the bias is 0, so $\mathrm{MSE}(T_0)=1$ for every $\theta$. For $T_1=X/2$, the variance is $1/4$. Its expected value is $\theta /2$, so its bias is $\theta /2-\theta =-\theta /2$, whose square is ${\theta }^2/4$. Thus $$\mathrm{MSE}(T_1)=\frac{1}{4}+\frac{{\theta }^2}{4}.$$ The biased rule wins when $$\frac{1}{4}+\frac{{\theta }^2}{4}<1⟺{\theta }^2<3⟺|\theta |<\sqrt{3}.$$ At $\theta =1$ the values are 1 and $1/4+1/4=0.5$, so halving wins by 0.5. At $\theta =3$ the values are 1 and $1/4+9/4=2.5$, so halving loses by 1.5. At the endpoint $|\theta |=\sqrt{3}$ the two mean squared errors are equal: both are 1. The example is a counterexample to the claim that an unbiased rule always has the smaller mean squared error. It is also an edge case for the opposite claim: the biased rule does not win everywhere, because its bias grows with $\theta$ while the unbiased rule's error stays fixed. Section 13.2 makes the same point with two scales; here the crossover is a number you can compute.

For one observation $X$, the log-likelihood is $$\ell (p)=X\log p+(1-X)\log (1-p).$$ The first derivative is $${\ell }^{\prime }(p)=\frac{X}{p}-\frac{1-X}{1-p}.$$ The second derivative is $${\ell }^{\prime \prime }(p)=-\frac{X}{p^2}-\frac{1-X}{(1-p{)}^2}.$$ Take the negative expectation and use $\mathbb{E}[X]=p$: $$F(p)=-\mathbb{E}[{\ell }^{\prime \prime }(p)]=\frac{p}{p^2}+\frac{1-p}{(1-p{)}^2}=\frac{1}{p}+\frac{1}{1-p}=\frac{1}{p(1-p)}.$$ This is the expression section 13.4 states; the derivation shows where it comes from. For $n$ independent observations the log-likelihoods add, so the second derivatives add and the information is $nF(p)$. The Cramér–Rao bound says every unbiased estimator has variance at least $1/(nF(p))=p(1-p)/n$. At $p=0.25$ the product $p(1-p)=0.25\times 0.75=0.1875$. With $n=80$, $$\frac{1}{nF(p)}=\frac{p(1-p)}{n}=\frac{0.1875}{80}=0.00234375,$$ which rounds to $0.002344$ at six decimal places. The maximum likelihood estimate $\hat{p}=\bar{X}$ attains this variance exactly, because its variance is $p(1-p)/n$. The bound is local and assumes the Bernoulli model is right; a narrow bound in a wrong model measures the wrong parameter.