One hidden layer

Take the same four rows and place two units between the inputs and the answer. The first unit answers AND: weights 1 and 1 with a threshold of 1.5, so it fires only on the example (1, 1). The second answers OR: weights 1 and 1 with a threshold of 0.5, so it fires on every example except (0, 0). A third unit reads the two answers and fires when the second fires and the first does not.

$$\hat{y}=\text{step}(-h_1+h_2-0.5),h_1=\text{step}(x_1+x_2-1.5),h_2=\text{step}(x_1+x_2-0.5)$$ !Two views of the same nine parameters. On the left the four rows sit between the two hidden boundaries, x₁ + x₂ = 1.5 for the AND unit and x₁ + x₂ = 0.5 for the OR unit; the shaded band between them is the inputs that answer 1. On the right each row is replaced by its pair of hidden answers, so four rows become three points, and one line h₂ = h₁ + 0.5 separates the row labelled 1 from the two labelled 0. Carried back into the input plane that line is the band, because the two hidden boundaries are parallel: the answer region has two edges, and they do not meet. Status: re-simulated.

That line says the output combines two hidden answers with weights -1 and +1 and a threshold of 0.5, while each hidden unit applies a weighted sum of its own. Both hidden units share the weight vector (1,1), so their boundaries are the parallel lines x1+x2=0.5 and x1+x2=1.5, and the output selects the band between them. The added layer is what lets a boundary depart from a single straight line, but the departure here is a second straight piece rather than a bend.

Worked example. Run the four examples through the three units. Example (0, 0) produces weighted sums of 0 and 0, so both hidden units report 0, and the output unit receives -0.5 and answers 0. Example (0, 1) produces sums of 1 and 1: the first unit stays below 1.5 and reports 0, the second clears 0.5 and reports 1, and the output unit receives 0.5 and answers 1.

Example (1, 0) gives the same pair of reports by symmetry, and example (1, 1) gives sums of 2 and 2, so both hidden units report 1, the output unit receives -0.5, and the answer is 0. All four examples are correct, and the network contains nine parameters: six weights and three thresholds.

The two middle units are called hidden units, because nothing in the data records what they report [S:63:457]. Only the inputs and the final answers appear in the table. During a forward pass their values are computed and used, but no reader of the data can check them against a recorded number. That absence is what makes the layer a layer rather than another column.

The change of coordinates is the answer to the section question. Each hidden unit reports which side of its own boundary an input occupies, so the pair (h1, h2) is a two-number description of that input, constructed from the input rather than supplied with it [S:67:39]. The four original examples collapse to three descriptions: (0, 0), (0, 1) twice, and (1, 1). Three non-collinear points in the plane can be given any labelling by a line; three collinear points cannot, and the (+,-,+) labelling is the standard witness [S:75:53] [S:75:54]. The three hidden descriptions (0, 0), (0, 1) and (1, 1) are not collinear, which is why the single line h2-h1=0.5 separates (0, 1) from the other two.

Translated back into the original plane, that selection becomes the band 0.5 < x1+x2 < 1.5. On the unit square the boundary is the two parallel segments joining (0, 0.5) to (0.5, 0) and (0.5, 1) to (1, 0.5): two components, both straight, enclosing a convex region. A corner appears only when the two hidden weight vectors point in different directions, because the pieces of the boundary then meet where the hidden boundaries cross. What the hidden layer buys here is a boundary with two components, not a bend, and no boundary built from thresholds is curved, only piecewise linear.

The selection of AND and OR is not arbitrary either. XOR says that at least one input is on and that the two inputs are not both on, which is the conjunction of an OR with the negation of an AND. The two hidden units supply those two halves, and the output unit combines them, so the network is a sentence in logic written as a small circuit. The same trick produces every logical function of the inputs, which is why a hidden layer is such a large step in expressive power [S:63:458].

The construction also shows where the parameters come from. I chose the two hidden boundaries and the output boundary by hand, using the four examples as a guide, and the circuit works because I already knew the answer. Learning those nine numbers from the examples is a different problem, taken up in the last two sections of this chapter.

The guarantee that comes with one hidden layer is stated carefully, and the care matters. A network with one hidden layer, enough units and a suitable nonlinearity can approximate any continuous function on a closed and bounded region as closely as you like [S:66:177]. The theorem does not say that the units are few, and it does not say that training will locate them [S:66:177-178]. A sufficiently large network may be capable of representing a function that its training procedure never finds.

The gap between what a network can represent and what training can locate is where most practical work lives. In the worst case the number of hidden units required grows exponentially with the number of inputs [S:66:178]. One hidden layer is sufficient in principle, and principle carries a great deal of weight in that sentence.

The formal statement. Let $K$ be a closed and bounded region of ${\mathbb{R}}^n$, let $f$ be a continuous function on $K$, and let the activation be a squashing function, meaning bounded, non-decreasing and non-constant. For every $ϵ>0$ there is a network with one hidden layer, a linear output and finitely many hidden units whose output is within $ϵ$ of $f$ everywhere on $K$ [S:66:177]. The unit count is not part of the conclusion. The theorem says that a suitable finite network exists, and it fixes nothing about its size.

The derivation. The standard construction approximates $f$ by a sum of ridges. A single unit with large weights and a squashing activation behaves like an indicator of a halfspace: as the weights grow, the transition zone narrows. A sum of such indicators with adjustable output weights is a step function on a partition of $K$ into thin slabs. Choosing the slabs fine enough makes the sum uniformly close to $f$, because $f$ is continuous on a compact set and therefore uniformly continuous. The proof that the limit of such sums is dense in the continuous functions needs the Hahn–Banach theorem and a measure argument. I omit that step here. It is the reason the result is quoted as a theorem rather than derived in a paragraph.

The edge case. The conclusion is about existence, and the Core's careful reading stands: a network can represent a function that training never finds [S:66:177]. The size is the sharper edge. With binary inputs the number of functions is enormous, and representing one of them in the worst case needs a number of hidden units exponential in the input count [S:66:178]. A guarantee of the form "one layer suffices" is therefore compatible with a network too large to train, too large to store, and too large to evaluate on the data you hold.

What the Core hid. The Core said a "suitable nonlinearity". A second, weaker hypothesis, due to Leshno and coauthors, is that the activation is not a polynomial, and it covers the rectifier as well as the sigmoid; every squashing function already satisfies it, since a bounded non-constant function on the real line cannot be a polynomial [S:66:179]. A polynomial activation of degree $k$ is the case that fails: with $L$ layers the network is a polynomial of degree $k^L$. The Core also hid a counting fact that makes the bend concrete: with $m$ hidden units every piece of the boundary lies on one of the $m$ hidden lines, so width bounds how many directions the boundary can use.

Where the analogy breaks. The Core calls the hidden layer a change of coordinates. A change of coordinates is invertible, and this map is not. The four XOR rows collapse to three hidden descriptions, so two different inputs share one representation and the information is lost on purpose. What the layer builds is a partition of the input space, with one cell per distinct hidden pattern, and the output unit then draws a line across the cells. Depth composes partitions, and composition is where the corners multiply.

Original sources. The theorem is attributed to Hornik and coauthors and to Cybenko, and the extension to rectifiers to Leshno and coauthors [S:66:179]. The hand-built circuit that shows a hidden layer computes a logical function is Murphy's worked example [S:63:457] [S:63:458]. The width bound for a single layer is Barron's [S:66:178].

An open question. Rates. The theorem answers "can" and stays silent on "how many". For a broad class Barron gives size bounds [S:66:178], and in the worst case the count is exponential in the number of inputs [S:66:178]. The unsettled question is the width that a given depth needs to reach a stated error on a stated class of functions. Evidence that would settle it: matching upper and lower bounds for that class, both stated with constants and both checked on the same examples.

A frontier result. Spectral bias says networks learn low-frequency structure before high-frequency structure [S:308:1]. That is a statement about training rather than about representability, and it is the sharpest answer available to the gap the theorem leaves. The tool behind it is Fourier analysis of the piecewise-linear function that a rectifier network computes, which is why its claims are about frequencies rather than about units. Two networks with the same capacity can represent the same function and reach it at different speeds, so a capacity argument alone does not predict what a run will learn.

What to read next. Read the architecture-design section of Goodfellow twice, once for the statement and once for the list of things it does not promise [S:66:177] [S:66:178]. Read Murphy 13.2 for the circuit and for the sentence that says an MLP represents any logical function [S:63:458]. Read the opening of the spectral-bias paper for the training half of the story [S:308:1]. Keep the depth-versus-width comparison from the Dive into Deep Learning chapter beside it when you meet a very wide model [S:69:211].