Neighbours and weight sharing

A photograph reaches a computer as a grid of numbers, and a one-megapixel image holds a thousand rows of a thousand values. A layer, one stage of the computation, connects that grid to another grid of the same size. Give it a full set of weights, the numbers it multiplies its inputs by, one weight for every pair of positions, and the layer requires a trillion weights [S:69:276]. A five-by-five kernel over the same image requires 25 weights and one bias, a single number added to every output [S:58:288]. The difference between those two numbers decides whether a layer is trainable at all, whether training can set its numbers from examples.

That gap is the cost. The reason a neighbour carries information is a fact about the data: nearby values are more alike than distant ones, so a small window already holds most of what a detector needs. A model therefore reads restricted neighbourhoods and applies the same detector at every position, so a visual pattern learned in one corner of an image remains available at every other corner. !One three-tap kernel, with entries 1, 0 and −1, slides along the row 0, 0, 5, 5, 5, 0, 0. The output is zero where the row is flat and takes opposite signs at the falling and rising edges, because the same three weights meet the same slope in two directions. The kernel never learns where it is; its five stopping positions produce five numbers, and the seven-value row loses its two ends to the window. Status: schematic.

The arrangement is a convolution, a sliding weighted sum. Place a small kernel over one patch, multiply each value by its weight, add the products, move one step, and repeat the operation at every position [S:58:288]. Each stop, one placement of the kernel, gives one output. A shared weight serves every position, so one short list of numbers covers the whole image [S:58:288]. The kernel is not a hand-drawn rule, because its entries are weights and training sets them from examples. !A fully connected layer needs one weight for every pair of input and output positions, so its parameter count rises with the square of the pixel count. A 5 by 5 kernel needs 26 numbers wherever it is applied, so the two lines separate by ten orders of magnitude at a megapixel and by more as the image grows. The flat line is the whole claim of weight sharing. Status: synthetic.

Bishop's textbook lists three mechanisms in a convolutional network: local receptive fields, weight sharing and subsampling, which shrinks a map, a grid of outputs, by summarising small patches [S:58:288]. The first two do the work in this section. Each output sees a small local receptive field, the patch of input it can read, and every output in a row shares the same kernel. The row of outputs is a feature map, which records one detector's answer at each position [S:58:288]. A layer usually holds several maps, one for each kernel, so the same patch feeds many detectors at once [S:58:289].

In one dimension the arithmetic is short. For an output at position $j$,

That line says the output at one stop is a weighted sum of the values under the kernel. The summation sign adds one product for each entry $a$ of the kernel, and $a$ runs over the three entries when the kernel is three wide.

Worked example. Take a row of seven values, 0, 0, 5, 5, 5, 0, 0, and a three-tap kernel, one with three entries, 1, 0 and -1. Slide the kernel across the row and add the three products at each stop.

Read each column downwards: the first product pairs the kernel's first entry with the window's first value. The output is zero where the row is flat and large where it changes, and the sign separates a falling edge, values dropping from 5 to 0, from a rising one, values climbing from 0 to 5. The same three weights found both edges, because the kernel never learns where it is. With no padding, the kernel cannot hang over either end, so a kernel of width three turns seven values into five outputs. The set of stopping positions is itself a design decision.

Now count the parameters, the numbers that training sets. A full layer with five outputs and seven inputs learns 35 weights and five biases, forty parameters in all. The shared kernel learns three weights and one bias, four parameters in all. The kernel gives up the freedom to treat each position differently, and on this tiny row the sharing buys a factor of ten.

The same trade on a megapixel image happens in two steps. Dropping the dependence on position takes the layer from about a trillion parameters to about four million, a factor of two hundred and fifty thousand, or about five and a half orders of magnitude [S:69:276]. Restricting each unit to a small window takes it from there to $4{\Delta }^2$, a few hundred parameters for a window whose half-width $\Delta$ is below ten, and the text calls that another four orders of magnitude [S:69:277]. The two steps together remove about ten orders, and sharing accounts for about five of them.

The reduction is the point, and the assumption is the price. A convolutional layer states that the same local pattern matters wherever it appears, which is a claim about the world rather than a fact about arithmetic [S:69:277]. When that claim holds, the layer requires far fewer examples to learn a detector. When it fails, no amount of training data will make the layer see a pattern that only makes sense at one absolute position. That is an inductive bias, an assumption built into the model's structure, and its cost appears whenever the data violates the assumption.

The gain has a name. A convolution is translation equivariant: shift the input, and the output shifts by the same amount and is otherwise unchanged [S:58:288]. Nearby pixels are more alike than distant ones, which is the fact about the data that the first page named [S:58:287]. Stack the layers and the reach grows, so two layers of three-tap kernels see five values and three layers see seven. Deeper stacks build wider views out of narrow ones.

The alternative is worse than it looks. A fully connected layer, in which every input reaches every output, has no reason to give the same answer to the same patch in two places. It must learn each detector again at every position, and a shifted copy is a new pattern unless the training data taught it otherwise [S:69:276]. The convolutional layer bakes both habits, locality and shared weights, into its structure.

The bargain fails when distance is the point. A sentence whose first word must meet its last needs many layers, and each layer widens the view by only a fixed amount [S:217:6]. The same-pattern-everywhere assumption also fails for text, where a word's position can change its role. The next section takes another route: read the sequence one item at a time and carry a memory forward.

The formal statement. Take an input row $x_1,\dots ,x_n$ and a kernel of length $k$, with entries $w_0,\dots ,w_{k-1}$ and one bias $b$. With stride one and no padding, the layer returns

$$y_i=b+\sum _{a=0}^{k-1}{w}_a{x}_{i+a},i=1,\dots ,n-k+1.$$

The line says each output is a weighted sum of the $k$ values under the kernel, and the same $k$ weights serve every output. The hypotheses are: the row is longer than the kernel, positions advance by one, values outside the row are not read, and one channel arrives at a time [S:58:288].

The derivation. Count the numbers that training sets. A dense map from $n$ inputs to $m$ outputs holds $mn$ weights and $m$ biases. The shared version holds $k$ weights and one bias, whatever $n$ is. Bishop's feature map makes the gap concrete: 100 units on a $10\times 10$ grid, each reading a $5\times 5$ patch, hold 25 weights and one bias for the whole map [S:58:288].

The count falls in two steps, and the core now keeps them apart. Dropping the dependence on position takes a megapixel layer from $10^{12}$ parameters to about $4\times 10^6$ [S:69:276]. Restricting each unit to a small window takes it from there to $4{\Delta }^2$ [S:69:277]. Weight sharing is the first step. Locality is the second. Each buys a different property, and each carries its own assumption.

The proof. Shift the row one place left, so the shifted row holds $x_{i+1}$ where the old row held $x_i$. Then

$$y_i(\text{shifted})=b+\sum _a{w}_a{x}_{i+1+a}=y_{i+1}(\text{old}).$$

The output moves one place and changes in no other way. The proof uses one fact: $w_a$ carries no index $i$. That is why sharing, and not locality, produces equivariance.

The edge case. Put stride two in the same layer. Outputs then sit at every second position, and a shift by one changes which values enter each output. Equivariance survives only for shifts that are a multiple of the stride. Zero padding produces the same failure at the two ends, over $k-1$ positions. The core's sentence, shift the input and the output shifts, is exact in the interior of a stride-one layer.

What the core hid. It read the parameter count as evidence about data. The count bounds capacity, and a bound on capacity is not a bound on the examples a detector needs. That step needs a generalisation argument, which this layer does not give. The omission is deliberate, because the argument needs capacity measures that arrive later. The core's weaker claim, that fewer parameters make a detector cheaper to learn, survives.

Where the analogy breaks. Calling a feature map unit a detector suggests a yes or a no. The unit returns a weighted sum through a nonlinearity, and its value depends on the whole window, not on the presence of a named feature. Two different windows can produce the same response.

Original sources. Bishop names the convolutional network for handwritten digits and cites Le Cun et al. 1989 and LeCun et al. 1998 [S:58:287]. He lists the three mechanisms the architecture uses: local receptive fields, weight sharing and subsampling [S:58:288]. D2L separates the two principles behind them and gives each a lineage: translation invariance from Zhang et al. 1988, and time-delay neural networks from Waibel et al. 1989 as an early use of the shared window [S:69:277].

An open question. How much data replaces the convolutional assumption? The vision transformer states the trade in its own words. Transformers lack the translation equivariance and locality of convolutional networks, and so "do not generalize well when trained on insufficient amounts of data" [S:229:2]. With 14M to 300M images the authors report that "large scale training trumps inductive bias" [S:229:2]. What remains unsettled is the shape of the transition. Nothing in this corpus states how many images per parameter are needed before a patch transformer matches a convolutional network at equal compute. Evidence that would settle it: one sweep of training-set size at fixed model size, fixed augmentation and fixed optimiser, run for both architectures, reporting the size at which the two curves cross.

A frontier result. The vision transformer applies this section's idea at one remove. It cuts an image into patches and feeds the linear embeddings of those patches to a standard transformer encoder [S:229:1][S:229:3]. A patch is a local window, so the convolutional assumption survives at the input and disappears inside the model. The method depends on the parameter sharing that makes a long sequence of patches affordable, and it works only once the data is large.

What to read next. Read Bishop 5.5.6 for the three mechanisms and the shift argument [S:58:288]. Read D2L 7.1 for the two separate reductions and for the sentence that names the price as an inductive bias [S:69:276][S:69:277]. Then read the first three pages of the vision transformer for the patch embedding and the data-scale result [S:229:1][S:229:3].