Kernel and image

A machine with rows (1, 2) and (2, 4) takes the list (2, -1) and returns (0, 0). Feed it (1, 0) and it returns (1, 2); feed it (0, 1) and it returns (2, 4). So one input disappears completely and every output remains on a single line.

The section question asks which inputs vanish and which outputs are reachable, and the two answers have names. The inputs that the machine sends to zero form its kernel, and the outputs it can produce form its image [S:25:70]. Both are sets of lists, and both are computable from the four numbers in the table. !The machine with rows (1, 2) and (2, 4) sends every multiple of (2, −1) to zero, so the kernel is that line. It sends (1, 0) and (−1, 1) to the same output (1, 2), and their difference is a kernel vector, which is why one output stands for a whole line of inputs. The reachable outputs are the multiples of (1, 2), so (3, 6) is reachable and (1, 3) is not; the image has dimension one and the kernel one, and 1 + 1 = 2 is the input dimension. The two panels are different planes. Status: schematic.

The kernel is the solution set of the equation that sets the output to zero. Here that means two equations: a first entry plus twice a second entry is zero, and twice the first entry plus four times the second is zero [S:25:70]. The second equation is twice the first, so it adds nothing, and the first says that the first entry is minus twice the second. Every vanishing input is therefore a multiple of (2, -1): (4, -2) vanishes, (-6, 3) vanishes, and the kernel is that whole line, which runs through the origin and therefore contains the zero list.

The image is built from the columns, because the output is always a combination of the two columns [S:25:70]. Column one is (1, 2) and column two is (2, 4), which is twice column one, so the reachable outputs are exactly the multiples of (1, 2). The machine cannot reach (1, 3), no matter which input you supply, and it can reach (3, 6) by sending in (1, 1).

The display demonstrates that the input (2, -1) is annihilated — sent to zero — because each row pairs with it to give zero.

Now count: the domain — the space of inputs — is the plane, of dimension two, and the kernel is a line, of dimension one. The image is also a line, of dimension one, and the two dimensions add to the dimension of the domain [S:23:215]. The dimension of the image has its own name: it is the rank of the machine [S:25:70]. Rank plus kernel dimension equals input dimension, always, which turns two separately computed numbers into one law.

The rank is a coarse instrument with three settings on the plane: rank two loses nothing, rank one flattens the plane onto a line, and rank zero crushes it to the origin.

The behaviour of a $2\times 2$ table is fixed by its two rows. When the second row is a multiple of the first, the rank is one, the kernel is a line, and the image is a line. When it is not, the rank is two, the kernel is the single zero list, and the image is the entire plane. The table in this section falls on the first side, and the table in the previous section falls on the second.

The accounting also explains the collision of inputs. The lists (1, 0) and (-1, 1) both land on (1, 2), because the rows give one and two for each of them. Their difference is (2, -1), which lies in the kernel, and that is not a coincidence: two inputs share an output exactly when their difference vanishes. So the kernel measures the machine's ambiguity, one line of it here, and a machine with a one-dimensional kernel associates every output with an entire line of inputs.

A non-zero kernel also settles uniqueness. If one input reaches an output, then every input of the form answer plus kernel element reaches the same output, so an answer is unique exactly when the kernel is the single zero list. The next section spends its time on elimination for this reason, since the kernel is the obstruction and elimination identifies it.

A shadow on the ground is the familiar analogy for a map that loses information, and it breaks in an instructive place. A shadow on the ground keeps the outline of the object and loses only its height, so the shadow of a pole is a line. A matrix offers no such courtesy: the table with every entry zero sends the entire plane to the origin, so its image has dimension zero and its kernel is everything. The word image is the mathematician's term for the reachable set, not a picture of the input. The next section uses the same table to answer an older question, namely when a system of equations has a solution at all.

The formal statement. Let $A$ be an $m\times n$ matrix. Its kernel is the set of inputs it sends to zero, and its image is the set of outputs it can produce. Both contain zero and both are closed under addition and scaling, so both are subspaces [S:25:70]. The rank is the dimension of the image [S:25:70]. The rank-nullity theorem states that the dimension of the kernel plus the rank equals $n$, the dimension of the input space [S:23:215].

The derivation. Take a basis of the kernel and extend it to a basis of the whole input space. Extending is the step left out, and it is the exchange argument of Section 7.2 applied to the kernel inside its space. Every input is a combination of the extended list. The kernel members die, so the images of the added members span the image. Suppose a combination of those images is zero. Then the same combination of the added members lies in the kernel, so it is a combination of the kernel basis. Independence of the extended basis forces every coefficient to vanish. The added images are therefore independent, and their count is $n$ minus the kernel dimension.

The edge case. Rank is an integer, and a small change in the entries can change it. The table with $1$ in the top-left corner, a tiny positive number in the bottom-right corner, and zeros elsewhere has rank two for every nonzero entry and rank one at zero. L1's three settings, two and one and zero, describe exact arithmetic. A computer rarely hands you a zero that is exact; it hands you a tiny number, and the question becomes how small counts as small. The condition number measures the ratio of the largest stretch to the smallest [S:28:103-104]. A more honest summary for data is the effective rank: for a symmetric positive-semidefinite table $\Sigma$ it is the sum of the eigenvalues divided by the largest eigenvalue, which lies between one and the exact rank and, away from the zero table, moves continuously with the entries [S:20:172][S:20:179].

What L1 hid. L1 said the kernel measures what the machine cannot distinguish and the image measures what it can produce. Both are properties of the map, not of the table, so the same map written in another basis keeps its rank. L1 also implied that you can read the kernel off the rows by inspection. For the repeated-row example you can, and in general you cannot: you compute it by elimination, which is the next section.

Where the analogy breaks. The shadow keeps the outline and loses the height, and a shadow is a projection, so casting it twice changes nothing. A general matrix is not a projection: apply it twice and you may land somewhere else again. The shadow also suggests that the image is a faithful picture of the input. The zero matrix settles the limit: its image is one point, its kernel is everything, and no shadow behaves that way.

Where the idea comes from. Treil defines the kernel as the solution set of the homogeneous equation and the range as the set of right sides for which the equation has a solution [S:25:70]. On the same page he defines rank as the dimension of the range, and he shows that the range is the column space. Rank-nullity appears in Hefferon as the sentence that a linear map's rank plus its nullity equals the dimension of its domain [S:23:215].

What the field has not settled. Whether the update a fine-tuning method learns is really low-rank, and what rank to use when it is. The LoRA paper asks the question in its own words: is the optimal adaptation matrix really rank-deficient, and if so what rank works in practice [S:237:10]. Evidence that would settle it is a measured spectrum of the update across many tasks and model sizes, together with a curve of task performance against rank that shows where the curve flattens. Until that curve is measured on your task, a rank is a guess.

What depends on it. Low-rank adaptation. LoRA constrains the update to a product of two thin matrices and reports up to 10,000 times fewer trainable parameters without loss on the tasks it tested [S:237:9]. The saving is a statement about the rank of the update, which is this section's rank, and the reported factor is a parameter count, not a claim that every update is low-rank.

What to read next. Read chapter 7 of Treil for the four subspaces and the rank [S:25:70]. Read the rank-nullity theorem in Hefferon for the counting argument [S:23:215]. Read section 3.13 of Thomas's notes for the same ledger written with the SVD [S:26:21]. Then, if you have a fine-tuning run of your own, plot the singular values of its weight change and look for the knee.