Vectors and the spaces they live in

A flat in Milan measures 18 square metres and has one room. A flat in Turin measures 24 square metres and has two rooms. Written as lists, the two flats are (18, 1) and (24, 2), and the order of the entries matters, because (1, 18) would describe a very different flat.

The chapter asks what a vector is and what a matrix — the rectangular table of numbers that §7.3 introduces — does to one. This section answers the first half. In this book a vector is an ordered list of numbers that you may add to another list and multiply by a single number. In general a vector is any member of a vector space, and the Deepen layer gives members that are not lists at all. !Milan is (18, 1) and Turin is (24, 2). Adding the two lists pairs the slots and gives (42, 3); doubling the first scales every slot and gives (36, 2); halving the sum scales every slot again and gives (21, 1.5). The vertical rule separates the two slots and no operation crosses it. The hatched entry is a legal vector and not a buildable flat, because the two moves never look at what a slot measures. Status: schematic.

Add the two flats entry by entry and you obtain (42, 3), and multiply every entry of the first list by two to obtain (36, 2). That lone multiplier is a scalar, and the two moves together are the entirety of vector arithmetic.

The display indicates that the sum of two lists is formed slot by slot, and a scaled list scales every slot. Doubling both flats before adding them gives the same answer as adding them and then doubling, which is one of the rules below doing its work.

Nothing above mentions arrows, angles or space, and that is the surprise. The arrow picture — a list drawn as an arrow from the origin — is only one picture among many, and it needs length and angle before it earns its keep. Here a space means a collection of lists, not a region of the page.

What matters is which properties lists of numbers share, so the definition preserves the properties. A set (a collection) with an addition, a scalar multiplication, and the usual rules is a vector space, and its members, the things in it, are the vectors [S:24:26]. The bracket names the source and the page inside it.

The rules are the ones you already use: addition commutes (the order does not matter) and associates (the grouping does not matter), and it has a zero and negatives. Multiplication by one changes nothing, scaling twice in a row is the same as scaling once by the product, $a(bv)=(ab)v$, and scalar multiplication distributes over addition, meaning that scaling a sum scales each part and then adds [S:24:26]. Textbooks often add two more, namely that a sum of two vectors stays inside the set and so does a scaled copy [S:23:94]. Those two closure conditions are the quiet ones, and every later section leans on them.

None of those rules is decoration: they are the licence for every rearrangement you already perform without thinking, from reordering a sum to pulling a common factor out of a bracket and collecting like terms. A structure that retained the lists but discarded one rule would break one of those moves, and that is why the definition bothers to list them.

Whole-number lists fail at once: halve (18, 1) and you land on (9, 0.5), which is not a whole-number list. A collection that leaks outside itself under a permitted move is not a vector space, which is why the closure conditions earn their place. The same failure sinks the set of lists with no negative entries, since scaling by minus one exits it at once.

The zero vector is the list of zeros, (0, 0), and adding it to any list moves nothing. The negative of (18, 1) is (-18, -1), and the two together give the zero vector, all of it legal for the flat example and none of it describing a building.

Now average the two flats, which means adding the lists and halving every entry.

The first entry of the average is a floor area you could actually construct, and the second is one and a half rooms. Nothing in the definition objected, because the definition never looked at what the entries mean; scale by minus one and you obtain (-18, -1), which is equally legal and equally unbuildable.

Lists are not the only vectors: polynomials, with the usual addition and the usual multiplication by a number, satisfy the same rules and form a vector space [S:24:44]. The functions from numbers to numbers do too: add two functions by adding their values at each input, and scale one by multiplying every value [S:24:44]. The lists stay in this book because a computer stores a vector as an array of numbers, and every calculation below reduces to the two moves on such an array.

A feature vector is the same object wearing a job title. One row of a data table gives one item, with a fixed meaning for each slot: a floor area in the first, a room count in the second. Swap two slots and you have not changed the arithmetic, but you have changed which measurement sits under which name, so every later step reads the wrong entry. The order is part of the object and not a convenience of printing.

The definition is generous, and that is its cost: any list of numbers qualifies once the two moves behave, so the word vector tells you nothing about whether an entry is a price, a count or a temperature. A program that adds a list of prices to a list of temperatures produces a vector and an answer with no meaning. The next section asks how few lists you need in order to build all the others, which is the first question that uses the rules rather than stating them.

The formal statement. Fix the real numbers as your scalars. A vector space is a set $V$, an addition $V\times V\to V$, and a scalar multiplication $\mathbb{R}\times V\to V$ such that the following hold for all $u,v,w\in V$ and all $a,b\in \mathbb{R}$. Addition commutes and associates. It has an identity $0$ and an inverse for every member. Scalar multiplication associates with itself, $(ab)v=a(bv)$, multiplication by $1$ fixes every vector, and the two distributive laws hold [S:24:26]. Some books write two further conditions, that a sum of two members lies in $V$ and that a scaled member does too [S:23:94]. Those two are not extra rules when the operations are defined as functions into $V$. They become rules only when you define the operations on a larger set and ask whether $V$ sits closed inside it.

What the rules already give you. Three familiar facts are theorems, not axioms. The zero vector is unique, the negative of a vector is unique, and every vector satisfies $0v=0$. The last one takes three lines.

$$0v=(0+0)v=0v+0v$$

The display says that writing the scalar zero as a sum changes nothing, and distributivity turns that into a statement about $0v$. Subtract $0v$ from both sides, using the inverse the rules provide, and $0v=0$ follows. Nothing in the argument reads the entries, which is the point of the whole structure.

The edge case. Take the positive real numbers. Define the sum of $x$ and $y$ to be the product $xy$, and the scaling of $x$ by $c$ to be $x^c$. Every rule holds, and the rule about scaling twice is the identity $(x^b{)}^a=x^{ab}$. The zero vector is the number $1$, the negative of $x$ is $1/x$, and the members are not lists at all. L1 said "the zero vector is the list of zeros", which is true of ${\mathbb{R}}^n$ and false here. The construction is not a curiosity: the logarithm turns this addition into ordinary addition, which is why ratios are often easier to model than raw counts. This is the edge case where the flat example misleads most, because the flat example trains you to look for a list.

What L1 hid. L1 said "every list of numbers qualifies once the two moves behave". The qualifier carries more weight than the sentence suggests, because the scalars must come from a field. L1 also compressed the rules into one sentence, and the flat example exercises only commutation and distribution. The list earns its place at the one example in your data where a rule fails, and at that point you need the rule written down.

Where the analogy breaks. The two flats invite you to read the entries as measurements with units. The rules never do. They hand you addition and scaling and nothing else: no length, no angle, no distance and no order come with them, and those are the extra structure someone must add and justify. A line through the origin is not extra structure. It is the span of a single vector, which the next section defines from the two moves alone [S:23:95].

Where the idea comes from. The definition you met is the one Axler gives: a set with an addition and a scalar multiplication satisfying the listed properties [S:24:26]. Hefferon writes the same definition with closure among ten numbered conditions, and he warns that the definition contains two kinds of addition and two kinds of multiplication [S:23:94]. The step from lists of numbers to points of meaning has a separate history. Speech and Language Processing traces it to three sources in the 1950s. The first is distributional linguistics. The second is Osgood's proposal that the meaning of a word is a point in a multidimensional Euclidean space. The third is the vector space model of information retrieval [S:98:125]. The same page dates Salton's model to 1971.

What the field has not settled. Whether a learned representation uses its vector space linearly. The open-problems list for mechanistic interpretability asks how far models encode concepts linearly, and how to characterise representations that are not linear [S:446:75]. The evidence that would settle it is causal rather than correlational. A direction found by probing should be sufficient to change behaviour when it is added, and necessary when it is removed, across many concepts and several models.

What depends on it. Sparse-autoencoder work now reports features that are not one-dimensional. A 2024 study defines irreducible multi-dimensional features and finds them in GPT-2 and Mistral 7B, including circular features for days of the week and months of the year [S:437:1]. The result uses the vector-space picture and complicates the simplest reading of it: a concept can occupy a subspace rather than a single direction. If you treat "vector" as "one number per concept", you will misread that paper.

What to read next. Read the first chapter of Axler for the definition and for the examples that fail one condition [S:24:26]. Read the second chapter of Hefferon for the closure conditions written out and checked [S:23:94]. Read the embeddings chapter of Speech and Language Processing for the route from distributional linguistics to vectors [S:98:125]. Then take a feature you believe is one-dimensional and write the experiment that would tell you apart from a two-dimensional subspace.