Dot products and angles

Two shoppers leave a market: the first carries 3 apples and 4 loaves, and the second carries 4 apples and 3 loaves. Their baskets are close in composition without being identical, and you want a single number that says how close. Multiply matching entries and add them: 3 × 4 + 4 × 3 = 24.

The result is the dot product of the two lists, written $\mathbf{a}\cdot \mathbf{b}$, and it counts how much the entries agree while its size grows with the amount of shopping. Thirty apples with forty loaves, matched against forty apples with thirty loaves, scores 2400, a hundred times larger, because scaling each list by ten multiplies the dot product by 10 × 10 = 100. Dividing by the length of each list removes that dependence on scale. !Three two-entry lists, each of length 5, drawn from one origin, so only direction differs. A=(3,4) and B=(4,3) point nearly the same way and score a cosine of 0.96; A and C=(4,−3) are perpendicular and score 0. The scale below places those scores between −1 and 1, where the sign says which way the lists agree, not how much. Status: re-simulated.

Both baskets have length 5, since 9 + 16 = 25; the scaled basket has length 50, and 2400 divided by 50 and again by 50 returns 0.96. A list's length is its norm, written $\Vert \mathbf{a}\Vert$, the square root of the sum of its squared entries [S:23:52]. The bracketed codes mark a source and a page: S, then the source number, then the page number. The rest of the chapter uses this product to answer one question: why the best-fitting line is the one that minimises squared error.

Dividing 24 by 5 and again by 5 leaves 24/25 = 0.96, and that ratio survives any doubling of the shopping. The geometry of triangles gives the ratio a name: the cosine of the angle between the two lists [S:23:53]. In a right triangle the cosine of an angle is the side next to that angle divided by the longest side. For two lists drawn as arrows from a common point, the angle is the one between the arrows, and its cosine is the number below.

The fraction is the dot product divided by the two norms, and it is the number a calculator returns for the cosine of the angle between two lists. To turn a cosine back into an angle, ask the calculator for the inverse cosine: the angle whose cosine is 0.96 is about 16 degrees. Read the cosine as a score that runs from −1 to 1: values near 1 mean the lists point the same way, values near 0 mean they meet at a right angle, and negative values mean they point in opposite ways. Two lists score 0 exactly when they are orthogonal, which is another word for perpendicular [S:24:201].

A cosine near zero is a statement about directions, not about relatedness: neither list has had its average subtracted. Two lists that share a large common part score near 1 however differently their remaining entries move, and lists whose entries are all positive cannot score near 0 unless their non-zero entries sit in different positions. An embedding table shows the effect at its worst: its vectors all lean in one shared direction, so unrelated words score high until the entries are centred [S:98:218]. Reading relatedness off a cosine takes that centring first.

The sign matters, and it is easy to lose. A cosine of −0.9 is a stronger statement than a cosine of −0.1, because both lists are built from the same ingredients with opposite signs, like a basket of apples and a basket of debts for apples. A cosine near 0 is the weak case, where the two lists share nothing that survives the comparison although either of them may be long. Reading the score as a percentage of agreement would be wrong, since a negative score is not a small amount of agreement but a disagreement.

Nothing in the calculation requires two entries: the same operations work for a list of a thousand entries, and the angle is defined there too, although you cannot draw it [S:23:53]. The cosine also never leaves its range, because the absolute value of the dot product of two lists is at most the product of their norms, with equality exactly when one list is a multiple of the other [S:22:293][S:24:203]. The absolute value carries both ends at once: it says $\mathbf{a}\cdot \mathbf{b}\le \Vert \mathbf{a}\Vert \Vert \mathbf{b}\Vert$ and also $-\mathbf{a}\cdot \mathbf{b}\le \Vert \mathbf{a}\Vert \Vert \mathbf{b}\Vert$, so the ratio lands between −1 and 1.

That bound is the Cauchy–Schwarz inequality, and the Deepen derives it from the squared length of a list built as one list minus a multiple of the other. Without it the ratio could exceed 1 and no angle would correspond to the score.

An angle compares shape and ignores size, which is the property machine learning borrows. A feature is one column of a data table, one measurable property of an item. Two baskets with the same proportions score a cosine of 1 even when one holds a hundred times more food, and two baskets of equal total weight can score a cosine near 0 when they hold different goods. When a row of data records how much of each feature an item has, the cosine asks whether two items are made of the same ingredients rather than whether they are large.

The dot product underneath is a weighted sum: each entry multiplied by a number, its weight, and the results added. A linear model predicts with a weighted sum of the features, and fitting chooses the weights rather than being given them.

Worked example. Three lists make the arithmetic visible.

Lists A and B give a dot product of 24 and a cosine of 0.96, so they agree closely. List C is A with its second entry flipped, so A and C give 12 − 12 = 0, and those two lists sit at a right angle. The pair B and C gives 16 − 9 = 7, with cosine 7/25 = 0.28. The angle whose cosine is 0.28 is about 74 degrees, which the inverse-cosine key returns. One changed sign moved the second list from a close match to a right angle, and the arithmetic shows why: the cosine adds signed agreements, so a disagreement subtracts from the total instead of being ignored.

The cosine has a blind spot: it ignores magnitude. For any list $\mathbf{b}$ and any positive number $\lambda$, the cosine of $\lambda \mathbf{a}$ with $\mathbf{b}$ equals the cosine of $\mathbf{a}$ with $\mathbf{b}$. A list of 3 and 4 therefore scores against a fixed list exactly what one of 3000 and 4000 scores; a negative $\lambda$ flips the sign. That is why a tiny basket can agree perfectly with a huge one. The score also treats every entry as equally important and measured in the same units, which fails when one column is in kilograms and another in millimetres.

Scaling the entries before you compare them is not optional in those cases, because you have chosen the units and the scaling, and a different choice returns a different similarity. That choice is part of the model, not a detail of the arithmetic, and two analysts can compare the same table and report different similarities without either of them making an arithmetic mistake. The next section puts the dot product to work on a narrower question: given a list and a surface, which point of the surface is closest?

The formal statement. An inner product on a real vector space assigns a number $⟨a,b⟩$ to each pair of lists, with three properties. It is symmetric, so $⟨a,b⟩=⟨b,a⟩$. It is linear in each slot. It is positive definite, so $⟨a,a⟩>0$ whenever $a$ is not the zero list. The dot product ${\sum }_i{a}_i{b}_i$ is one example [S:23:53]. The norm is $\Vert a\Vert =\sqrt{⟨a,a⟩}$.

The derivation. Cauchy–Schwarz comes from a square that cannot be negative. For every real $t$,

$$\Vert a-tb{\Vert }^2=\Vert a{\Vert }^2-2t⟨a,b⟩+t^2\Vert b{\Vert }^2\ge 0.$$

Read the right side as a quadratic in $t$. It opens upward and never dips below zero, so its discriminant cannot be positive. That gives $⟨a,b{⟩}^2\le \Vert a{\Vert }^2\Vert b{\Vert }^2$, and taking square roots gives the inequality [S:24:203]. When $b$ is the zero list, both sides are zero and the argument is vacuous. Every other case needs no extra hypothesis.

The inequality makes the cosine a definition rather than a hope. Divide the inner product by the two norms and you land in the interval from −1 to 1. Every number there is the cosine of exactly one angle between 0 and π, so the angle exists and is unique. L1 said the ratio survives any doubling. The deeper reason is that the numerator and the denominator scale by the same factor.

The edge case. The cosine is not a property of two lists alone. Choose a symmetric positive definite matrix $M$ and define the weighted inner product $⟨a,b{⟩}_M=a^{\mathsf{T}}Mb$. The three properties hold, so Cauchy–Schwarz holds, and a cosine follows. Reweighting the entries changes the angle while the lists stay fixed. Two analysts who scale their columns differently therefore report different similarities, and neither has made an arithmetic mistake. L1's blind spot named scaling; the formal version says the metric itself is a modelling choice.

The zero list is the second edge case. The ratio is 0/0 and the angle is undefined, not zero. Software that returns 0 for that case reports a number the mathematics does not supply.

What L1 hid. L1 presented the range −1 to 1 as a fact about cosines. It is a theorem about inner products, and it holds for every one of them, which is why the range survives any reweighting. What does not survive is the value. The simplification was safe in this section because both worked examples were nonzero and the standard inner product was the only metric in play.

Where the analogy breaks. "Agreement between baskets" treats every entry as an equal vote and assumes one step in every column means the same thing. Absence is not neutral: a basket with three apples and no loaves scores zero against one with no apples and three loaves, not one half. Negative entries make agreement negative, which the basket story does not prepare you for.

Original sources. Hefferon defines the dot product and derives the angle formula from the law of cosines [S:23:53]. Axler proves Cauchy–Schwarz from an orthogonal decomposition and states the equality case [S:24:203]. Together they give the section its two halves: the definition on one page, the inequality that licenses it on another. The equality case is the one to keep, because it says that a cosine of exactly 1 or −1 means the two lists are multiples of one another.

An open question. Cosine similarity is the standard score for comparing word embeddings [S:96:357], but the field has not settled what a given value means. Two effects pull in opposite directions. Independent random vectors with mean-zero, equal-variance entries in dimension $d$ have a cosine whose size is about $1/\sqrt{d}$ with high probability, so independently drawn items score near zero [S:20:77][S:69:942]. The zero-mean condition is essential: vectors that share a mean direction score near 1, and trained language models do exactly that, their embeddings for unrelated words reaching cosines close to 1, a property called anisotropy [S:98:218]. A threshold tuned on one model therefore means something different on another. Evidence that would settle a given case: a calibration curve from cosine to human similarity, measured on held-out pairs for each model, plus a stated transformation that removes the anisotropy before the threshold is chosen.

A frontier method. Retrieval over word embeddings ranks items by cosine and returns the closest few [S:96:357]. The method depends on this section twice. The score is the cosine, and the ranking is interpretable partly because the score lies in a fixed interval, so a value can be compared with values from other pairs.

What to read next. Read Hefferon section II.2 [S:23:52] for the length and angle construction, Axler section 6A [S:24:202] for the proof and the equality case, and Vershynin [S:20:77] for the high-dimensional geometry that makes small cosines ordinary. Then read Eisenstein section 14.6 [S:96:357] for how the score is evaluated, and the anisotropy discussion in [S:98:218] for why an embedding space is not a sphere of random vectors.