ablation
Removing one component of a system to measure what that component contributes.
Example: Retraining without dropout to see whether dropout helped.
Sections:
Appendix C on the web
Glossary
552 terms, A to Z. Search runs in this browser, and every entry links back to the sections where the term is recorded.
What is real here. The entries come from registry/glossary.json, copied into src/data/glossary.json at export time. The section lists come from the loaded chapters; in this fixture build only chapter 1 contributes occurrences.
Search runs in this browser. No query is sent anywhere.
552 entries shown.
A
abstraction barrier
The boundary that separates the users of an abstraction from its implementation.
Example: Code outside a queue never reads its internal array.
Sections:
accumulation
Adding the contributions that arrive at a node from more than one reader.
Sections:
action
One of the choices available to the agent at a step.
Sections:
action-value function
A value function with one entry per state and action.
activation function
A nonlinear function applied to a unit's weighted sum.
Example: ReLU, which returns the input if positive and zero otherwise.
actor-critic
A policy-gradient method that learns a policy and a value function together, using the value function to judge actions.
Sections:
Adam
An optimiser that adapts a separate step size for each parameter using running averages of gradients and squared gradients.
Example: AdamW is the usual default for transformer training.
Sections:
adjoint
The derivative of the seeded output with respect to a node, accumulated from every path that reaches the node.
Sections:
agent
The learner and decision maker that chooses actions.
Sections:
algorithm
A finite, precise procedure for solving a class of problems.
Example: Binary search on a sorted list.
alignment
The problem of making a system pursue the outcomes its designers intended, including under distribution shift and scale.
Example: A model that follows a specification but violates its intent.
Sections:
ancestral sampling
Drawing each variable in turn from its conditional distribution given the variables already drawn.
arithmetic intensity
The number of operations a kernel performs divided by the number of bytes it moves.
Sections:
array
A block of slots in a program, reached by a whole-number index.
Sections:
assumption
A statement a model treats as true without testing it in the fit.
Sections:
asymptotic normality
The tendency of an estimator's sampling distribution to become Gaussian as the sample size grows.
Sections:
attention
A weighted average over a set of items, where the weights depend on how relevant each item is to the current query.
Example: A decoder focusing on the words of a source sentence.
Sections:
attention head
One set of query, key and value maps together with the attention they perform.
Sections:
attention weight
A non-negative number that says what fraction of one value enters an output, with the weights for one output summing to one.
Sections:
automatic differentiation
Computing exact derivatives of a program by applying differentiation rules to each operation.
Example: Reverse-mode autodiff on a neural network's forward pass.
Sections:
average length
The expected number of bits per symbol, weighted by symbol probabilities.
Sections:
B
back substitution
Solving a triangular table from the bottom row upward, one unknown at a time.
Sections:
backpropagation
Computing the gradient of a scalar output with respect to every parameter by applying the chain rule backwards through the computation graph.
Example: The gradient of the loss with respect to an early layer's weights.
backpropagation through time
Backpropagation applied to an unfolded recurrent chain, with one gradient step per position.
Sections:
backward pass
The run through a computation graph that carries the derivative back from the loss to each parameter.
bandit problem
A problem with one state and several actions, each paying a random reward.
Sections:
base case
The smallest instance of a recursive task, the one the rule answers without calling itself.
base rate
The prevalence of a condition in a population before any test result is seen.
Sections:
baseline
The comparison method that a new method has to beat on the same data and the same budget.
basis
A set of vectors that spans a space and has no redundancy.
Example: (1,0) and (0,1) form a basis of the plane.
batch
The small group of examples processed together in one training step.
batch normalisation
Normalising a layer's activations across the batch, then applying a learned scale and shift.
Example: Subtracting the batch mean and dividing by the batch standard deviation.
batch size
The number of examples averaged into one stochastic gradient.
Sections:
Bayes' rule
A formula that turns a conditional probability around using the prior and the evidence.
Formal definition: P(A|B) = P(B|A) P(A) / P(B)
Example: Updating a disease belief after a test result.
Sections:
Bayesian
A reading of probability as the strength of a belief in a proposition.
Sections:
Bellman equation
A recursive equation expressing a value in terms of the immediate reward and the next state's value.
Formal definition: V(s) = E[r + gamma V(s')]
Example: The value of a state equals the reward now plus the discounted value of where you land.
benchmark
A fixed test set with recorded scores, used to compare methods.
Sections:
bias
The expected difference between an estimator and the quantity it estimates.
Example: A scale that always reads one kilogram too high.
bias correction
A division that removes the downward bias of an average that started at zero.
Sections:
big-O
An upper bound on growth, up to a constant factor, for large inputs.
Example: 3n^2 + 5n is O(n^2).
Sections:
bijection
A function that is both injective and surjective, pairing the two sets off exactly.
Sections:
binary symmetric channel
A channel that flips each bit independently with the same probability.
Sections:
bit
The unit of information, equal to the surprise of one fair coin flip.
Sections:
block
One piece of the chain whose derivative is available as a single local rule.
Sections:
block code
A code that encodes a fixed number of symbols at once.
Sections:
bootstrap
Resampling the observed data with replacement to estimate the variability of a statistic.
Example: A thousand resampled means give a standard error.
Sections:
C
cache
A small, fast memory that holds recently used data close to the processor.
Example: A 32 KB level-one cache holding the current array block.
Sections: no section recorded yet
calibration
The match between announced probabilities and observed frequencies over many cases.
Sections:
cancellation
Losing precision when subtracting two nearly equal numbers.
Example: 1.0000001 minus 1.0000000 leaves only the uncertain digits.
Sections: no section recorded yet
cancellation ratio
The largest operand in a computation divided by the size of its answer.
Sections:
cardinality
The number of elements in a set, or the size of an infinite set under a matching rule.
Example: The set of digits has cardinality ten.
Sections: no section recorded yet
catastrophic cancellation
Subtracting two nearly equal numbers that already carry error, so that the earlier error arrives in the answer at full size.
Sections:
Cauchy–Schwarz inequality
The statement that the size of the dot product of two lists never exceeds the product of their norms.
Sections:
central difference
A finite difference that steps equally to both sides of the point and divides by the full width.
Sections:
central limit theorem
The sum of many independent, similarly sized random variables is approximately Gaussian.
Example: The distribution of sample means becomes bell-shaped.
Sections:
chain rule
The rule for differentiating a composition: multiply the derivatives of the parts.
Formal definition: (f(g(x)))' = f'(g(x)) g'(x)
Example: Backpropagation applies the chain rule layer by layer.
chain rule for entropy
The rule that the entropy of a pair equals the entropy of one variable plus the conditional entropy of the other.
channel capacity
The maximum rate at which information can be sent reliably through a noisy channel.
Example: A binary symmetric channel with error rate 0.1 has capacity below one bit per use.
Sections:
characteristic polynomial
The polynomial in lambda whose roots are the eigenvalues of a matrix.
Sections:
charge
One point assessed for a row on which a candidate rule and the recorded answer disagree, and zero where they agree.
Sections:
checkpoint
One intermediate value deliberately kept so the backward pass can rebuild the rest from it.
Sections:
checkpointing
Keeping a few intermediate values and recomputing the rest during the backward pass to save memory.
Sections:
Cholesky factorisation
Writing a positive-definite matrix as a lower-triangular matrix times its transpose.
Formal definition: A = L L^T
Example: The matrix [[4,2],[2,5]] factors with L = [[2,0],[1,2]].
Sections: no section recorded yet
classifier-free guidance
Guidance built by mixing a conditional and an unconditional estimate from one network, with no separate classifier.
closure
The property that adding two members or scaling one keeps the result inside the set.
code rate
The number of message bits carried per use of the channel.
Sections:
coefficient
One of the numbers a model adjusts to fit the data, such as a slope or an intercept.
collinearity
Two or more inputs carrying nearly the same information, so the fit cannot separate their effects.
Sections:
column space
The set of all combinations of the columns of a table, which is the set of outputs it can produce.
Sections:
comparison tree
A binary tree whose nodes are the comparisons a sorting procedure makes and whose leaves are the orders it can output.
Sections:
composition
The procedure obtained by applying one procedure to the result of another.
computation graph
A diagram whose nodes are operations and whose edges carry the values between them.
compute-bound
Limited by arithmetic, because the kernel needs more operations per byte than the machine's ridge.
Sections:
concentration inequality
A bound on the probability of a large deviation, computed from a few summary facts about a random variable.
Sections:
condition number
The largest curvature of the loss divided by the smallest; a measure of how stretched the bowl is.
conditional entropy
The average uncertainty that remains about one variable after the other is known.
conditional probability
The probability of one event given that another event occurred.
Formal definition: P(A|B) = P(A and B) / P(B)
Example: The chance of a disease given a positive test.
confidence interval
A procedure that produces an interval which contains the true value in a stated long-run fraction of samples.
Example: A 95 percent interval computed by the same rule in repeated samples.
consistent
An estimator is consistent when its estimates converge to the true value as the sample size grows.
Sections:
constructor
A procedure that builds an object of a given kind out of other objects.
contamination
Evaluation rows, or text very close to them, appearing in the training data.
Sections:
continuing task
A task whose interaction goes on without a natural end.
Sections:
continuity
A function is continuous at a point when small changes in the input cause small changes in the output.
Example: A curve you can draw without lifting the pen.
Sections: no section recorded yet
continuous
A function is continuous at a point when its limit there equals its value there, so small input changes give small output changes.
continuous random variable
A random variable whose possible values you obtain by measuring, so between any two values sit more values.
Sections:
contour line
A line that collects the points where a function has one fixed value.
Sections:
contract
The promise a program makes to its caller: what it accepts, what it returns, and what it guarantees.
Example: Precondition: a positive number. Postcondition: a non-negative result whose square is within tolerance.
contradiction
A proof that assumes the claim is false and derives an impossibility.
Example: Assuming the square root of 2 is a fraction in lowest terms forces both numerator and denominator to be even.
Sections:
contrapositive
The contrapositive of if P then Q is if not Q then not P, which is true in exactly the same cases.
convention
A rule adopted so that a program returns some value where the mathematics leaves a choice.
Sections:
converge
A sequence converges to a value when, after some step, every later term stays arbitrarily close to it.
Sections:
convergence rate
The factor by which the error shrinks at each step of an iterative method.
Sections:
converse
The converse of if P then Q is if Q then P, which is a different claim.
convex function
A function whose graph lies below the straight line between any two of its points.
Example: The bowl f(x) = x^2.
Sections:
convolution
A sliding weighted sum that shares the same weights at every position.
Example: A 3-by-3 edge detector applied across an image.
correlation
A covariance rescaled by the two standard deviations, which removes the units and always lands between minus one and one.
Sections:
cosine
The dot product of two lists divided by the product of their norms, which measures the angle between them.
Sections:
countable
A set whose members can be listed with every member at some finite position.
Sections:
counterexample
One case that makes a universal claim false.
Example: The number 2 is a counterexample to the claim that every prime is odd.
Sections:
counting sort
A sorting procedure that counts how many times each value occurs and writes the values out in order of their counts, without comparing two keys.
Sections:
covariance
A measure of how two random variables move together.
Formal definition: Cov(X,Y) = E[(X - E[X])(Y - E[Y])]
Example: Height and weight tend to have positive covariance.
Sections:
covariance matrix
A square table whose diagonal holds each measurement's variance and whose off-diagonal entries hold the covariances between pairs.
Sections:
coverage
The share of the possible data examples that a model can produce at all.
Sections:
Cramer-Rao bound
A lower bound on the variance of any unbiased estimator.
Example: No unbiased estimator beats the inverse Fisher information.
Sections: no section recorded yet
Cramér–Rao bound
A lower bound on the variance of any unbiased estimator, equal to the inverse of the total Fisher information.
credit assignment
The problem of deciding which earlier action deserves credit for a later reward.
Sections:
cross-attention
Attention in which the queries come from one sequence and the keys and values from another.
Sections:
cross-entropy
The average surprise of a model's predictions, measured in bits or nats per token.
Sections:
cross-validation
Splitting the data into folds, training on some and testing on the rest, and rotating which fold is held out.
Example: Five-fold cross-validation reports the average error over five held-out folds.
curvature
The rate at which the slope itself changes, measured by the second derivative.
D
data abstraction
Building a compound value behind a small set of operations, so users never depend on how it is stored.
Example: A queue with put and get, regardless of whether it uses an array or a linked list.
Sections: no section recorded yet
data structure
A way of holding a collection of values in a program, together with the operations that read and change it.
Sections:
dataset
A collection of examples used for learning or evaluation.
Example: Sixty thousand labelled digit images.
Sections:
decoder
A network that maps a hidden code to a distribution over examples.
Sections:
degradation problem
The finding that a deeper plain network can end with a higher training error than a shallower one.
Sections:
degree of belief
A number between zero and one that says how strongly you hold an uncertain proposition.
Sections:
delegate
Whoever receives a smaller instance of the same problem together with the same promise, and whose workings the caller does not inspect.
Sections:
denoising
Removing an estimate of the noise from a corrupted example.
Sections:
derivative
The instantaneous rate of change of a function, or the best linear approximation near a point.
Formal definition: f'(x) = lim_{h to 0} (f(x+h) - f(x)) / h
Example: The speed shown on a car's speedometer.
descent direction
A direction whose inner product with the gradient is negative, so a small enough step along it lowers the loss.
Sections:
descent lemma
The inequality that puts a parabola above the loss along any line and so guarantees a decrease for a step of 1/L.
determinant
A number that records how a matrix scales volume and whether it is invertible.
Example: A 2-by-2 determinant is the signed area scale factor.
Sections:
diagonalisation
A construction that builds an object outside any proposed complete list.
Example: Cantor's argument that the real numbers cannot be listed.
dictionary
A structure that maps a key to a value, so a stored answer can be found from the input it belongs to.
Sections:
difference quotient
The ratio of the change in a function's value to the change in its input over an interval.
differentiable
A function is differentiable at a point when its derivative exists there.
diffusion model
A generative model that learns to reverse a gradual noising process.
Example: Starting from noise and denoising step by step into an image.
Sections:
dimension
The number of vectors in any basis of a vector space.
Example: The plane has dimension two.
Sections:
direct proof
A proof that begins at the assumptions and derives the claim by steps running forward.
direction
A chosen list of rates, one per input, that says which way the inputs move.
Sections:
discount rate
A number between zero and one that reduces the value of each later reward.
Sections:
discrete random variable
A random variable whose possible values you obtain by counting, so they can be listed one by one.
Sections:
distance metric
A rule that assigns a non-negative distance to each pair of points, with symmetry and the triangle inequality.
Example: Euclidean distance between two feature vectors.
Sections:
divergence
The failure in which the loss grows without bound or becomes infinite or NaN.
divide and conquer
Solving a problem by splitting it, solving the parts, and combining the answers.
Example: Merge sort splits the list in half, sorts each half, then merges.
Sections:
dominant eigenvalue
The eigenvalue with the largest absolute value, which controls the long-run growth of repeated application.
dot product
The sum of the products of matching entries of two lists of the same length.
dropout
A training trick that removes each unit at random and doubles the units it keeps, so the expected signal is unchanged.
dynamic programming
Solving a problem by storing the answers to overlapping subproblems.
Example: Computing edit distance row by row in a table.
dynamical system
A rule that turns a state into the next state, applied over and over.
Sections:
E
eigenbasis
A basis made entirely of eigenvectors of one matrix.
eigenspace
All eigenvectors for one eigenvalue, together with the zero vector, which form a subspace.
Sections:
eigenvalue
A number by which a matrix scales one of its special directions.
Formal definition: A v = lambda v for a nonzero vector v
Example: The matrix [[2,0],[0,3]] has eigenvalues 2 and 3.
eigenvector
A nonzero vector that a matrix only scales, without rotating.
Example: The x-axis for a diagonal matrix.
Sections:
embedding
A learned vector that represents a discrete object such as a word or a token.
Example: A 768-dimensional vector for the token cat.
Sections:
empirical risk
The average loss of a predictor over the examples you hold.
Formal definition: Rhat(h) = (1/n) sum_i ell(h(x_i), y_i)
Example: The average squared error over the training set.
empirical risk minimisation
Choosing the candidate in a class whose average loss on the training examples is smallest.
Sections:
encoder
A network that maps an example to a distribution over hidden codes.
Sections:
entropy
The average surprise of a random variable, or the shortest average code length for its outcomes.
Formal definition: H(X) = -sum_x p(x) log p(x)
Example: A fair coin has one bit of entropy.
Sections:
environment
Everything outside the agent that the agent interacts with.
Sections:
episode
One run of agent-environment interaction that ends in a terminal state.
Sections:
episodic task
A task made of episodes that each end and then restart.
Sections:
epsilon-greedy
A rule that explores with probability epsilon and otherwise takes the best action so far.
Sections:
equivalence relation
A relation that is reflexive, symmetric and transitive, which sorts a set into classes whose members are linked to one another.
Sections:
error
The fraction or count of predictions that are wrong on a set of examples.
Example: Three wrong answers out of fifty give an error of 0.06.
Sections:
error curve
The graph of selection-set error against a tuning parameter such as k.
Sections:
error term
The part of a recorded answer that the systematic rule does not explain.
Sections:
estimate
The number an estimator returns on one particular dataset.
Sections:
estimator
A rule that computes a guess about a population quantity from a sample.
Example: The sample mean as an estimate of the population mean.
Sections:
even
A whole number is even when it equals twice some whole number.
event
A subset of the sample space.
Example: At least one head in two flips.
Sections:
evidence lower bound
A tractable lower bound on the log likelihood of a latent-variable model.
Formal definition: log p(x) >= E_q[log p(x,z) - log q(z)]
Example: The training objective of a variational autoencoder.
Sections:
example
One item in a dataset, usually a set of features together with a label.
Example: One 28-by-28 image of a handwritten digit.
Sections:
exclusive or
The function of two yes-or-no inputs that answers yes when the two inputs differ and no when they agree.
Sections:
expectation
The probability-weighted average of a random variable.
Formal definition: E[X] = sum_x x P(X = x)
Example: The expected number of heads in ten fair flips is five.
Sections:
experiment
A procedure that produces a result and can be repeated under recorded conditions.
Example: Training the same model with five different random seeds.
Sections:
exploding gradient
The condition in which the gradient grows without bound and the updates destroy the weights.
exploitation
Choosing the action that has paid best so far.
Sections:
exploration
Trying actions in order to learn how much reward they produce.
Sections:
exponent field
The eight bits that store the scale of the number, read as an unsigned integer with an offset.
Sections:
expression swell
The growth of a derivative expression far beyond the size of the formula it came from.
Sections:
F
factorial
The product of all whole numbers from one up to a given number.
faithful
Of an explanation, correctly predicting what the model does when an input is changed.
Sections:
fan-in
The number of inputs that enter a unit.
Sections:
fan-out
The number of units that a layer feeds.
feature
A measurable property of an example, usually one coordinate of its vector.
Example: The count of dark pixels in a region of an image.
feature attribution
A method that assigns part of a prediction to each part of the input.
Sections:
feature map
A grid of outputs holding one detector's response at every position.
Sections:
feature vector
The ordered list of measurements for one item, one slot per feature.
feedforward layer
Two linear maps with a nonlinearity between them, applied to every position separately.
Sections:
finite difference
An estimate of a derivative obtained by dividing a change in the function's value by a small but nonzero change in its input.
Sections:
first-class
A status held by elements that may be named by a variable, passed as an argument, returned as a result, and stored inside a data structure.
Sections:
Fisher information
How sharply the likelihood changes with the parameter, and therefore how much one observation can tell you.
Example: A narrow likelihood gives more information than a flat one.
fitting
Whatever a method does with the stored rows before it answers a query.
Sections:
fixed point
A value that a rule maps to itself, so applying the rule again leaves it unchanged.
Sections:
floating point
A computer representation of a real number as a sign, an exponent and a fraction.
Example: The binary64 value 0.1 is not exactly one tenth.
flow matching
A generative method that learns a velocity field transporting a simple distribution to the data distribution.
Example: Learning straight paths from noise to data.
Sections: no section recorded yet
forward difference
A finite difference that steps only to the right of the point.
Sections:
forward mode
Automatic differentiation that carries one input direction forward beside the program as it runs.
Sections:
forward pass
The run through a computation graph that produces the output and stores the intermediate values.
forward process
The fixed sequence of small random noising steps that turns a picture into noise.
Sections:
fraction
The twenty-three bits that store the digits after the leading one.
Sections:
Fréchet inception distance
A distance between the feature distributions of real and generated images, each approximated by a Gaussian.
Sections:
free name
A name a procedure uses but does not create, whose value the procedure keeps from where it was written.
Sections:
frequentist
A reading of probability as the long-run fraction of trials in which an event occurs.
Sections:
function
A rule that assigns exactly one output to each allowed input.
Example: The rule that sends each integer to its square.
Sections:
fundamental subspaces
The column space, row space, kernel and left null space of a table, taken together.
Sections:
G
Gaussian distribution
The bell-shaped distribution defined by a mean and a variance.
Formal definition: p(x) = (1 / sqrt(2 pi sigma^2)) exp(-(x - mu)^2 / (2 sigma^2))
Example: Heights in a population.
Sections: no section recorded yet
Gaussian elimination
A systematic row operation procedure that turns a linear system into an equivalent triangular one.
Example: Using row swaps and subtractions to solve three equations in three unknowns.
Sections:
generalisation
Performance on examples the model did not see during training.
Example: Accuracy on the held-out test set.
generalised linear model
A model with a linear score and a link function that maps it to the mean of a distribution.
Example: Logistic regression and Poisson regression are both GLMs.
Sections:
generative model
A model that represents a probability distribution over data and can sample from it.
Example: A model that produces new images resembling a training set.
Sections:
geometric series
A sum in which each term is a fixed multiple of the one before it.
gradient
The vector of partial derivatives, pointing in the direction of steepest increase.
Example: For f(x,y) = x^2 + y^2, the gradient at (1,2) is (2,4).
gradient descent
Repeatedly stepping in the direction opposite the gradient to reduce a function.
Formal definition: x_{t+1} = x_t - eta grad f(x_t)
Example: Walking downhill on a bowl-shaped loss surface.
Sections:
gradient norm
The length of the gradient vector, which measures how large the update signal is.
Sections:
greedy policy
A policy that always takes the action with the best estimate so far.
guidance
Combining two model estimates at sampling time so the draw leans toward a condition.
H
half precision
The 16-bit floating-point format with five exponent bits and ten fraction bits.
Sections:
harness
One program that runs every step of an experiment from a written configuration and records one row per run.
Sections:
He initialisation
A rule that sets the weight variance to 2 divided by the fan-in, compensating for the units a rectified linear unit switches off.
Sections:
Hessian
The matrix of second partial derivatives, describing local curvature.
Example: For a quadratic bowl, the Hessian is constant and positive definite.
Sections: no section recorded yet
higher-order function
A function that takes another function as an argument or returns one as its result.
Example: A sorting function that accepts the comparison it should use.
Sections:
history
Everything the agent has observed and done so far.
Huffman code
A prefix code built by repeatedly merging the two least frequent symbols.
Example: Assigning short codes to common letters.
Sections:
hypothesis
A condition that must hold for a claim or a rule to apply.
Sections:
hypothesis class
The set of candidate predictors a learning procedure is allowed to choose from.
Sections:
hypothesis test
A rule that decides between a null hypothesis and an alternative using data and a chosen error rate.
Example: Rejecting a null hypothesis when the p-value is below 0.05.
Sections: no section recorded yet
I
identity mapping
A skip that adds the input to the output without changing it.
Sections:
image
The set of outputs a function can produce.
Example: The image of a 2-by-2 matrix is its column space.
Sections:
Inception score
An image metric that rewards confident class predictions and variety across classes.
Sections:
independence
Two events are independent when one occurring does not change the probability of the other.
Formal definition: P(A and B) = P(A) P(B)
Example: Two separate coin flips.
index
One of the whole numbers used to label the cases of an indexed claim.
Sections:
induction
A proof that a statement holds for a first case and that each case implies the next.
Example: Proving a sum formula for n = 1 and then from n to n + 1.
inductive bias
An assumption about the data that is built into a model's structure rather than learned.
inductive step
The part of an induction that assumes one case and derives the next.
Sections:
infinite set
A set that no finite list of its members contains entirely.
Sections:
influential point
An example whose input lies far from the rest, so that removing it moves the fitted line a long way.
Sections:
information content
The surprise of one outcome, measured in bits as the base-two logarithm of one over its probability.
Sections:
initialisation
The rule that sets the starting values of a model's parameters before training.
Example: Drawing weights from a Gaussian scaled by the layer width.
Sections:
injective
A function is injective when no output receives two arrows.
Sections:
inner alignment
The gap between the written objective and the objective a model actually learned.
Sections:
intercept
The output a fitted line predicts when the input is zero.
Sections:
interface
A set of operations together with the meaning of each: what it may be handed and what it returns.
Sections:
internal covariate shift
The idea that each layer keeps facing a moving distribution of inputs as the layers before it change.
interpretability
The study of explanations for a model's behaviour that a person can check and use.
Example: Attributing a prediction to the input features that most affected it.
Sections:
interpreter
The program that carries out a program's text, and which maintains the deferred chain of a recursive process.
Sections:
invariant
A statement that is true before and after every step of a loop or process.
Example: In a loop that adds one item per step, the running total equals the sum of the items seen so far.
Sections: no section recorded yet
iteration
Repeating a block of steps until a condition holds, without the procedure calling itself.
Example: A while loop that multiplies a running product by each integer from 1 to n.
Sections: no section recorded yet
iterative process
A process whose state a fixed number of variables describes completely, together with a fixed rule for updating them and an optional end test.
Sections:
J
Jacobian
The matrix of all first partial derivatives of a vector-valued function.
Example: A 2-by-2 matrix for a map from the plane to the plane.
Sections: no section recorded yet
Jacobian-vector product
The derivative of a function applied to one direction, which is one column of the Jacobian.
Sections:
joint entropy
The entropy of two variables taken together.
K
k-nearest neighbours
A predictor that takes a vote or average over the k closest stored examples.
Formal definition: hat y(x) = majority vote of the labels of the k nearest stored examples to x
Example: With k = 3, two neighbours say cat and one says dog, so the prediction is cat.
Sections:
kernel
The set of vectors a linear map sends to zero.
Example: For projection onto the x-axis, the kernel is the y-axis.
key
A learned vector that says what an item advertises to a query.
Sections:
kink
A point where the slope from the left and the slope from the right disagree.
Sections:
KL divergence
A non-negative measure of how much one probability distribution differs from another.
Formal definition: D_KL(P||Q) = sum_x p(x) log(p(x)/q(x))
Example: The extra bits needed to encode P's outcomes using Q's code.
L
label
The answer attached to an example in a supervised learning problem.
Example: The digit 7 written beside an image.
Sections:
language model
A probability distribution over sequences of tokens.
Example: A model that assigns a probability to the next word.
Sections:
lasso
Least squares with a penalty on the absolute size of the coefficients, which can set some to zero.
Example: Selecting three features out of fifty.
Sections:
latency
The time one request takes to return.
Sections:
latent variable
A hidden variable that the model does not observe but uses to explain the data.
Example: A low-dimensional code from which an image is generated.
law of large numbers
The sample average of independent draws approaches the expectation as the sample grows.
Example: A thousand coin flips give a proportion close to one half.
Sections:
layer normalisation
Normalising each example's activations across the feature dimension.
Example: The standard normalisation inside a transformer block.
leakage
Information from the test set or the future reaching the model during training.
Example: Scaling the features using the mean of the whole dataset before splitting.
learner
A procedure that reads labelled examples and returns a predictor.
Sections:
learning rate
The size of the step taken in each gradient descent update.
Example: A step of 0.01 versus a step of 10.
learning-rate schedule
A rule that sets the step size at each iteration instead of holding it fixed.
Sections:
least squares
The rule that picks the line whose squared vertical misses add up to the smallest total.
Sections:
leave-one-out
Cross-validation with one row per fold, so each model is fitted on all rows but one.
left null space
The set of output lists that pair with every column to give zero.
Sections:
likelihood
The belief in the evidence given that a particular hypothesis is true.
limit
The value a sequence or function approaches as its input approaches a point.
Example: The sequence 1, 1/2, 1/3, ... approaches zero.
Sections:
linear approximation
The straight-line formula that stands in for a function near a point, built from the value and the derivative there.
linear combination
A sum of scaled copies of a list of vectors.
Sections:
linear independence
The property of a list in which no member can be built from the others.
Sections:
linear map
A function between vector spaces that preserves addition and scalar multiplication.
Formal definition: L(ax + by) = a L(x) + b L(y)
Example: Multiplying every vector by a fixed matrix.
Sections:
linear regression
Fitting a straight-line rule to data by least squares, with one weight per input.
linearithmic
Growing like n times the logarithm of n, the cost of the standard sorting algorithms.
Sections:
linearly separable
Two classes are linearly separable when one straight line puts every item of one class on one side and every item of the other class on the other side.
Sections:
link function
The transformation applied to the mean of the answer so that the transformed mean can be a straight line in the inputs.
Sections:
local fidelity
The accuracy of an explanation near the single input it explains, rather than everywhere.
Sections:
local noise
The chance that a stored row near the query carries an atypical or wrong label.
Sections:
local receptive field
The small patch of input that one output unit can see.
Sections:
locality
Using data that sits close together in memory, so caches work.
Example: Walking an array in order instead of jumping across it.
Sections: no section recorded yet
log-likelihood
The logarithm of the likelihood, which turns a product over observations into a sum.
Sections:
log-odds
The logarithm of the odds, a scale on which a logistic model is a straight line.
log-sum-exp
A numerically stable way to compute the logarithm of a sum of exponentials.
Formal definition: log sum_i exp(z_i) = m + log sum_i exp(z_i - m), with m = max z_i
Example: Subtract the largest score before exponentiating.
Sections: no section recorded yet
logarithmic
Growing like the logarithm of the input, which for base two is the number of halvings needed to bring the input down to one.
Sections:
logistic function
The S-shaped curve that turns any score into a number between zero and one.
Sections:
logistic regression
A linear score passed through the logistic function to produce a probability.
Formal definition: p(y=1|x) = 1 / (1 + exp(-w^T x))
Example: Estimating the chance of a click from two features.
Sections:
logit
A raw score that a model produces before any probability is formed.
Sections:
look-ahead
Using future rows to fit a model that will be judged on past rows.
Sections:
loss
A number that says how bad one prediction is.
Formal definition: ell(y, yhat) is the cost of predicting yhat when the answer is y
Example: Squared error (y - yhat)^2.
loss curve
A plot of the loss against the training step or epoch.
Sections:
loss scaling
Multiplying the loss by a power of two before the backward pass and dividing the gradients back before the update.
Sections:
loss spike
A single narrow jump in the loss that is followed by a return to the previous level.
Sections:
lower bound
A statement that no procedure of a stated kind can use fewer than a stated amount of work.
Sections:
M
magnitude
The size of a set, meaning for a finite set the number a count reports.
Sections:
manifold
The region of data space where real examples lie.
Sections:
marginal entropy
The entropy of one variable on its own, as opposed to a joint or conditional entropy.
Markov decision process
A model of an agent choosing actions in a world with states, transitions and rewards.
Example: A grid world where each move gives a reward.
Markov property
The next state and reward depend only on the current state and action, not on the earlier history.
mask
A rule that sets chosen scores to negative infinity so those positions receive no weight.
master copy
A higher-precision copy of the weights to which the update is applied.
Sections:
matrix
A rectangular array of numbers that represents a linear map between vectors.
Example: A 2-by-2 array that rotates the plane.
maximum likelihood
Choosing the parameter that makes the observed data most probable.
Example: Fitting a Gaussian by using the sample mean and variance.
mean
The ordinary average of a set of numbers.
mean squared error
The expected squared difference between an estimate and the true value.
Formal definition: MSE = bias^2 + variance
Example: The average squared error of a fitted curve.
Sections:
measure
A whole number attached to an instance that strictly decreases at every hand-over and never falls below zero, which is what guarantees that the hand-overs stop.
Sections:
memoisation
Caching the result of a function so repeated calls with the same input are free.
Example: A dictionary from input to computed Fibonacci number.
Sections:
memory-bound
Limited by memory, because the kernel needs fewer operations per byte than the machine's ridge.
Sections:
minibatch
A small random draw of examples whose average gradient is used for one step.
Sections:
mixed precision
Using lower-precision numbers for most operations while keeping a high-precision copy of sensitive values.
Example: Training in 16-bit arithmetic with a 32-bit master copy of the weights.
Sections:
model
A rule that says which next state and reward follow each state and action.
model criticism
Examining where a fitted model fails, on which rows, and whether the failures share a cause.
Sections:
momentum
An update rule that accumulates a velocity in directions that keep pointing the same way.
Example: A ball rolling downhill speeds up in a consistent slope.
Sections:
monochromatic
One colour throughout: every member of the collection has the same colour.
Sections:
multi-head attention
Several attention heads run in parallel and concatenated, so different relations can be read at once.
Sections:
multilayer perceptron
A feedforward neural network with one or more hidden layers of units.
Example: An input layer, one hidden layer of 64 units, and an output layer.
Sections: no section recorded yet
multivariate Gaussian
The bell curve for several measurements at once, fixed by a mean for each measurement and a covariance matrix.
Sections:
mutual information
How much knowing one variable reduces uncertainty about another.
Formal definition: I(X;Y) = H(X) - H(X|Y)
Example: Knowing a word reduces uncertainty about the next word.
Sections:
N
nearest neighbour
The stored example whose features are closest to the input under a chosen distance.
Example: The training image with the smallest Euclidean distance to a new image.
Sections:
negation
The statement that is true exactly when the original statement is false.
Sections:
neural network
A stack of simple units, each applying a weighted sum and a nonlinearity, composed into a function.
Example: A two-layer network that separates a curved boundary.
Sections: no section recorded yet
next-token prediction
The task of choosing a probability for the token that follows the tokens already read.
Sections:
no-free-lunch theorem
The theorem that for every learner there exists a task on which it fails, even though another learner succeeds on that task.
Sections:
noise floor
The gap at which the noise added per step balances the decrease, so a constant step stops improving.
Sections:
noise schedule
The list of noise amounts used at the successive steps of the forward process.
Sections:
noisy-channel coding theorem
The result that every rate below capacity is achievable with long blocks and a good code, and no rate above it is.
Sections:
norm
The length of a list: the square root of the sum of its squared entries.
Sections:
normal equations
The square system whose solution gives the least-squares coefficients, with one equation per column of the data table.
Sections:
normal number
A number whose significand keeps its hidden leading one.
Sections:
NP-hard
A problem at least as hard as every problem in NP, so a fast general algorithm is not known.
Example: Finding the shortest tour that visits every city.
null hypothesis
The boring explanation a test assumes, usually that there is no effect.
Sections:
numerical differentiation
Estimating a derivative from values of a function at nearby points.
Sections:
O
object
A value a program can hold and pass around without caring what it is made of.
Sections:
observation
What the agent senses at one step.
occupancy
The number of warps resident on a processor divided by the number it could hold.
Sections:
odd
A whole number is odd when it equals twice some whole number plus one.
odds
The chance of an event divided by the chance against it, written p/(1-p).
Sections:
optimal policy
A policy that does at least as well as every other policy in every state.
ordered pair
Two objects in a fixed order, written (a, b), so that (2, 3) and (3, 2) are different pairs.
Sections:
orthogonal
Having dot product zero, which for arrows means meeting at a right angle.
Sections:
orthogonal matrix
A square matrix whose columns are orthonormal, so it rotates or reflects without changing lengths.
Sections:
orthonormal basis
A basis whose vectors all have length one and are perpendicular to each other.
outcome
One member of the sample space: a single possible result.
Sections:
outer alignment
The gap between the objective written down and the outcome the designer actually wants.
Sections:
overdetermined system
A set of equations with more equations than unknowns, so that no exact solution usually exists.
Sections:
overfitting
Fitting the training examples so closely that performance on new examples gets worse.
Example: A curve that passes through every training point and predicts nonsense between them.
P
p-value
The probability, assuming the null hypothesis, of a result at least as extreme as the one observed.
Example: A p-value of 0.03 means three percent of null worlds would look this extreme.
Sections:
parameter
A fixed but unknown number that indexes a family of probability distributions.
partial correctness
The property that a procedure's final results, when it produces any, satisfy what was required of them.
Sections:
partial derivative
The rate of change of a function in one coordinate while every other coordinate is held fixed.
patch
The region around the query that holds the k nearest stored rows.
Sections:
path length
The number of steps information travels between two positions in a network.
penalty
A term added to the fitting total that grows with the size of the coefficients, so the fit pays for complexity.
perceptron
A single unit that computes a weighted sum and fires when it passes a threshold.
Example: A linear classifier that separates two clouds of points.
Sections:
perplexity
The effective number of equally likely tokens among which a model is choosing, equal to two raised to the cross-entropy in bits.
Sections:
piecewise linear
Built from straight segments, so its slope is constant between corners and jumps at them.
Sections:
pivot
The nonzero entry in a column that is used to clear the entries below it.
Sections:
plausible
Of an explanation, accepted by a person as a convincing account, whether or not it is faithful.
Sections:
policy
A rule that chooses an action, or a probability distribution over actions, in each state.
Example: Always move right unless blocked.
Sections:
policy evaluation
Computing the value function of a fixed policy.
policy gradient
Estimating the gradient of expected return with respect to policy parameters and ascending it.
Example: Increasing the probability of actions that led to high reward.
Sections:
policy improvement
Making a policy greedy with respect to its own value function.
policy iteration
Alternating policy evaluation and policy improvement until the policy stops changing.
polynomial
A sum of constant coefficients times whole-number powers of a variable.
positive definite
A symmetric matrix whose quadratic form is positive for every nonzero vector.
Formal definition: x^T A x > 0 for all x != 0
Example: The identity matrix is positive definite.
postcondition
The clause of a contract that states what is true of the returned value, provided that the procedure terminates.
Sections:
posterior
The belief in a hypothesis after the evidence has been taken into account.
power law
A relationship in which multiplying the input by a fixed factor multiplies the output by a fixed factor.
Sections:
power method
Finding the dominant eigenvector by multiplying a vector by the matrix and rescaling, over and over.
Sections:
precision
The fraction of generated samples that lie inside the manifold of real data.
Sections:
precondition
The clause of a contract that states what every caller must establish before the call, which is the caller's obligation.
Sections:
prediction
The value a predictor returns for one input.
Example: The model answers 7 for an image of a seven.
Sections:
predictor
Any procedure that accepts an input and returns a predicted label or value.
Example: A rule that returns the label of the nearest stored example.
Sections:
prefix code
A code in which no codeword is the beginning of another codeword.
Sections:
pretraining
Training a model on a broad body of text before any particular task is named.
Sections:
primal value
The ordinary value a program computes, before any derivative is attached to it.
Sections:
principal component analysis
Projecting data onto the directions of greatest variance, computed from the SVD of the centred data.
Example: The first principal component of a cloud of points is its longest direction.
Sections: no section recorded yet
prior
The belief in a hypothesis before the new evidence arrives.
probability
A number between zero and one assigned to an event, obeying the sum rule over disjoint events.
Example: The probability of a fair coin landing heads is one half.
probability distribution
The list of a random variable's values together with the probability of each value.
probe
A small classifier trained on a layer's vector to test whether a property can be read from it.
Sections:
procedure
One named block of steps inside a program, which carries a promise about its behaviour to everyone who uses it.
Sections:
product rule
The rule that the belief in two events together equals the belief in the first times the belief in the second given the first.
Sections:
program
A sequence of steps that turns inputs into outputs, together with a promise about the result.
Example: A square-root routine that returns a number whose square is within 0.001 of the input.
projection
The point of a subspace closest to a given list, found by dropping a perpendicular onto the subspace.
Sections:
promise
The statement of what a caller may rely on when a procedure runs, which the contract makes precise on four lines.
Sections:
proof
A finite chain of valid steps from assumptions to a conclusion.
Example: Assuming n is even, write n = 2k, so n squared is even.
provenance
The record of where data, code and results came from and how they were produced.
Example: A log line naming the commit, the seed and the input file.
Sections:
pseudo-alignment
Agreement with the intended objective on the training distribution that does not hold on new inputs.
Sections:
pseudoinverse
The rule that returns the least-squares coefficients, and among ties the shortest coefficient list.
Sections:
Q
Q-learning
A value-based reinforcement learning method that learns action values from experience without a model of the world.
Example: Updating a table of state-action values after each step.
QR factorisation
Writing a table as an orthonormal basis for its columns times a triangular table of coefficients, A = QR.
Sections:
quadratic form
An expression of the shape x-transpose A x, where A is a symmetric matrix and x is a vector of variables.
quantifier
A phrase such as for all or there exists that states how many objects a claim covers.
Example: For every integer n there exists an integer m larger than n.
Sections:
quantifier prefix
The opening run of quantifiers in a sentence, read from left to right, each choosing after the ones before it.
Sections:
query
A learned vector that says what one position is looking for in the other items.
Sections:
R
random seed
The number that initialises a pseudo-random generator and fixes a run.
Example: Seed 17 reproduces the same shuffle on every machine.
Sections:
random variable
A function that assigns a number to each outcome of an experiment.
Example: The number of heads in ten flips.
Sections:
rank
The dimension of the image of a linear map.
Example: A 2-by-2 matrix that maps the plane onto a line has rank one.
recall
The fraction of real examples that lie inside the manifold of generated samples.
Sections:
recomputation
A forward evaluation spent during the backward pass to rebuild a value that was not stored.
Sections:
rectified linear unit
An activation function that returns its input when the input is positive and zero otherwise.
recurrence
An equation that defines a sequence or a running time in terms of smaller cases.
Example: T(n) = 2 T(n/2) + n.
Sections:
recurrent neural network
A network that processes a sequence one step at a time while carrying a hidden state.
Example: Reading a sentence word by word.
Sections:
recursion
A procedure that calls itself on a smaller version of the same problem.
Example: Factorial(n) = n times Factorial(n-1), with Factorial(0) = 1.
Sections: no section recorded yet
recursion tree
A picture of a recurrence with one node per recursive call, whose level sums add up to the total cost.
Sections:
recursive procedure
A procedure whose definition refers, directly or indirectly, to the procedure itself, which is a fact about the text rather than about the run.
recursive process
A process that builds a chain of operations to be performed later, expanding until it reaches a case it can answer outright and then contracting as the deferred operations run.
Sections:
regularisation
Adding a penalty or constraint that discourages complicated models.
Example: Adding lambda times the squared coefficient size to the loss.
REINFORCE
A policy-gradient algorithm that weights each update by the return of the whole episode.
Sections:
relation
A set of ordered pairs, expressing which objects are related.
Example: The relation less-than on the integers.
Sections:
relative error
The size of the error divided by the size of the true answer.
remainder
The gap between a Taylor polynomial and the function it approximates.
reparameterization trick
Rewriting a random draw as a fixed random input plus a shift and a scale, so gradients can pass around the draw.
Sections:
representation
A way of storing a kind of object, together with the conditions that a stored value must satisfy to be a valid one.
reproducibility
The property that another person can rerun a procedure and obtain the same result.
Example: A one-file script that regenerates every table in a paper.
resample
One new sample drawn from your data, with replacement, and the same size as the original.
Sections:
residual
The difference between a recorded value and the value a model predicts for that row.
residual block
A block whose output is its input plus a learned correction.
Sections:
residual connection
Adding a layer's input to its output so information and gradients can bypass the layer.
Formal definition: y = x + F(x)
Example: The skip connection in a ResNet block.
residual network
A network built from stacked residual blocks.
Sections:
residual plot
A graph of residuals against the input or the fitted value, used to reveal structure the model missed.
residual sum of squares
The total cost of a fit: the sum of the squared residuals over all rows.
Sections:
return
The sum of the rewards ahead, with each later reward reduced by a discount factor.
Sections:
reverse mode
Automatic differentiation that walks the operations backwards from one output and accumulates a derivative at every node.
Sections:
reverse process
The learned sequence of steps that removes noise and rebuilds a picture.
Sections:
reward
A single number the environment returns at each step, which the agent seeks to maximize over time.
Sections:
reward hacking
Satisfying the letter of a written objective in a way that defeats its purpose.
Sections:
ridge point
The number of operations a machine performs per second divided by the number of bytes it can deliver per second.
Sections:
ridge regression
Least squares with a penalty on the squared size of the coefficients.
Example: Shrinking coefficients toward zero without deleting any.
Sections:
rounding error
The difference between a true value and its stored or computed approximation.
Example: Adding 0.1 and 0.2 gives 0.30000000000000004.
Sections: no section recorded yet
row space
The set of all combinations of the rows of a table.
Sections:
S
saddle point
A point where a function curves up in some directions and down in others.
sample space
The set of all possible outcomes of an experiment.
Example: The pair of results from two coin flips.
sampling
Checking a claim on finitely many cases drawn from the collection it covers.
Sections:
sampling distribution
The distribution of a statistic, such as an average, across repeated samples of the same size.
scalar
A single number used to multiply a vector or another scalar.
Example: The number 2.5.
scaled dot-product attention
Dot-product attention with the scores divided by the square root of the key length before the softmax.
Sections:
scaling law
A regular relationship between model size, data, compute and loss.
Example: Loss falling as a power law in parameter count.
Sections:
schedule
A rule fixed before the backward pass that says which values are kept and which are recomputed.
Sections:
secant
A straight line through two points of a curve.
second derivative
The derivative of the derivative, which records how the slope itself changes.
second moment
The running average of the squared gradient, whose square root measures the typical gradient size.
Sections:
seed
The value the backward walk starts from at the output, which selects the combination of outputs being priced.
Sections:
selection set
A part of the data read while choosing among candidate rules, before a final number is reported.
selector
A procedure that takes an object apart and returns one of its parts.
Sections:
self-attention
Attention in which the queries and the items come from the same sequence.
self-supervised learning
Learning in which the supervision signal is generated from the data itself rather than supplied by a person.
Sections:
sequence
An ordered list of values, one for each whole-number step.
Sections:
series
A sum of terms taken in order, which converges when its partial sums approach a fixed number as more terms are added.
Sections:
set
A collection of distinct objects, with no order and no repeats.
Example: The set of digits {0, 1, ..., 9}.
Sections:
sigmoid
An activation function that squashes every input into the interval between zero and one.
sign bit
The single bit that says whether the number is positive or negative.
Sections:
significance level
The largest probability of a false alarm that the test is allowed to have.
Sections:
singular value
One of the non-negative stretch factors in the SVD, equal to the length of the image of a unit right singular vector.
Sections:
singular value decomposition
Writing any matrix as a rotation, a non-negative scaling, and another rotation.
Formal definition: A = U Sigma V^T
Example: A 2-by-2 shear decomposes into two rotations and a stretch.
Sections:
singular vector
One of the perpendicular input or output directions that the SVD rotates onto the axes before and after stretching.
Sections:
skip connection
A connection that carries a signal past one or more layers unchanged.
Sections:
slope
The rise divided by the run between two points on a line.
slot
One stored intermediate value, the unit in which memory is counted here.
Sections:
smoothness
A bound on how fast the gradient of a function can change.
Example: A Lipschitz gradient constant L.
soft-thresholding
Shrinking a number toward zero by a fixed amount and leaving it at zero once it would cross.
Sections:
softmax
A function that turns a list of scores into positive numbers that sum to one.
Formal definition: softmax(z)_i = exp(z_i) / sum_j exp(z_j)
Example: Scores 2, 1, 0 become roughly 0.665, 0.245, 0.090.
source coding theorem
The result that long blocks from a source can be compressed to about their entropy in bits and no further without loss.
Sections:
spacing
The distance from a number to the next representable number of the same format.
Sections:
span
All linear combinations of a set of vectors.
Example: The span of (1,0) and (0,1) is the whole plane.
Sections:
sparse
Having most entries equal to zero, so that only a few inputs or coefficients are in play.
Sections:
sparse autoencoder
A network trained to reconstruct a model's internal activations through a narrow, sparse layer whose active units are candidate features.
Sections:
spectral theorem
Every symmetric real matrix can be diagonalised by an orthogonal change of basis.
Formal definition: A = Q Lambda Q^T with Q orthogonal
Example: A covariance matrix has perpendicular principal axes.
squared loss
A loss that charges the square of the difference between the produced answer and the true answer.
Sections:
standard deviation
The square root of the variance, in the same units as the variable.
Example: A spread of about two centimetres around a mean height.
standard error
The standard deviation of an average across repeated samples, equal to the spread of one observation divided by the square root of the sample size.
state
The information that decides what a process does next, namely the values its next step reads.
state-value function
A value function with one entry per state.
statistical efficiency
The rate at which an estimator's error shrinks with sample size, compared with the best available rate.
Sections:
steepest descent
Gradient descent: repeatedly stepping in the direction opposite the gradient.
Sections:
step size
The number that scales how far one gradient step moves.
Sections:
stochastic gradient
The average gradient over a minibatch, used in place of the gradient over the whole table.
Sections:
stochastic gradient descent
Gradient descent using a gradient estimated from a small random batch instead of the whole dataset.
Example: Updating weights after each minibatch of 32 examples.
Sections:
stored row
A row the rule may keep and consult, which in this chapter means a row that carries a label.
Sections:
string
A finite row of symbols drawn from a fixed alphabet.
Sections:
strong convexity
A lower bound on the curvature of the loss, so the loss bends at least as much as a fixed quadratic.
Sections:
subgradient
Any slope that supports a function at a point where the function has no derivative.
Sections:
subnormal
A number below the smallest normal one, whose leading digit is zero and whose precision is smaller than the format's.
Sections:
subsampling
Shrinking a feature map by summarising each small patch with one number.
Sections:
subset
A set whose members all lie in another set.
Sections:
subspace
A flat sheet through the origin, such as a line, a plane, or a higher-dimensional relative of the two.
Sections:
sum rule
The rule that beliefs in mutually exclusive events add.
Sections:
surjective
A function is surjective when every output receives at least one arrow.
Sections:
swamping
The loss of an update that is smaller than the step of the weight it is added to.
Sections:
symbolic differentiation
Differentiating a formula by the rules of calculus and returning another formula.
Sections:
symmetric derivative
The average of the two one-sided slopes, which is what a centred difference returns at a corner.
Sections:
symmetric matrix
A square matrix that equals its own transpose, so its entry in row i and column j equals the entry in row j and column i.
Sections:
T
tail
The probability that a random variable lands far from its centre, usually beyond some stated distance.
Sections:
tangent
The straight line through a point of a curve whose error shrinks faster than the distance from the point.
tape
An append-only list implementation of a trace.
Sections:
Taylor approximation
Approximating a function near a point by a polynomial built from its derivatives there.
Formal definition: f(x+h) = f(x) + f'(x)h + f''(x)h^2/2 + ...
Example: Approximating sin(x) by x near zero.
Sections: no section recorded yet
Taylor polynomial
The polynomial of degree n whose first n derivatives at a chosen point equal the function's first n derivatives there.
TD error
The gap between the old value estimate and the one-step target.
temporal difference
Learning a value from the reward received and the next value estimate, without waiting for the episode to end.
terminal state
The special state in which an episode ends.
Sections:
termination
The property that a procedure produces a final value at all, rather than running forever.
Sections:
test
One clause of a promise, executed on particular values so that something objects when the clause fails.
Sections:
test set
Examples kept away from learning and used once to estimate performance.
Example: The last 20 percent of the table, opened after the model is frozen.
test statistic
The single number computed from the data that the test judges.
Sections:
thread
One stream of instructions with its own private working storage.
Sections:
throughput
The rate at which completed work leaves the machine.
Sections:
time complexity
How the number of basic steps grows with the input size.
Example: Merge sort takes time proportional to n log n.
Sections:
token
A small unit of text, such as a word piece or a character sequence, that a language model reads.
Example: The word playing split into play and ing.
Sections:
tokeniser
A program that cuts text into the tokens a model can read.
Sections:
tolerance
A fixed amount by which an approximate answer is allowed to differ from the exact one before the procedure stops.
topological order
A listing of the nodes in which every node comes after the nodes that feed it.
Sections:
total variation distance
Half the sum of the absolute differences between the probabilities two distributions assign to the same outcomes.
Sections:
trace
The sum of the diagonal entries of a square matrix.
training loss
The average loss that the model achieves on the examples it is trained on.
Sections:
training phase
The part of a method's run in which it reads stored rows and fixes what it will use later.
Sections:
training set
The examples a learning procedure may use to choose its predictor.
Example: The first 80 percent of a labelled table.
transformer
A network built from attention, small feedforward layers, normalisation and residual connections.
Example: The architecture behind a modern language model.
Sections:
transformer block
One attention sublayer plus one feedforward sublayer, each wrapped in a residual connection and a normalisation.
Sections:
transition
One step from a state and an action to a next state and a reward.
translation
A shift of the input, such as moving an image a few pixels to the right.
Sections:
transpose
The same table flipped about its diagonal, so that its columns become rows; multiplying by it takes dot products with the original columns.
Sections:
true risk
The average loss of a predictor over the whole population it will meet.
Formal definition: R(h) = E[ell(h(X), Y)]
Example: The error the model would show on all future images, not only the stored ones.
Sections:
truncation
Stopping the backward pass after a fixed number of steps instead of following the whole chain.
Sections:
truncation error
The gap between a finite-difference estimate and the exact derivative, caused by using a step that is not zero.
Sections:
tuning parameter
A setting chosen from data other than the training rows, such as the number of neighbours.
two-sample test
A statistical test of whether two sets of samples could have been drawn from the same distribution.
Sections:
typical set
The small set of long sequences that carries almost all the probability of a source.
Sections:
U
unbiased
An estimator is unbiased when its average over repeated samples equals the quantity it estimates.
underfitting
Using a model too simple to capture the pattern in the data.
Example: A straight line fitted to data that clearly curves.
uniform sample space
A sample space in which every outcome carries the same probability.
Sections:
unit
A small calculation that weights an input vector, adds a bias and returns one number.
Sections:
unit roundoff
The largest relative error that one rounding can cause.
Sections:
V
validation loss
The average loss that the model achieves on examples held out from training.
Sections:
validation set
Examples used to choose settings, kept separate from the final test set.
Example: A middle 10 percent used to pick k.
value
The content an item passes on once attention has decided how much of it to read.
value function
The expected return from a state, or from a state-action pair, under a policy.
Example: The expected total reward starting from the top-left cell.
value iteration
One combined sweep that evaluates and improves at the same time.
vanishing gradient
The gradient becoming extremely small as it propagates through many layers, so early layers learn very slowly.
Example: A deep sigmoid network whose first-layer gradients are near zero.
variance
The expected squared distance of a random variable from its mean.
Formal definition: Var(X) = E[(X - E[X])^2]
Example: The variance of one fair coin flip is 0.25.
variational autoencoder
A latent-variable model trained by maximizing the evidence lower bound with an amortised encoder.
Example: An encoder that maps an image to a Gaussian code and a decoder that reconstructs it.
Sections:
vector
An ordered list of numbers that can be added and scaled.
Example: The pair (3, 4) in the plane.
vector space
A set with an addition and a scalar multiplication that obey the usual rules.
vector-Jacobian product
The transpose of the Jacobian applied to a seed, which is the derivative of a scalar output with respect to every input.
Sections:
vocabulary
The finite list of tokens a model is allowed to use.
Sections:
vote
The count of labels among the neighbours, whose largest count decides the prediction.
Sections:
W
warp
Thirty-two threads that execute one instruction together.
Sections:
weight
The number that sets how strongly guidance leans toward the condition.
weight sharing
Using one set of weights at every position instead of a separate set for each position.
Sections:
witness
A specific object that makes a there-exists claim true.
Sections:
X
Xavier initialisation
A rule that sets the weight variance to 2 divided by the sum of fan-in and fan-out.
Sections:
Z
zero-one loss
A loss that charges one for a wrong answer and zero for a right one.
Sections: