Recurrence and its gradient

Read the sentence "The film was not good" one word at a time. By the time you reach "good", "film" is three words back. A model that keeps only the two words before the current one has already discarded it, and the meaning of the sentence depends on that connection.

A recurrent neural network solves the problem by carrying a hidden state from one word to the next. The state is a list of numbers, and each new word updates it [S:69:389]. Whatever survives inside the state is what the model can use later, because the state is its only memory. !The same cell, with the same weights, applied once per word. Each state is a function of the current word and the state before it, and the chain is the computation that training must walk backwards. Nothing in the picture depends on the length of the sentence, which is why one set of parameters covers a document of any length. Status: schematic.

The update has a compact form. At step $t$, !Each step of the backward pass multiplies the signal by the recurrent factor, so the influence of an early word is that factor raised to the number of steps. A factor of 0.9 leaves a third of the signal after ten steps and almost nothing after a hundred; a factor of 1.1 grows past thirteen thousand over the same span. The line at one is the boundary, and no amount of training data moves a factor across it. Status: synthetic.

The line says the new state depends on the current word and on the previous state, and on nothing else [S:69:390].

The cell is the same small set of operations at every step. Unfold it once per word and the computation becomes a chain. The identical weights are applied at every link: the same input map, the same recurrent map and the same bias serve every step, so the parameter count does not grow with the length of the sentence [S:69:390]. That sharing is the sequence version of the convolution's bargain. A ten-word sentence and a thousand-word document pass through the same compact machine, and the state must compress whichever one arrives into the same fixed box.

A fixed window has its own price. A model whose prediction depends on the previous four words must store a table whose size grows exponentially with the window, because every combination of words needs a separate entry [S:69:388]. The recurrent state avoids that table by learning a summary instead, and the summary is what makes the parameter count independent of the length.

Think of the state as a compressed representation written on a single index card. Every word must be added to the card, the card never grows, and the network therefore has to decide which detail to discard. That decision is learned, and the card is the only route by which an early word can reach a later computation [S:69:388].

The compression is where the difficulty begins. A state that preserved everything read so far would be exact and far too large, so the network learns what to retain [S:69:388]. Training then has to decide which early words deserved credit for a late output, and the gradient, the list of slopes that says how each weight must move, carries that credit backwards. The backward pass, the trip that carries the error from the output back to the weights, runs through the unfolded chain and is called backpropagation through time. It applies the chain rule once per position, and the chain rule says that when one change passes through several steps, the slopes multiply [S:69:405].

Worked example. Use the simplest cell, $s_t=0.9s_{t-1}+x_t$, and feed an input of 1 at every step. With $s_0=0$, the state reaches $s_1=1$, then $s_2=0.9\times 1+1=1.9$, then $s_3=0.9\times 1.9+1=2.71$. Each update multiplies whatever the state already holds by 0.9 and adds the new input.

The first word's contribution is multiplied once for every word that follows it, so at $t$ words its share is $0.9^{t-1}$: about $0.387$ at ten words and about $3.0\times 10^{-5}$ at a hundred. The exponent counts the updates between the two positions, not the number of positions. Change the factor to 1.1 and the same share grows to about $2.36$ at ten words and about $1.25\times 10^4$ at a hundred.

The factor is not a metaphor. Each position contributes a derivative, the slope of the state with respect to a small change in what came before. The chain rule multiplies those slopes together, so the backward signal is a product with one term per position [S:69:405]. A product of many small terms is tiny, and a product of many large terms is enormous. The same arithmetic explains the vanishing gradient, the decay of that signal to nothing, in a deep feedforward network, one with many layers in a chain, with time playing the role of depth.

Over a hundred updates an early word either vanishes or swamps the sum [S:69:408]. A factor of magnitude below one produces a vanishing gradient, and a factor of magnitude above one produces an exploding gradient. Neither outcome is a software defect. Both follow from multiplying many numbers whose magnitudes all lie on one side of one.

The standard remedy truncates the chain. Truncation stops the backward pass after a fixed number of positions, which keeps training affordable and limits how far credit can travel [S:69:408]. A complete pass is exact and infeasible for a long document, while a truncated pass is cheap and approximate [S:69:405]. The truncation length becomes a design decision, and it determines the longest dependency the architecture can learn in practice.

Newer cells add gates, internal switches that hold one path close to one from position to position, so a value can survive for longer [S:69:409]. The gates help, and they do not remove the loop over positions.

The cost is architectural. Each position waits for the position before it, so the loop cannot run in parallel across time, and long sequences train slowly [S:217:2]. Memory consumption grows with the length of the chain because the backward pass stores every intermediate result. Truncation keeps the bill payable, at the price of a shorter memory. The next section removes the delay by letting every position examine every other position simultaneously.

The formal statement. A recurrent layer keeps a state $h_t$ and updates it with

$$h_t=f\left(x_t,h_{t-1}\right),$$

where $f$ holds two learned matrices, a bias and a nonlinearity, and the same $f$ serves every step [S:69:390]. The hypotheses are: one set of weights at every step, a differentiable update, a finite sequence, and a state whose width is fixed before the first word arrives. That last hypothesis is the compression: the same box holds one word or a thousand.

The derivation. Write the scalar case the core used, $h_t=0.9h_{t-1}+1$, and expand it. Each term of the expansion is an earlier input multiplied by one factor of 0.9 for every update between that input and the state you read. The gradient obeys the same arithmetic. Differentiate the last state with respect to an early one and every step in between contributes one factor. The number of factors is the number of updates between the two positions, not the number of positions.

A product of many numbers whose magnitudes all sit on one side of one is either tiny or enormous. The size of the product is set by the sum of the logarithms of the magnitudes, and that sum decides how fast the product moves away from one.

The edge case. The scalar factor stands in for a matrix. With a matrix, the product is a product of matrices, and its size in different directions is set by different singular values. A network can shrink the gradient along one direction while it stretches the gradient along another, so "the gradient vanishes" describes a direction, not the network [S:69:407]. A scalar has one direction and cannot show this.

A second edge case sits at magnitude exactly one, where the product neither decays nor grows. Order-insensitivity comes from the additive form, not from the eigenvalue. In the purely additive case $h_t=h_{t-1}+x_t$ the state is a running sum whose value is independent of the order of the inputs, because addition commutes. With a separate input map $W_x$ or a nonlinearity the state still carries the order, and the neutral eigenvalue buys stability of scale rather than commutativity. Gradient clipping repairs the exploding side and does not touch the vanishing side [S:69:410].

What the core hid. The factor is not a constant of the model. Each Jacobian depends on the state at that step, so one trained network can have a stable gradient on one input and an unstable one on another. The core's fixed 0.9 describes one linearised path through the model.

Truncation, precisely. A truncated backward pass is not a cheaper version of the exact gradient. It is the exact gradient of a different objective, one that ignores dependencies longer than the window. D2L reports that short truncation acts as a regulariser [S:69:405], which is what a changed objective does.

Where the analogy breaks. The index card suggests writing and later reading. The update multiplies the whole state by a matrix and then adds, so old content is mixed into every coordinate. No coordinate holds a retrievable copy of the first word.

Original sources. The latent-variable form of the recurrent model arrives with its approximation sign: the next token's probability is conditioned on a hidden state rather than on the whole prefix, and the state is updated from the current input and the previous state [S:69:388]. The same chapter derives backpropagation through time, expands the gradient into a sum of powers of the recurrent weight matrix, and names the consequence. Eigenvalues of magnitude below one vanish and those of magnitude above one diverge, while an eigenvalue of magnitude exactly one neither decays nor grows [S:69:407]. The memory cell arrives as the repair, with a self-connected recurrent edge of fixed weight one [S:69:410].

An open question. What sets the longest dependency a truncated model can learn? The truncation length is chosen for cost. The chapter reports that randomised truncation does not beat regular truncation by much, and that short truncation acts as a regulariser [S:69:405]. What the corpus does not give is a rule that maps truncation length and state width to the dependency lengths a model will learn. Evidence that would settle one: a task with a single relevant item at a known distance, swept over distance, truncation length and state width, with the accuracy cliff reported for each setting.

A frontier result. Titans treats the two architectures as two kinds of memory. Attention keeps the whole context and models dependencies exactly, at quadratic cost. A recurrent state compresses history into a fixed-size memory, which is cheap and lossy [S:396:1]. The paper adds a neural memory module that learns to write and read during inference, and reports context windows beyond two million tokens [S:396:1]. Its own definition of a recurrent network, as an update and a retrieval on a vector-valued memory, is the cleanest formal statement of the idea this section introduced [S:396:2].

What to read next. Read D2L 9.4 for the state update and 9.7 for the gradient expansion [S:69:388][S:69:407]. Read the opening of D2L chapter 10 for the constant-weight edge [S:69:410]. Then read the first three pages of Titans for the memory taxonomy and the long-context claim [S:396:1][S:396:3].