Unrolling RNN

"Unrolling" an RNN means redrawing its self-referencing loop as an equivalent chain of copies — one per time step — turning a recursive structure into a picture that looks like an ordinary deep feedforward network. This reframing is exactly what makes backpropagation through an RNN concrete and computable.

From a Loop to a Chain

The recurrent connection from Recurrent Connections — \(\mathbf{h}_t = f(\mathbf{x}_t, \mathbf{h}_{t-1})\) — can be "unrolled" by writing out one copy of the cell per time step, each one receiving the previous copy's hidden state as input. Critically, every copy in this unrolled picture uses the identical shared weights (\(\mathbf{W}_{xh}, \mathbf{W}_{hh}, \mathbf{W}_{hy}\)) — the unrolled diagram is a visualization trick, not a claim that separate weight sets exist per step.

Diagram — A 4-Step Unrolled RNN

Why Unrolling Matters for Training

Once unrolled, an RNN looks structurally like a very deep feedforward network — and the same backpropagation machinery from the Backpropagation category applies directly: gradients flow backward through this unrolled chain, from the last time step to the first, exactly like flowing backward through the layers of a deep MLP. This specific application — backpropagation through an unrolled RNN — has its own name, Backpropagation Through Time (BPTT), covered in full in the next note but two.

The Depth Analogy

A sequence of length \(T\), once unrolled, behaves like a network with effectively \(T\) layers for the purposes of gradient flow — a 100-word sentence processed by an RNN is, from backpropagation's perspective, comparable to backpropagating through a 100-layer feedforward network. This framing directly explains why the vanishing/exploding gradient problems from the Backpropagation category apply with particular severity to RNNs over long sequences, covered later in this category.

Code — PyTorch Handles Unrolling Automatically

import torch.nn as nn

rnn = nn.RNN(input_size=10, hidden_size=20, batch_first=True)
sequence = torch.randn(1, 50, 10)   # a sequence of 50 time steps

output, h_final = rnn(sequence)
# Internally, PyTorch conceptually "unrolls" this into 50 sequential cell
# applications and builds a computational graph exactly as long -- you never
# have to write the unrolling loop by hand, but this IS what's happening

Common Mistakes

• Thinking of unrolling as creating genuinely separate weight copies — it's purely a way of visualizing and reasoning about the computation graph; the underlying weights remain a single shared set throughout.
• Underestimating how "deep" a long sequence effectively makes an RNN for gradient-flow purposes — a 200-step sequence is, in this specific sense, deeper than most feedforward networks covered elsewhere in this hub.

Interview Relevance

Q: "In what sense is processing a long sequence with an RNN similar to training a very deep feedforward network?" Once "unrolled," an RNN processing a sequence of length \(T\) forms a computational graph with \(T\) sequential applications of the same cell — from backpropagation's perspective, this is structurally comparable to a \(T\)-layer feedforward network. This is exactly why long sequences make RNNs especially susceptible to vanishing and exploding gradients, the same mechanism covered in the Backpropagation category, just with "depth" now meaning "sequence length."

Practice Question

If an RNN processes a sequence of 30 time steps, roughly how many "layers deep" is the unrolled computational graph, for the purposes of reasoning about gradient flow?

Related DL Notes

• Recurrent Connections
• Backpropagation Through Time
• RNN Vanishing Gradient