Recurrent Neural Networks (RNNs) were designed to model sequences—text, time-series signals, clickstreams, sensor readings—by maintaining a hidden state that carries information forward in time. Yet classic RNNs are notoriously difficult to train on long sequences. The central reason is the behaviour of gradients during backpropagation through time (BPTT): they can shrink toward zero (vanishing gradients) or grow uncontrollably (exploding gradients). This article gives a mathematical treatment of why these issues occur and how modern practice mitigates them. If you are building deeper intuition for sequence learning through an AI course in Delhi, this topic is foundational for understanding why certain architectures and training tricks became standard.
A Quick Recap: RNN Forward Dynamics
A simple (Elman) RNN updates its hidden state as:
ht=ϕ(Whht−1+Wxxt+b)h_t = \phi(W_h h_{t-1} + W_x x_t + b)ht=ϕ(Whht−1+Wxxt+b)and outputs:
yt=g(Wyht+c)y_t = g(W_y h_t + c)yt=g(Wyht+c)Here, hth_tht is the hidden state at time ttt, xtx_txt is the input, WhW_hWh is the recurrent weight matrix, ϕ(⋅)\phi(\cdot)ϕ(⋅) is a nonlinearity such as tanh\tanhtanh or ReLU, and g(⋅)g(\cdot)g(⋅) maps hidden states to outputs. Training uses BPTT, which “unrolls” the network across time steps and applies the chain rule through the sequence.
Backpropagation Through Time and the Product of Jacobians
Let the total loss over a sequence be:
L=∑t=1Tℓt(yt,y^t)\mathcal{L} = \sum_{t=1}^{T} \ell_t(y_t, \hat{y}_t)L=t=1∑Tℓt(yt,y^t)To learn long-range dependencies, gradients from later losses must influence earlier states and parameters. Consider the gradient of the loss at time ttt with respect to a hidden state hkh_khk where k<tk < tk<t:
∂ℓt∂hk=∂ℓt∂ht∏i=k+1t∂hi∂hi−1\frac{\partial \ell_t}{\partial h_k} = \frac{\partial \ell_t}{\partial h_t}\prod_{i=k+1}^{t}\frac{\partial h_i}{\partial h_{i-1}}∂hk∂ℓt=∂ht∂ℓti=k+1∏t∂hi−1∂hiThe key term is the product of Jacobians:
Ji=∂hi∂hi−1=diag(ϕ′(ai)) WhJ_i = \frac{\partial h_i}{\partial h_{i-1}} = \text{diag}(\phi'(a_i))\, W_hJi=∂hi−1∂hi=diag(ϕ′(ai))Whwhere ai=Whhi−1+Wxxi+ba_i = W_h h_{i-1} + W_x x_i + bai=Whhi−1+Wxxi+b, and diag(ϕ′(ai))\text{diag}(\phi'(a_i))diag(ϕ′(ai)) is a diagonal matrix of activation derivatives.
This product determines whether information from time ttt can flow back to earlier time steps. If the norm of this product becomes tiny, gradients vanish. If it becomes huge, gradients explode.
Why Gradients Vanish or Explode: A Norm-Based View
Using operator norms, we can bound the gradient magnitude:
∥∏i=k+1tJi∥≤∏i=k+1t∥Ji∥≤∏i=k+1t∥diag(ϕ′(ai))∥⋅∥Wh∥\left\|\prod_{i=k+1}^{t} J_i\right\| \le \prod_{i=k+1}^{t} \|J_i\| \le \prod_{i=k+1}^{t} \|\text{diag}(\phi'(a_i))\| \cdot \|W_h\|i=k+1∏tJi≤i=k+1∏t∥Ji∥≤i=k+1∏t∥diag(ϕ′(ai))∥⋅∥Wh∥For tanh\tanhtanh and sigmoid, ∣ϕ′(z)∣≤1|\phi'(z)| \le 1∣ϕ′(z)∣≤1, and often much less than 1 when activations saturate. That means ∥diag(ϕ′(ai))∥\|\text{diag}(\phi'(a_i))\|∥diag(ϕ′(ai))∥ is typically < 1. If ∥Wh∥\|W_h\|∥Wh∥ is also less than 1, the product shrinks exponentially with t−kt-kt−k. This yields vanishing gradients: earlier states receive almost no learning signal.
Conversely, if ∥Wh∥\|W_h\|∥Wh∥ (or effective ∥Ji∥\|J_i\|∥Ji∥) exceeds 1 on average, the product can grow exponentially as the time gap increases. That yields exploding gradients: gradients become numerically unstable and can cause erratic updates.
A more precise picture comes from eigenvalues. If WhW_hWh has eigenvalues whose magnitudes are consistently below 1, repeated multiplication contracts vectors; above 1, it expands them. In practice, the nonlinearity’s derivative term modulates this, but the basic dynamic remains.
Learners in an AI course in Delhi often first see this as “long sequences are hard,” but the real reason is that BPTT creates repeated multiplications that amplify or suppress gradients over time.
Practical Consequences During Training
Vanishing gradients make it hard for an RNN to learn dependencies that are many steps apart. The model may focus on short-term patterns because only nearby time steps provide strong gradient signals. Exploding gradients cause training instability: loss spikes, parameters become NaN, and optimization diverges.
These effects are not rare edge cases. They are common when:
- Sequences are long
- tanh\tanhtanh or sigmoid saturates
- Recurrent weights are poorly initialised
- Learning rates are too large
Mitigation Strategies Used in Modern Workflows
Several techniques are used to manage RNN gradient dynamics.
Gradient clipping
This is the standard fix for exploding gradients. If ∥g∥\|g\|∥g∥ exceeds a threshold, rescale:
g←g⋅τ∥g∥g \leftarrow g \cdot \frac{\tau}{\|g\|}g←g⋅∥g∥τClipping prevents rare but destructive spikes from destabilising training.
Careful initialisation and regularisation
Initialising WhW_hWh with attention to its spectral radius (roughly, keeping eigenvalues near 1) can reduce both extremes. Regularisation and smaller learning rates also help control growth.
Truncated BPTT
Instead of backpropagating through the entire sequence, training backpropagates through a fixed window (e.g., 50–200 steps). This reduces the chain length, making gradients more stable, but it limits the model’s ability to learn very long dependencies.
Gated architectures (LSTM/GRU)
LSTMs and GRUs were designed specifically to maintain more stable gradient flow using gating mechanisms and additive state updates. Their structure creates paths where gradients can propagate with less repeated multiplicative shrinkage, improving learning over long horizons.
Normalisation techniques
Layer normalisation and related stabilisers can improve training by controlling activation scale, which indirectly helps keep ϕ′(ai)\phi'(a_i)ϕ′(ai) in a healthier range.
These are not optional “nice-to-haves.” They are the reason sequence models became reliable enough for real applications. Many applied projects covered in an AI course in Delhi use at least gradient clipping and a gated RNN baseline even when Transformers are the final choice.
Conclusion
The vanishing and exploding gradient problem in RNNs comes directly from the mathematics of BPTT: gradients depend on products of Jacobians across time, which can shrink or grow exponentially with sequence length. Vanishing gradients block long-range credit assignment, while exploding gradients destabilise optimisation. Modern practice addresses these issues through gradient clipping, careful initialisation, truncated BPTT, normalisation, and especially gated architectures like LSTMs and GRUs. If you are strengthening your sequence modelling fundamentals through an AI course in Delhi, mastering these dynamics will help you understand not just how RNNs work, but why certain training strategies are standard in real deployments.