Understanding Gradient Descent

The usual picture of gradient descent is a ball rolling down a smooth bowl. It is a useful picture, right up to the moment a real training run lands on a landscape with ravines, plateaus, noisy slopes, and millions of directions. The core idea survives; the landscape is what makes optimization interesting.

What Gradient Descent Does

Training a model means choosing parameters that make a loss function smaller. Gradient descent uses the local slope of that loss to decide the next step.

If the parameters are collected in a vector $\theta$ and the objective is $J(\theta)$ , one update is:

\theta_{t+1}=\theta_t-\eta\nabla_\theta J(\theta_t)

The gradient points in the direction of steepest increase, so the minus sign moves the parameters downhill. The learning rate $\eta$ controls the step size.

A Small Example

Consider a one-parameter objective:

f(x)=(x-3)^2+1

Its derivative is:

f'(x)=2(x-3)

Starting at $x_0=0$ with $\eta=0.1$ gives:

x_1=0-0.1(-6)=0.6

Repeated updates approach the global minimum at $x=3$ . This example is deliberately well behaved: it is convex, smooth, and has one minimum. Neural-network losses are rarely that friendly.

def gradient_descent(x=0.0, learning_rate=0.1, steps=50):
    for _ in range(steps):
        gradient = 2 * (x - 3)
        x -= learning_rate * gradient
    return x

print(gradient_descent())

Local Minima, Global Minima, and Saddles

A local minimum is lower than nearby points. A global minimum is no higher than any point in the whole domain. For a convex objective, every local minimum is global. Deep networks produce non-convex objectives, so that guarantee does not apply.

A saddle point is a stationary point that curves upward in some directions and downward in others. Its gradient is zero, but it is neither a local minimum nor a local maximum. A broad flat region may also have tiny gradients, but flatness alone is not the definition of a saddle.

Noise from mini-batches and accumulated momentum can help move through some shallow or saddle-like regions. They do not guarantee escape from every poor solution.

The Learning Rate

The learning rate is often the first setting to inspect when training behaves badly:

Too large: the loss may oscillate, diverge, or become NaN.
Too small: progress may be stable but painfully slow.
Appropriate early, too large late: training reaches a useful region but never settles.

Schedules reduce the learning rate over time. Warm-up begins with smaller updates, which can help large-batch or transformer training. Adaptive optimizers change effective step sizes per parameter, but they still have a global learning-rate setting that matters.

Batch, Stochastic, and Mini-Batch Gradients

These terms describe how the gradient is estimated, not three unrelated optimization algorithms.

Full-batch gradient descent

The gradient uses the entire training set. With fixed inputs and no stochastic operations or nondeterministic kernels, it produces the same mathematical update from the same parameters, but each update can be expensive.

Stochastic gradient descent

In the strict sense, stochastic gradient descent uses one randomly selected example per update. The estimate is noisy and cheap.

Mini-batch gradient descent

Most deep-learning code uses a batch of examples. Matrix operations make it efficient on accelerators, while the estimate still has some sampling noise.

For batch $B_t$ :

g_t=\frac{1}{|B_t|}\sum_{i\in B_t}\nabla_\theta \ell_i(\theta_t)

In everyday usage, people often call this optimizer “SGD” even though it uses mini-batches.

Momentum

Momentum smooths updates by carrying a running direction:

v_t=\beta v_{t-1}+g_t

\theta_{t+1}=\theta_t-\eta v_t

It can accelerate progress along a consistent slope and reduce zig-zagging in narrow valleys. The exact sign convention varies between texts and libraries, so compare equations before copying hyperparameters.

RMSprop

RMSprop tracks an exponential average of squared gradients:

s_t=\rho s_{t-1}+(1-\rho)g_t^2

\theta_{t+1}=\theta_t-\eta\frac{g_t}{\sqrt{s_t}+\epsilon}

Parameters with consistently large gradients receive smaller effective steps. This can make optimization less sensitive to differences in gradient scale, but it does not remove the need to tune the learning rate.

Adam and AdamW

Adam combines a first-moment estimate with a squared-gradient estimate:

m_t=\beta_1m_{t-1}+(1-\beta_1)g_t

v_t=\beta_2v_{t-1}+(1-\beta_2)g_t^2

After bias correction:

\hat m_t=\frac{m_t}{1-\beta_1^t},\qquad \hat v_t=\frac{v_t}{1-\beta_2^t}

\theta_{t+1}=\theta_t-\eta\frac{\hat m_t}{\sqrt{\hat v_t}+\epsilon}

Adam is a useful baseline for many problems, but it is not universally best. SGD with momentum remains competitive in areas such as image classification, and results depend on schedules, regularization, batch size, and architecture.

AdamW decouples weight decay from Adam’s gradient-based update. With adaptive optimizers, adding an L2 penalty to the loss and applying decoupled weight decay are not generally equivalent.

Comparison

Method	What changes	Useful when	Main caution
Full batch	Uses every example per update	Datasets or models are small enough	Expensive updates; little sampling noise
Mini-batch SGD	Uses a batch estimate	A simple, well-understood baseline is wanted	Learning rate and schedule need care
Momentum	Accumulates a moving direction	Gradients zig-zag or progress slowly	Momentum and learning rate interact
RMSprop	Scales by recent squared gradients	Gradient scales vary by parameter	No universal default guarantees convergence
Adam	Tracks first and second moments	Fast initial progress and sparse/noisy gradients	Can generalize differently from SGD
AdamW	Adam plus decoupled decay	Weight decay is part of the training recipe	Exclude biases and normalization parameters when appropriate

Problems Gradient Descent Does Not Cause by Itself

Vanishing and exploding gradients are often listed as defects of gradient descent. They arise mainly from the derivatives produced by deep architectures, activation functions, initialization, recurrence, and normalization. The optimizer receives those gradients; it is not their sole cause.

Common responses include suitable initialization, residual connections, normalization, gated recurrent units, stable activation functions, and gradient clipping. Clipping controls extreme updates but can hide a deeper numerical problem if used without diagnosis.

A Practical Training Checklist

Plot training and validation loss rather than watching one final number.
Start with a small, known-to-work model and optimizer.
Check input scales, target encoding, and loss reduction.
Inspect gradients for NaN, infinity, or consistently tiny norms.
Try learning rates over several orders of magnitude.
Change one major training choice at a time.
Treat optimizer choice and its schedule as a pair.
Keep a validation set and a reproducible record of each run.

Standardizing numeric features often improves conditioning and speeds convergence, especially for linear models. Unscaled features do not always lead to an incorrect solution, but they can make the route to it unnecessarily difficult.

Conclusion

Gradient descent is a local update rule, not a promise that training will find the best possible model. Its success depends on the objective, gradient quality, parameterization, learning rate, data, and update rule. Understanding those pieces is more useful than treating Adam, momentum, or any other optimizer as a magic escape hatch.