What is the gradient descent update rule?

The rule is θ_new = θ − α × ∇L(θ), where θ is the current parameter, α is the learning rate, and ∇L is the gradient of the loss with respect to θ. Subtracting the gradient moves the parameter in the direction that reduces the loss.

What learning rate should I use?

Typical starting values range from 0.001 to 0.1. Too high a learning rate causes overshooting; too low slows convergence. The chart shows this behavior visually — watch for oscillations as a sign your rate is too large.

What does a negative gradient mean?

A negative gradient means the loss increases as the parameter decreases. Subtracting a negative number moves θ in the positive direction, climbing toward the minimum from the left side of the loss curve.

What loss curve does the chart use?

The visualization uses a simple parabolic loss L(θ) = θ² with minimum at 0. This is a standard teaching example that clearly shows convergence or divergence from any starting point.

How is this different from stochastic gradient descent?

This tool computes a single deterministic step (vanilla gradient descent). SGD adds noise by computing gradients on random mini-batches. The update rule is identical; only the source of the gradient differs.

Gradient Descent Step Calculator

Name: Gradient Descent Step Calculator
Availability: InStock
Author: Nham Vu

Compute the next parameter value after one gradient descent step and visualize convergence on a loss curve.

Parameters

Current parameter value (θ)

Starting position on the loss curve.

Learning rate (α)

Controls step size. Try values from 0.001 to 1.0.

Gradient ∇L(θ) — optional override

Override auto-computed gradient

Leave unchecked to use the parabolic curve gradient 2θ.

Steps to visualize

1 15 40

Next Step Result

θ (current): —
Gradient ∇L(θ): —
Step size α×|∇L|: —
θ_new = θ − α∇L: —

Formula:

θ_new = θ − α × ∇L

Convergence on L(θ) = θ²

Each dot is one gradient descent step. Minimum is at θ = 0.

Summary

Compute the next parameter value after one gradient descent step and visualize convergence on a loss curve.

How it works

Enter your current parameter value (θ), the learning rate (α), and the gradient (∇L) at that point.
The tool applies the update rule: θ_new = θ − α × ∇L to compute the next parameter.
The step size (how far the parameter moves) is shown as α × |∇L|.
A loss curve chart plots multiple gradient descent steps from your starting point so you can see convergence behavior.
Adjust the learning rate to observe overshooting (too large) versus slow convergence (too small).

Use cases

Understand the gradient descent update rule while studying machine learning fundamentals.
Debug training instability by checking step sizes for given gradients and learning rates.
Demonstrate how learning rate affects convergence speed in lectures or tutorials.
Quickly verify manual gradient descent calculations during homework or research.
Explore the difference between large and small gradients on identical learning rates.
Visualize why a learning rate that is too high causes overshooting the minimum.

Frequently Asked Questions

Last updated: 2026-07-22 · Reviewed by Nham Vu