Which schedule works best for deep learning?

Cosine annealing with warmup is widely used for transformers and large models. Step decay is common for CNNs trained from scratch. Exponential decay works well for fine-tuning. The best choice depends on your optimizer, dataset size, and model architecture.

What is a warmup schedule?

Warmup linearly increases the learning rate from near zero to the initial LR over a set number of epochs. This stabilizes early training — especially for adaptive optimizers like Adam — before the main decay phase begins.

How do I choose an initial learning rate?

A common starting point is 1e-3 for Adam and 0.1 for SGD. Run a brief LR range test (train for one epoch while increasing LR from 1e-7 to 10) and pick the value just before the loss stops decreasing.

What is the step decay drop factor?

The drop factor (gamma) multiplies the current LR every N epochs. A gamma of 0.1 every 30 epochs is a classic ResNet schedule. Smaller gamma = more aggressive decay.

Can I export the schedule for use in PyTorch or TensorFlow?

Yes — use the "Export CSV" button to download a table of LR values per epoch. You can feed this into a LambdaLR (PyTorch) or LearningRateScheduler (Keras) callback.

Learning Rate Scheduler Helper

Name: Learning Rate Scheduler Helper
Availability: InStock
Author: Nham Vu

Visualize and compare learning rate schedules (step decay, exponential, cosine annealing, warmup) for machine learning training.

Schedule Parameters

Initial Learning Rate

Total Epochs

Schedule Type

Minimum LR (eta_min)

Key Values

Initial LR

—

Final LR

—

Midpoint LR

—

Decay Ratio

—

LR Schedule Curve

Cosine Annealing

LR per Epoch (preview)

Epoch	Learning Rate	Relative to Init

Showing first 15 and last 5 epochs. Export CSV for the full schedule.

Summary

Visualize and compare learning rate schedules (step decay, exponential, cosine annealing, warmup) for machine learning training.

How it works

Enter your initial learning rate, number of epochs, and schedule type.
Adjust schedule-specific parameters (step size, decay rate, warmup epochs, etc.).
The chart updates in real time showing the LR value at each epoch.
Switch between schedules to compare their curves side by side.
Click "Export CSV" to download the LR-per-epoch table for use in your training script.

Use cases

Plan a learning rate schedule before starting a long GPU training run.
Understand how cosine annealing compares to step decay for your epoch budget.
Tune warmup duration for transformer or large-model fine-tuning.
Generate an LR table to feed into a custom training loop or logger.
Teach ML fundamentals by visually demonstrating how each schedule behaves.
Verify that your hand-coded schedule matches the expected curve.

Frequently Asked Questions

Last updated: 2026-07-23 · Reviewed by Nham Vu