How is weight memory calculated?

Weight memory = number_of_parameters × bytes_per_element. FP32 uses 4 bytes, FP16/BF16 uses 2 bytes, and INT8 uses 1 byte per parameter. A 7B-parameter model in FP16 therefore needs roughly 14 GB just for weights.

What are activations and why do they scale with batch size?

Activations are intermediate tensor values produced during the forward pass that must be kept in memory for backpropagation. They are proportional to batch size: doubling the batch doubles activation memory.

Why does AdamW use so much more memory than SGD?

AdamW stores two additional FP32 tensors per parameter — the first moment (mean) and the second moment (variance) — each the same size as the weight tensor. That adds 8 bytes per parameter on top of the model itself. SGD with momentum adds only one FP32 moment vector.

Does gradient memory depend on batch size?

No. Gradients are one tensor per parameter (same shape as the weights) so they occupy the same memory regardless of batch size. With mixed-precision training the gradient copy is typically held in FP32 for numerical stability.

What is mixed-precision training?

Mixed precision keeps weights in FP16 or BF16 for the forward/backward pass (faster, less memory) but accumulates gradients and optimizer states in FP32 for numerical stability. This tool models that: when FP16/BF16 is chosen, weight memory uses 2 bytes per parameter but gradient/optimizer memory stays at 4 bytes.

Why does the estimate differ from what I see in nvidia-smi?

This tool models the dominant memory consumers. Additional overhead comes from CUDA context, framework buffers, temporary workspace, and kernel caches — typically 0.5–2 GB extra. Add a 15–20% safety margin for real workloads.

Batch Size Memory Estimator

Name: Batch Size Memory Estimator
Availability: InStock
Author: Nham Vu

Enter model parameters, data precision, and batch size to estimate training memory usage — weights, activations, gradients, and optimizer states included.

Model Configuration

Model Parameters

Total trainable parameters in the model

Numerical Precision

Optimizer

Batch Configuration

Batch Size

Number of samples per training step

Activation Memory per Sample (MB)

Rough intermediate tensor size for one sample during forward pass

Memory Breakdown

Configure the model and click "Estimate Memory"

Batch Size vs. Memory

Run an estimate first to see the sweep.

Batch Size	Weights + Grad + Opt	Activations	Total

Copied!

Summary

Enter model parameters, data precision, and batch size to estimate training memory usage — weights, activations, gradients, and optimizer states included.

How it works

Enter the total number of model parameters (e.g. 7000 for 7B, or use the "M/B" suffix selector).
Choose numerical precision: FP32, FP16/BF16, or INT8.
Select an optimizer: AdamW (standard), SGD with momentum, or None (inference only).
Enter the batch size and an average activation size per sample in MB.
Read the breakdown: weights, activations, gradients, optimizer state, and total.
Adjust batch size or precision until the total fits your available VRAM.

Use cases

Determine the largest batch size that fits on a specific GPU.
Compare memory cost of FP32 vs FP16 mixed-precision training.
Plan multi-GPU setups by knowing per-card memory demand.
Estimate whether a model fits in CPU RAM for inference.
Teach students how optimizer choice affects memory footprint.
Validate rough VRAM estimates before launching a cloud instance.
Plan gradient accumulation steps when full batch does not fit.

Frequently Asked Questions

Last updated: 2026-07-22 · Reviewed by Nham Vu