What does perplexity measure?

Perplexity measures how well a language model predicts a sequence. Formally it is the exponentiation of the cross-entropy loss: PP = exp(H). A lower perplexity means the model is more confident and accurate — a perplexity of 1 would mean perfect prediction, while very high values indicate the model is nearly guessing.

What is a "good" perplexity score?

It depends heavily on the task and vocabulary size. For word-level models on standard English benchmarks like Penn Treebank, scores below 100 are considered strong. State-of-the-art large language models often reach single-digit perplexity on character-level tasks. Always compare perplexity scores on the same dataset and tokenization scheme.

What is the difference between nats and bits?

Cross-entropy in nats uses the natural logarithm (base e). Cross-entropy in bits uses log base 2. To convert: bits = nats / ln(2) ≈ nats × 1.4427. Perplexity = exp(H in nats) = 2^(H in bits) — the result is the same either way.

Can I enter probabilities greater than 1?

No. Probabilities must be in the range (0, 1] exclusive of zero. If your model outputs log-probabilities (negative numbers), convert them first: p = exp(log_p). Values outside (0, 1] are flagged as invalid.

How is this different from bits-per-character (BPC)?

Bits-per-character (BPC) is cross-entropy in bits measured at the character level. Perplexity is exp of cross-entropy measured at the token (word or subword) level. Both measure the same underlying uncertainty — they differ only in the unit of measurement and the tokenization granularity.

Perplexity Calculator

Name: Perplexity Calculator
Availability: InStock
Author: Nham Vu

Enter per-token probabilities or a cross-entropy loss value to compute language model perplexity and understand model quality.

Input

Token probabilities (one per line, range 0–1)

Example: GPT output token probabilities from a log-softmax layer (converted to linear scale).

Quick examples

Enter probabilities or a loss value, then click Calculate.

Perplexity

Cross-entropy (nats)

H = −(1/N) ∑ ln(p_i)

Cross-entropy (bits)

H_bits = H_nats / ln 2

Token count (N)

Avg token probability

Formula

Per-token breakdown

Token #	Probability	−ln(p)	Contribution

Summary

Enter per-token probabilities or a cross-entropy loss value to compute language model perplexity and understand model quality.

How it works

Choose an input mode: enter per-token probabilities (one per line) or a single cross-entropy loss value.
For per-token mode, the tool computes the average negative log-likelihood: H = -1/N * sum(log(p_i)).
For cross-entropy mode, the value is used directly as H.
Perplexity is then PP = exp(H). Lower perplexity means the model assigns higher probability to the correct tokens.
The result panel shows perplexity, cross-entropy (nats and bits), and an interpretation of the score.

Use cases

Evaluate and compare language model quality on a held-out test set.
Convert a reported cross-entropy loss (e.g., from PyTorch training logs) into the more interpretable perplexity metric.
Verify perplexity calculations by hand when debugging NLP training pipelines.
Understand how individual token probabilities contribute to overall model perplexity.
Quickly sanity-check whether a reported perplexity figure is plausible given the token probabilities.
Learn the relationship between cross-entropy, bits-per-character, and perplexity.

Frequently Asked Questions

Last updated: 2026-07-22 · Reviewed by Nham Vu