Shannon Entropy

Tap a term to see what it means, or .

Calculate

Example: a source that emits four symbols, with chances of 1/2, 1/4, 1/8 and 1/8. How much information, in bits, does each symbol carry on average? Change the probabilities — or enter counts, which get converted — to work out your own case.

Probabilities or counts Separated by spaces. If they don't add up to 1, they are treated as counts.

Step by step

What each term means

Entropy (H): The average uncertainty about the outcome, in bits. Zero when the outcome is certain; highest, log₂ k, when the k outcomes are equally likely.
Sum with the Sign Flipped (−Σ): A sum over all outcomes. The minus sign is there because the log of a probability is negative, and the entropy should come out positive.
Probability (pᵢ): The weight of each outcome: the average is weighted by the chance of each one happening.
Surprise (log₂ pᵢ): −log₂ p is an outcome's surprise: 1 bit for a chance of 1/2, 3 bits for 1/8. Rare outcomes surprise more.