Sample, population and variance
Sample, population and variance
Standard deviation describes how spread out a set of numbers is around its mean. A small value means the data clusters tightly; a large one means it is widely dispersed.
It is what the average alone cannot tell you. Two datasets can share a mean of 50 while one runs 49–51 and the other 0–100. The mean is identical; the standard deviation immediately distinguishes them.
Population: σ = √( Σ(x − μ)² ÷ N )
Sample: s = √( Σ(x − x̄)² ÷ (n − 1) )
Worked through, the procedure is:
The square root in the final step is why standard deviation is preferred over variance for interpretation: variance is in squared units (squared dollars, squared centimetres), which is hard to reason about.
The two formulas differ only in the denominator: N for a population, n − 1 for a sample. That subtraction is called Bessel's correction, and it exists for a real reason.
When you calculate deviations from a sample mean rather than the true population mean, those deviations are systematically slightly too small — the sample mean sits, by construction, at the centre of the sample. Dividing by n − 1 rather than n inflates the result just enough to correct that bias.
Use the population formula only when your data genuinely includes every member of the group you care about — all 30 students in one class, if the class is the entire population of interest. Use the sample formula whenever the data is a subset used to infer something wider, which covers most real situations.
The difference shrinks as n grows. With 5 values it is substantial; with 500 it is negligible.
For data that is approximately normally distributed:
If test scores have a mean of 75 and a standard deviation of 8, roughly two-thirds of students scored between 67 and 83, and about 95% between 59 and 91. A score of 99 sits three standard deviations out — genuinely unusual.
This only holds for roughly bell-shaped distributions. For heavily skewed data the rule breaks down, which is one more reason to look at your data rather than only its summary statistics.
Finance. Standard deviation of returns is the standard measure of volatility, and therefore of risk. A fund returning 8% with a deviation of 5% is a very different proposition from one returning 8% with a deviation of 25%.
Quality control. Manufacturing tolerances are expressed in standard deviations; the “six sigma” methodology takes its name directly from this.
Research. Results are commonly reported as mean ± standard deviation, and statistical significance tests are built on it.
Education. Standardised test scores and grading curves both rely on it to place an individual result in context.
Standard deviation assumes the mean is a meaningful centre. For strongly skewed data it can be misleading in the same way the mean is, and interquartile range is often the better description of spread. As with averages, calculating the number is the easy part — deciding whether it is the right number for your data is the real work.
The population formula divides by N; the sample formula divides by n minus 1. The correction compensates for the fact that deviations measured from a sample mean are systematically slightly too small. Use the sample formula unless your data covers the entire population of interest.
It is named after Friedrich Bessel. Dividing by n minus 1 rather than n removes the downward bias that arises when the sample mean, rather than the true population mean, is used to compute deviations.
That values are widely spread around the mean. In finance that indicates volatility; in manufacturing it indicates inconsistency; in test scores it indicates a wide range of performance.
Variance is the average squared deviation from the mean. Standard deviation is its square root, which returns the figure to the original units and makes it far easier to interpret.