Bessel’s Correction

In the field of statistics, understanding the difference between population and sample statistics is crucial for accurate analysis. An important concept to understand is Besse’ls Correction. This note will explain the purpose and justification of this correction and it’s relationship to the concept of degrees of freedom.

What is Bessel’s Correction

Summary Statistics

Consider the sample and population statistics, summarised below:

	Population	Sample
Variance	$\sigma$	$s$
Mean	$\mu$	$\bar{x}$

$$\begin{array}{rl}\bar{x}& =\frac{1}{n}\sum _{i=1}^n\left[x_i\right]\\ \mu & =\frac{1}{N}\sum _{i=1}^N\left[x_i\right]\end{array}$$ $$\begin{array}{rl}{s}^2& =\frac{1}{n-1}\sum _{i=1}^n\left[{\left(x_i-\bar{x}\right)}^2\right]\\ {\sigma }^2& =\frac{1}{n}\sum _{i=1}^N\left[{\left(x_i-\mu \right)}^2\right]\end{array}$$

Comparing Population and Sample Statistics

The best estimate of the population mean from a sample is the mean of that sample, calculated in the exact same way, as a matter of fact, this fundamental assumption will later allow us to derive the gaussian distribution. The sample Variance, however, differs from the population variance in two important ways:

1. The difference concerns $\mu$ and not $\bar{x}$
   • If we have a sample we likely do not know $\mu$ and can only use $\bar{x}$
     • If, however, we did know $\mu$, then it would be more appropriate to use that and use the population formula to estimate $\sigma$
2. The denominator is $n-1$.

Bessel’s correction refers to the use of $n-1$ in the denominator instead of $n$. This correction is applied to ensure that the sample variance is an unbiased estimator of the population variance. The sample variance will tend to underestimate the population variance When using the sample mean as an estimate of the population mean, Bessel’s Correction will adjust for this.

Bias

The sample mean $\bar{x}$ is only an estimate of the population mean $\mu$, this means that the sample variance will be a less accurate when calculated with the sample mean than it would be with the population mean.

Since the sample mean is calculated from the same data points used to estimate the variance, it will be closer to those data points. Hence, the calculated variance will be smaller than the population variance.

If the sample mean is equal to the population mean, the distance from each observation will, of course, be equal, however, if the sample mean moves away from the population mean so too must the observations and the observations will always be closer to the sample mean than the population mean, because the sample mean is the centre of those points. Because this distance between $barx$ is less than sample variance will be less when using $\bar{x}$ as an estimator of the variance and it must be adjusted for, this is what Bessel’s Correction does, the reason $n-1$ is used relates to degrees of freedom and this will be explained below.

Note that using the mean of the distances (albeit squared), is still the best estimate of the population variance, just like the mean value, an assumption that leads to the derivation of a Gaussian Distribution is that the mean value of a sample is the best estimate of the population mean.

In summary, Bessel’s Correction compensates for the fact that the sample mean is likely to be closer to the data points in the sample than the true population mean, causing the variance to be slightly underestimated when using n as the denominator.

A Note on Maximum Absolute Deviation

A common question that arises in statistics, tangential to this topic, is why the squared-distance is used rather than the absolute distance (i.e. $\frac{1}{n}{\sum }_{i=1}^N\left|x_i-\mu \right|$), this relates to the assumption mentioned before, i.e. that the sample mean is the best estimate of the population mean, this is discussed further in deriving the normal distribution.

Degrees of Freedom and Variance

Degrees of freedom (df) is a concept that is fundamental to statistics. It refers to the number of independent pieces of information or parameters that can be freely changed when estimating a statistical parameter. More specifically it reflects the number of values in the final calculation that can vary without violating any constraints.

When calculating the sample variance, the sample mean $\bar{x}$ is used as an approximation of the population mean $\mu$, this consumes one degree of freedom, to see this, consider:

$$\begin{array}{r}\bar{x}=\frac{x_1+x_2+x_3}{3}\end{array}$$

if $\bar{x}$ is already known, any one of the remaining observations can easily be calculated, e.g. to get $x_1$:

$$\begin{array}{r}{x}_1=3\bar{x}-x_2-x_3\end{array}$$

Variance is concerned with the amount of variation between each point (observation), however, if one of the points is essentially fixed because it is a function of the others, then there is only $n-1$ points that are varying, and so the sample variance should only be calculated with respect to those n-1 points.

Conclusion

Bessel’s correction is necessary to accurately estimate the population variance when the population mean is not known. Understanding this correction and the connection to degrees of freedom lost by using sample mean is essential for accurate statistical analysis. By using Bessel’s correction, we can obtain reliable variance estimates and perform tests that rely on this value (e.g. t-test, ANOVA).

Further Proof

Variance in Terms of Expected Differences

$$\begin{array}{rl}\mathrm{Var}\left(X\right)& =\frac{1}{n}\sum _{i=1}^n\left[{\left(X_i-\text{mean}\left(X\right)\right)}^2\right]\\ \text{E}\left[\text{Var}\left(X\right)\right]& =\text{E}\left[\frac{1}{n}\sum _{i=1}^n\left[{\left(X-\text{E}\left[X\right]\right)}^2\right]\right]\\ & \text{The mean value is an unbiased estimator}\\ & \text{i.e. expected value of mean is the value}\\ & =\text{E}\left[{\left(X-\text{E}\left[X\right]\right)}^2\right]\\ & =\text{E}\left[X^2-2X\cdot \text{E}\left[X\right]+{\left(\text{E}\left[X\right]\right)}^2\right]\\ & =\text{E}\left[X^2\right]-\text{E}\left[2X\cdot \text{E}\left[X\right]\right]+{\left(\text{E}\left[X\right]\right)}^2\\ & =\text{E}\left[X^2\right]-2{\left(E\left[X\right]\right)}^2+E{\left[X\right]}^2\\ & =E\left[X^2\right]-\text{E}{\left[X\right]}^2\\ ⟹\text{E}\left[\frac{1}{n}\sum _{i=1}^n\left[X_i-\text{mean}\left(X\right)\right]\right]& =\mathrm{E}\left[X^2\right]-{\left(\text{E}\left[X\right]\right)}^2\end{array}$$

Summing Variances

$$\begin{array}{rl}\mathrm{Var}\left(X\pm Y\right)& =\mathrm{E}\left[{\left(X\pm Y\right)}^2\right]-\mathrm{E}{\left[X\pm Y\right]}^2\\ & =\mathrm{E}\left[X^2\pm 2XY+Y^2\right]-{\left(\mathrm{E}\left[X\right]\pm \mathrm{E}\left[Y\right]\right)}^2\\ & =\mathrm{E}\left[X^2\right]\pm 2\mathrm{E}\left[XY\right]+\mathrm{E}\left[{\mathrm{Y}}^2\right]-{\left(\mathrm{E}\left[X\right]\pm \mathrm{E}\left[Y\right]\right)}^2\\ & \text{If X and Y are Independent, the Expected Value is the Product}\\ & =\mathrm{E}\left[X^2\right]\pm 2\mathrm{E}\left[X\right]\mathrm{E}\left[Y\right]+\mathrm{E}\left[{\mathrm{Y}}^2\right]-{\left(\mathrm{E}\left[X\right]\pm \mathrm{E}\left[Y\right]\right)}^2\\ & \text{Expand the Square}\\ & =\mathrm{E}\left[X^2\right]\pm 2\mathrm{E}\left[X\right]\mathrm{E}\left[Y\right]+\mathrm{E}\left[{\mathrm{Y}}^2\right]-\mathrm{E}\left[X\right]\mp 2\mathrm{E}\left[X\right]\mathrm{E}\left[Y\right]-\mathrm{E}{\left[Y\right]}^2\\ & \text{Cancel out the ± and ∓}\\ & =\mathrm{E}\left[X^2\right]+\mathrm{E}\left[{\mathrm{Y}}^2\right]-\mathrm{E}\left[X\right]-\mathrm{E}{\left[Y\right]}^2\\ & =\left(\mathrm{E}\left[X^2\right]-\mathrm{E}{\left[X\right]}^2\right)+\left(\mathrm{E}\left[Y^2\right]-\mathrm{E}{\left[Y\right]}^2\right)\\ & =\mathrm{Var}\left(X\right)+\mathrm{Var}\left(Y\right)\end{array}$$

Multiplying Variance by a Constant}

$$\begin{array}{rl}\mathrm{Var}\left(aX\right)& =\mathrm{E}\left[{\left(aX\right)}^2\right]-\mathrm{E}{\left[ax\right]}^2\\ & \text{Take the constant from the left term}\\ & =a^2\mathrm{E}\left[X^2\right]-\mathrm{E}{\left[ax\right]}^2\\ & \text{Take the constant from the right term}\\ & =a^2\mathrm{E}\left[X^2\right]-{\left(a\mathrm{E}\left[X\right]\right)}^2\\ & =a^2\mathrm{E}\left[X^2\right]-a^2\mathrm{E}{\left[X\right]}^2\\ & a^2\left(\mathrm{E}\left[X^2\right]-\mathrm{E}{\left[X\right]}^2\right)\\ & a^2\mathrm{Var}\left(X\right)\end{array}$$

Variance of a Sample Mean

Consider a sample of size $n$, the population will have a variance $\sigma$ and a mean $\mu$, the sample will have a variance $s$ and a mean $\bar{x}$. Many different samples could be taken, so there is a distribution of $\bar{x}$ values that could occur. The variance of sample means $\mathrm{var}\left(\bar{x}\right)$ is given by $\mathrm{var}\left(x\right)=\frac{{\sigma }^2}{n}$, this is a part of the CLT and is shown here.

$$\begin{array}{rl}\mathrm{Var}\left(\bar{X}\right)& =\mathrm{Var}\left(\frac{1}{n}\sum _{i=1}^n\left[X_i\right]\right)\\ & \text{Applying the product rule from earlier}\\ & ={\left(\frac{1}{n}\right)}^2\mathrm{Var}\left(\sum _{i=1}^n\left[X_i\right]\right)\\ & \text{Applying the sum rule}\\ & =\frac{1}{n^2}\sum _{i=1}^n\left[\mathrm{Var}\left(X_i\right)\right]\\ & \text{The variance is already known}\\ & =\frac{1}{n^2}\sum _{i=1}^n\left[{\sigma }^2\right]\\ & =\frac{1}{n^2}n{\sigma }^2\\ & =\frac{{\sigma }^2}{n}\end{array}$$

Bessel’s Correction

Suppose that the sample variance $\left(s\right)$ was given by the typical formula:

$$\begin{array}{rl}s& \underset{?}{=}\frac{1}{n}\sum _{i=1}^n\left[\left(X_i-\mathrm{E}\left[X\right]\right)\right]\\ ⟹\mathrm{E}\left[s\right]& =\mathrm{E}\left[X^2\right]-\mathrm{E}{\left[X\right]}^2\end{array}$$

The expected value should be $\sigma$, in that sense it would be an un-biased estimator of the population mean.

The key here is to recognise that $s$ corresponds to a sample, by introducing $\bar{X}$we can solve for variance in terms of expected values and the sample mean. The sample introduces the bias so it is necessary to use the sample mean.

$$\begin{array}{rl}{s}^2& \underset{?}{=}\frac{1}{\mathrm{n}}\sum _{\mathrm{i}=1}^{\mathrm{n}}\left[{\left({\mathrm{X}}_{\mathrm{i}}-\bar{\mathrm{X}}\right)}^2\right]\\ ⟹\mathrm{E}\left[s^2\right]& =\mathrm{E}\left[\frac{1}{\mathrm{n}}\sum _{\mathrm{i}=1}^{\mathrm{n}}\left[{\left({\mathrm{X}}_{\mathrm{i}}-\bar{\mathrm{X}}\right)}^2\right]\right]\\ & \text{Expand the squre}\\ & =\mathrm{E}\left[\frac{1}{n}\sum _{i=1}^n\left[X_i^2-2X\bar{X}+{\bar{X}}^2\right]\right]\\ & \text{Distribute the sum}\\ & =\mathrm{E}\left[\frac{1}{n}\sum _{i=1}^n\left[X_i^2\right]-2\bar{X}\sum _{i=1}^n\left[X_i\right]+\frac{1}{n}\sum _{i=1}^n\left[{\bar{X}}^2\right]\right]\\ & \text{Distribute the expectation}\\ & =\mathrm{E}\left[\frac{1}{\mathrm{n}}\sum _{\mathrm{i}=1}^{\mathrm{n}}\left[{\mathrm{X}}_{\mathrm{i}}^2\right]\right]-2\mathrm{E}\left[{\bar{\mathrm{X}}}^2\right]+\mathrm{E}\left[\frac{1}{\mathrm{n}}\sum _{\mathrm{i}=1}^{\mathrm{n}}\left[{\bar{\mathrm{X}}}^2\right]\right]\\ & \text{Simplify the right-most term}\\ & =\mathrm{E}\left[\frac{1}{\mathrm{n}}\sum _{\mathrm{i}=1}^{\mathrm{n}}\left[{\mathrm{X}}_{\mathrm{i}}^2\right]\right]-\mathrm{E}\left[2{\bar{\mathrm{X}}}^2\right]+\mathrm{E}\left[\frac{1}{\mathrm{n}}\mathrm{n}{\bar{\mathrm{X}}}^2\right]\\ & =\mathrm{E}\left[\frac{1}{n}\sum _{i=1}^n\left[X_i^2\right]\right]-\mathrm{E}\left[2{\bar{X}}^2\right]+\mathrm{E}\left[{\bar{X}}^2\right]\\ & =\mathrm{E}\left[\frac{1}{n}\sum _{i=1}^n\left[X_i^2\right]\right]-\mathrm{E}\left[{\bar{X}}^2\right]\\ & \text{The mean value is an unbiased estimator of the mean}\\ & \text{and the values are i.i.d.}\\ & =\mathrm{E}\left[X^2\right]-\mathrm{E}\left[{\bar{X}}^2\right]\end{array}$$

\subsection*{Expected Value Squared}

The expected value of $X$ is the mean value:

$$\mathrm{E}{\left[X\right]}^2={\mu }^2$$

Expected Square Value

Recall the definition of Variance from earlier:

$$\begin{array}{rl}\mathrm{Var}\left(\mathrm{X}\right)\triangleq & \mathrm{E}\left[X^2\right]-\mathrm{E}{\left[X\right]}^2\\ ⟹\mathrm{E}\left[{\mathrm{X}}^2\right]& =\mathrm{Var}\left(X\right)+\mathrm{E}{\left[X\right]}^2\\ & ={\sigma }^2+{\mu }^2\end{array}$$

Applying this to the sample mean:

$$\begin{array}{rl}\mathrm{Var}\left(\bar{\mathrm{X}}\right)\triangleq & \mathrm{E}\left[{\bar{X}}^2\right]-\mathrm{E}{\left[\bar{X}\right]}^2\\ ⟹\mathrm{E}\left[{\bar{X}}^2\right]& =\mathrm{Var}\left(\bar{X}\right)+\mathrm{E}{\left[\bar{X}\right]}^2\\ & \text{The variance of the sample mean was derived earlier}\\ & =\frac{{\sigma }^2}{n}+\mu \end{array}$$

This step provides the key insight, if variance is taken on a population, there is no $n$ in the denominator

because the variance of a sample mean is different to the variance of a population statistic, the expected value of a squared sample mean is different to the expected value of a squared observation. The same $n$ from $\frac{\sigma }{\sqrt{n}}$is the same one that leands to $\frac{1}{n-1}.$The expected value of a squared sample mean is less than the expected value of a squared observation, because the sample mean is going to be more central.

\section*{Solving The Expected Sample Variance}

$$\begin{array}{rl}{s}^2& \underset{?}{=}\frac{1}{\mathrm{n}}\sum _{\mathrm{i}=1}^{\mathrm{n}}\left[{\left({\mathrm{X}}_{\mathrm{i}}-\bar{\mathrm{X}}\right)}^2\right]\\ ⟹\mathrm{E}\left[s^2\right]& =\mathrm{E}\left[X^2\right]-\mathrm{E}\left[{\bar{X}}^2\right]\\ & =\left({\sigma }^2+{\mu }^2\right)-\left(\frac{{\sigma }^2}{n}+{\mu }^2\right)\\ & =\frac{n{\sigma }^2-{\sigma }^2}{n}\\ & ={\sigma }^2\frac{n-1}{n}\end{array}$$

This shows that the variance formula, applied to a sample is biased. In order to correct that bias define $s_b=\frac{n}{n-1}s$ like so:

$$\begin{array}{rl}{s}_b^2& =\frac{n}{n-1}\frac{1}{n}\sum _{i=1}^n\left[{\left(X_i-\bar{X}\right)}^2\right]\\ & =\frac{1}{n-1}\sum _{i=1}^n\left[{\left(X_i-\bar{X}\right)}^2\right]\end{array}$$

The expected value of this is:

$$\begin{array}{rl}\mathrm{E}\left[s_b^2\right]& =\mathrm{E}\left[\frac{n}{n-1}s\right]\\ & =\frac{n}{n-1}\mathrm{E}\left[\mathrm{s}\right]\\ & =\frac{n}{n-1}\frac{n-1}{n}{\sigma }^2\end{array}$$