The Sample Variance

denote the

k

moment about the mean. In particular, note that

d 0 1

d 1 0

, and

d 2 σ 2

. We assume that

d 4

We repeat the basic experiment

n

times to form a new, compound experiment, with a sequence of independent random variables

X X 1 X 2 X n

, each with the same distribution as

X

. In statistical terms,

X

is a random sample of size

n

from the distribution of

X

. Recall that the sample mean

is a natural measure of the center of the data and a natural estimator of the distribution mean

μ

. In this section, we will derive statistics that are natural measures of the dispersion of the data and are natural estimators of the distribution variance

σ 2

. The statistics that we will derive are different, depending on whether

μ

is known or unknown; for this reason,

μ

is referred to as a nuisance parameter for the problem of estimating

σ 2

A Special Sample Variance

First we will assume that

μ

is known. Although this is almost always an artificial assumption, it is a nice place to start because the analysis is relatively easy. Let

Properties

Show that $W 2$ is the sample mean for a random sample of size $n$ from the distribution of $X μ 2$ .

Use the result of Exercise 1 to show that

$W 2 σ 2$ .
$W 2 1 n d 4 σ 4$ .
$W 2 σ 2$ as $n$ with probability 1.

Use basic properties of covariance to show that $M W 2 d 3 n$ . It follows that the sample mean and the special sample variance are uncorrelated if $d 3 0$ and are asymptotically uncorrelated in any case.

The square root of the special sample variance is a special version of the sample standard deviation, denoted

W X

Use Jensen's inequality to show that $W σ$ . Thus, $W$ is a biased estimator that tends to underestimate $σ$ .

Show that if $c$ is a constant then $W 2 c X c 2 W 2 X$

The Standard Sample Variance

Consider now the more realistic case in which

μ

is unknown. In this case, a natural approach is to average, in some sense,

X i M 2

over

i 12 n

. It might seem that we should average by dividing by

n

. However, another approach is to divide by whatever constant would give us an unbiased estimator of

σ 2

Use basic algebra to show that

i 1 n X i M 2 i 1 n X i 2 n M 2

Use the result in Exercise 6 and basic properties of expected value to show that

i 1 n X i M 2 n 1 σ 2

is an unbiased estimator of

σ 2

; it is called the sample variance. As a practical matter, when

n

is large, it makes little difference whether we divide by

n

n 1

Basic Properties

The following alternate formula follows immediately from Exercise 6, and it is better for some purposes.

Show that

S 2 1 n 1 i 1 n X i 2 n n 1 M 2

Use the formula in the previous exercise and the (strong) law of large numbers to show that $S 2 σ 2$ as $n$ with probability 1.

Show that if $c$ is a constant then $S 2 c X c 2 S 2 X$

Show that $S 2 n n 1 W 2 M μ 2$

The square root of the sample variance is the sample standard deviation, denoted

S X

Use Jensen's inequality to show that $S σ$ . Thus, $S$ is a biased estimator than tends to underestimate $σ$ .

Moments

In this section we will derive formulas for the variance of the sample variance and the covariance between the sample mean and the sample variance. Our first series of exercises will show that

Verify the following result. Hint: Start with the expression on the right. Expand the term $X i X j 2$ , and take the sums term by term.

S 2 1 2 n n 1 i 1 n j 1 n X i X j 2

It follows that

S 2

is the sum of all of the pairwise covariances of the terms in the expansion of Exercise 13.

Suppose that $i j$ , Verify the following results. (Hint: In $X i X j m$ , add and subtract $μ$ , and then expand and use independence.)

$X i X j 2 2 σ 2$
$X i X j 4 2 d 4 6 σ 4$

Finally, derive the formula for $S 2$ by showing that

$X i X j 2 X k X l 2 0$ if $i j$ or $k l$ or $i$ , $j$ , $k$ , $l$ are distinct.
$X i X j 2 X i X j 2 2 d 4 2 σ 4$ if $i j$ , and there are $2 n n 1$ such terms in the sum of covariances.
$X i X j 2 X k X j 2 d 4 σ 4$ if $i$ , $j$ , $k$ are distinct, and there are $4 n n 1 n 2$ such terms in the sum of covariances.

Show that $S 2 W 2$ . Does this seem reasonable?

Show that $S 2 0$ as $n$ .

Use similar techniques to show that $M S 2 d 3 n$ . In particular, note that $M S 2 M W 2$ . Again, the sample mean and variance are uncorrelated if $d 3 0$ , and asymptotically uncorrelated otherwise.

Examples and Special Cases

Simulation Exercises

Many of the applets in this project are simulations of experiments with a basic random variable of interest. When you run the simulation, you are performing independent replications of the experiment. In most cases, the applet displays the standard deviation of the distribution, both numerically in a table and graphically as the radius of the blue, horizontal bar in the graph box. When you run the simulation, sample standard deviation is also displayed numerically in the table and graphically as the radius of the red horizontal bar in the graph box.

In the binomial coin experiment, the random variable is the number of heads. Run the simulation 1000 times updating every 10 runs and note the apparent convergence of the sample standard deviation to the distribution standard deviation.

In the simulation of the matching experiment, the random variable is the number of matches. Run the simulation 1000 times updating every 10 runs and note the apparent convergence of the sample standard deviation to the distribution standard deviation.

Run the simulation of the exponential experiment 1000 times with an update frequency of 10. Note the apparent convergence of the sample standard deviation to the distribution standard deviation.

Data Analysis Exercises

The sample mean and standard deviation are often computed in exploratory data analysis, as measures of the center and spread of the data, respectively.

Compute the sample mean and standard deviation for Michelson's velocity of light data.

Compute the sample mean and standard deviation for Cavendish's density of the earth data.

Compute the sample mean and standard deviation of the net weight in the M&M data.

Compute the sample mean and standard deviation of the petal length variable for the following cases in Fisher's iris data. Compare the results.

All cases
Setosa only
Versicolor only
Verginica only

Interval Data

Suppose that instead of the actual data, we have a frequency distribution with classes

A 1 A 2 A k

, class marks

x 1 x 2 x k

, and frequencies

n 1 n 2 n k

. Thus,

In this case, approximate values of the sample mean and variance are, respectively,

These approximations are based on the hope that the data values in each class are well represented by the class mark.

In the interactive histogram, select mean and standard deviation. Set the class width to 0.1 and construct a frequency distribution with at least 6 nonempty classes and at least 10 values. Compute the mean, variance, and standard deviation by hand, and verify that you get the same results as the applet.

In the interactive histogram, select mean and standard deviation. Set the class width to 0.1 and construct a distribution with at least 30 values of each of the types indicated below. Then increase the class width to each of the other four values. As you perform these operations, note the position and size of the mean ± standard deviation bar.

A uniform distribution.
A symmetric, unimodal distribution.
A unimodal distribution that is skewed right.
A unimodal distribution that is skewed left.
A symmetric bimodal distribution.
A $u$ -shaped distribution.

In the interactive histogram, construct a distribution that has the largest possible standard deviation.

Based on your answer to Exercise 28, characterize the distributions (on a fixed interval $a b$ ) that have the largest possible standard deviation.

3. The Sample Variance

Preliminaries