The Sample Mean and the Law of Large Numbers

Suppose that we have a basic random experiment and that

X

is a real-valued random variable for the basic experiment. Now suppose we perform

n

independent replications of the basic experiment. This defines a new, compound experiment with a sequence of independent random variables

X X 1 X 2 X n

, each with the same distribution as

X

. Recall that in statistical terms,

X

is a random sample of size

n

from the distribution of

X

. The sample mean is simply the average of the variables in the sample:

The sample mean is a function of the random sample and thus is a statistic. Like any statistic, the sample mean is itself a random variable with a distribution, mean, and variance of its own. Many times, the distribution mean is unknown and the sample mean is used as an estimator of this unknown parameter. When the underlying sample is understood, we will usually suppress it from the notation.

In the dice experiment, the dice scores form a random sample from the specified die distribution. Select the average random variable; this variable is the sample mean for the sample of dice scores. For each die distribution, start with 1 die and increase the sample size $n$ . Note the shape and location of the probability density function of the sample mean at each stage. For selected values of $n$ , run the simulation 1000 times with an update frequency of 10. Note the apparent convergence of the empirical density function to the true density function.

Moments

Use basic properties of expected value to show that $M μ$ .

Exercise 2 shows that the sample mean

M

is an unbiased estimator of the distribution mean

μ

. Therefore, the variance of

M

is the mean square error, when

M

is used as an estimator of

μ

Use basic properties of variance to show that $M σ 2 n$ .

From Exercise 3, the variance of the sample mean is an increasing function of the distribution variance and a decreasing function of the sample size. Both of these make intuitive sense if we think of the sample mean as an estimator of the distribution mean.

In the dice experiment, select the average random variable, which as noted before, is the sample mean for the sample of dice scores. For each die distribution, start with 1 die and increase the sample size $n$ . Note that the mean of the sample mean stays the same, but the standard deviation of the sample mean decreases (as we now know, in inverse proportion to the square root of the sample size). For selected values of $n$ , run the simulation 1000 times, updating every 10 runs. Note the apparent convergence of the empirical moments of the sample mean to the true moments.

Compute the sample mean of the petal width variable for the following cases in Fisher's iris data. Compare the results.

All cases
Setosa only
Versicolor only
Verginica only

Linearity

Suppose that $X$ and $Y$ are random samples of size $n$ (defined on the same probability space), and that $c$ is a constant. Show that

$M X Y M X M Y$
$M c X c M X$

The Law of Large Numbers

The law of large numbers states that the sample mean converges to the distribution mean as the sample size increases, and is one of the fundamental theorems of probability. There are different versions of the law, depending on the mode of convergence.

Suppose again that

X

is a real-valued random variable for our basic experiment, with mean

μ

and standard deviation

σ

(assumed finite). We repeat the basic experiment indefinitely to create a new, compound experiment with an infinite sequence of independent random variables

X 1 X 2

, each with the same distribution as

X

. In statistical terms, we are sampling from the distribution of

X

. For each

n

, let

M n

denote the sample mean of the first

n

sample variables:

Weak Laws

From Exercise 3, note that

M n M n μ 2 0

n

. This means that

M n μ

n

in mean square.

Use Chebyshev's inequality to show that for any $ε 0$ .

M n μ ε 0 as n

This means that

M n μ

n

in probability. Recall that in general, convergence in mean square implies convergence in probability. The convergence of the sample mean to the distribution mean in mean square and in probability are known as weak laws of large numbers.

The Strong Law

The strong law of large numbers states that the sample mean

M n

converges to the distribution mean

μ

with probability 1 . As the name suggests, this is a much stronger result than the weak law. That is, the strong law of large numbers states that

The following exercises sketch the proof of the strong law of large numbers. First, let

Y n i 1 n X i

so that

M n Y n n

Use Chebyshev's inequality to show that for every $n$ and every $ε 0$ ,

M n 2 μ ε σ 2 n 2 ε 2

Use the result of the previous exercise and the first Borel-Cantelli lemma to show that for every $ε 0$ ,

M n 2 μ ε for infinitely many n 0

Use the result of the previous exercise and Boole's inequality to show that

There exists a rational ε 0 such that M n 2 μ ε for infinitely many n 0

Conclude from the previous exercise that $M n 2 μ as n 1$

Suppose temporarily that the underlying sampling variable is nonnegative:

X 0 1

Show that with probability 1, $Y n$ is increasing in $n$ .

For

n

, let

k n

be the unique positive integer such that

k n 2 n k n 1 2

Use the result of the previous exercise and algebra to show that, with probability 1,

Y k n 2 k n 1 2 Y n n Y k n 1 2 k n 2

Use Exercise 11 and Exercise 13 and the squeeze theorem for limits to conclude the strong law of large numbers for a nonnegative variable: $M n μ as n 1$

We now relax the condition that the underlying sampling variable

X

be nonnegative, and we recall the definitions of the positive and negative parts a real number

x

Use Exercise 6 and Exercise 14 to show that with probability 1,

$1 n i 1 n X i X as n$
$1 n i 1 n X i X as n$

Finally conclude that $M n μ as n$ with probability 1.

The proof of the strong law of large numbers sketched above requires that the variance of the sampling distribution be finite (note that this is critical in Exercise 8 and Exercise 9). However, there are better proofs that only require that

X

. See, for example, Probability and Measure by Patrick Billingsley

Simulation Exercises

In the dice experiment, recall again that the dice scores form a random sample from the specified die distribution. Select the average random variable, which is the sample mean of the sample of dice scores. For each die distribution, start with 1 die and increase the sample size $n$ . Note how the distribution of the sample mean begins to resemble a point mass distribution. Run the simulation 1000 times, updating every 10 runs. Note the apparent convergence of the empirical density of the sample mean to the true density.

Many of the applets in this project are simulations of experiments with a basic random variable of interest. When you run the simulation, you are performing independent replications of the experiment. In most cases, the applet displays the mean of the distribution numerically in a table and graphically as the center of the blue horizontal bar in the graph box. When you run the simulation, sample mean is also displayed numerically in the table and graphically as the center of the red horizontal bar in the graph box.

In the simulation of the binomial coin experiment, select the number of heads. For selected values of the parameters, run the simulation 1000 times updating every 10 runs and note the apparent convergence of the sample mean to the distribution mean.

In the simulation of the matching experiment, the random variable is the number of matches. For selected values of the parameter, run the simulation 1000 times updating every 10 runs and note the apparent convergence of the sample mean to the distribution mean.

In the simulation of the gamma experiment, the random variable represents a random arrival time. For selected values of the parameters, run the experiment 1000 times with an update frequency of 10. Note the apparent convergence of the sample mean to the distribution mean.

Extensions and Special Cases

Random samples and their sample means are ubiquitous in probability and statistics. In this subsection, we will see how sample means can be used to estimate probabilities, density functions, and distribution functions.

Relative Frequency

Suppose that

X

is a random variable for a basic experiment, taking values in a space

S

. Note that

X

might be the outcome variable for the entire experiment, in which case

S

would be the sample space. In any event,

S

is a general space so

X

may be vector-valued. Recall that the distribution of

X

is the probability measure on

S

given by

Now suppose that we repeat the basic experiment

n

indefinitely to form a random sample

X 1 X 2 X n

of size

n

from the distribution of

X

. The empirical distribution of

X

for this sample is defined by

Although we are suppressing the dependence on the sample in our notation, note that for each

A S

P n A

is a statistic that gives the proportion of sample values in

A

Show that for fixed $A$ , $P n A$ is the sample mean from a random sample of size $n$ from the distribution of the indicator random variable $X A$ . Thus, conclude that

$P n A X A$
$P n A 1 n X A 1 X A$
$P n A X A$ as $n$ with probability 1.

This special case of the law of large numbers is central to the very concept of probability.

Show that for a fixed sample, $P n$ satisfies the axioms of a probability measure.

The empirical distribution of

X

, based on the random sample, is a random, discrete distribution, concentrated at the distinct sample values

X 1 X 2 X n

. Indeed, it places probability mass

1 n

X i

for each

i 12 n

, so that if the sample values are distinct, the empirical distribution is uniform on these sample values.

Several applets in this project are simulations of random experiments with events of interest. When you run the experiment, you are performing independent replications of the experiment. In most cases, the applet displays the relative frequency of the event and its complement, both graphically in blue, and numerically in a table. When you run the experiment, the relative frequencies are shown graphically in red and also numerically.

In the simulation of Buffon's coin experiment, the event of interest is that the coin crosses a crack. Run the experiment 1000 times with an update frequency of 10. Note the apparent convergence of the relative frequency of the event to the true probability.

In the simulation of Bertrand's experiment, the event of interest is that a random chord on a circle will be longer than the length of a side of the inscribed equilateral triangle. Run the experiment 1000 times with an update frequency of 10. Note the apparent convergence of the relative frequency of the event to the true probability.

The Empirical Distribution Function

Suppose now that

X

is a real-valued random variable for a basic experiment. Recall that the distribution function of

X

is the function

F

given by

Now suppose that we repeat the basic experiment

n

times to form a random sample

X 1 X 2 X n

of size

n

from the distribution of

X

. It is natural to define the empirical distribution function for this sample by

Although we are suppressing the dependence on the sample in our notation, note that for each

x

F n x

is a statistic that gives the proportion of the sample variables that are less than or equal to

x

Show that $F n$ is the distribution function of the empirical distribution of $X$ , based on the random sample $X 1 X 2 X n$ . In particular,

$F n$ increases from 0 to 1.
$F n$ is a step function with jumps at the distinct sample values $X 1 X 2 X n$ .

Show that for each $x$ , $F n x$ is the sample mean from a random sample of size $n$ from the distribution of the indicator variable $X x$ . Thus, conclude that

$F n x F x$
$F n x 1 n F x 1 F x$
$F n x F x$ as $n$ with probability 1.

Empirical Density for a Discrete Variable

Suppose now that

X

is a random variable for a basic experiment with a discrete distribution on a countable set

S

. Note that no assumptions are imposed on

S

, other than countability, so that in particular,

X

may be vector-valued. Recall that the probability density function of

X

is the function

f

given by

Now suppose that we repeat the basic experiment

n

times to form a random sample

X 1 X 2 X n

of size

n

from the distribution of

X

. It is natural to define the relative frequency function or empirical density function of

X

for this sample as follows:

Although we are suppressing the dependence on the sample in our notation, note that for each

x S

f n x

is a statistic that gives the proportion of the sample variables that have the value

x

Show that $f n$ is the probability density function of the empirical distribution of $X$ , based on the sample $X 1 X 2 X n$ . In particular,

$f n x 0$ for $x S$
$x x S f n x 1$

Show that for each $x$ , $f n x$ , is the sample mean from a random sample of size $n$ from the distribution of the indicator variable $X x$ . Thus, conclude that

$f n x f x$
$f n x 1 n f x 1 f x$
$f n x f x$ as $n$ with probability 1.

Show that if $X$ is real-valued, then the sample mean of the random sample $X 1 X 2 X n$ is the mean computed relative to the empirical density function. That is,

1 n i 1 n X i x x S x f n x

Many of the applets in this project are simulations of experiments which result in discrete variables. When you run the simulation, you are performing independent replications of the experiment. In most cases, the applet displays the true density function numerically in a table and visually as a blue bar graph. When you run the simulation, the relative frequency function is also shown numerically in the table and visually as a red bar graph.

In the poker experiment, the random variable is the type of hand. Run the simulation 1000 times updating every 10 runs and note the apparent convergence of the empirical density function to the true density function.

In the simulation of the binomial coin experiment, the random variable is the number of heads. Run the simulation 1000 times updating every 10 runs and note the apparent convergence of the empirical density function to the true density function.

In the simulation of the matching experiment, the random variable is the number of matches. Run the simulation 1000 times updating every 10 runs and note the apparent convergence of the empirical density function to the true density function.

Empirical Density for a Continuous Variable

In particular

1

is the length measure on

2

is the area measure on

2

, and

3

is the volume measure on

3

Suppose now that

X

is a random variable for a basic experiment, with a continuous distribution on

S k

, and that

X

has probability density function

f

. Technically,

f

is the probability density function with respect to

k

. Thus, by definition,

Again we repeat the basic experiment

n

times to form a random sample

X 1 X 2 X n

of size

n

from the distribution of

X

. Suppose now that

A j j J

is a partition of

S

into a countable number of subsets. As before, we can define the empirical probability of

A j

, based on the sample, by

Clearly the empirical density function depends on the partition, as well as the sample, but we suppress this to keep the notation from becoming completely unwieldy. Of course, for each

x

f n x

is a random variable (in fact, a statistic), but by the very definition of density, if the partition is sufficiently fine (so that

k A j

is small for each

j

), and if the sample size

n

is sufficiently large, then by the law of large numbers,

Show that the empirical density function $f n$ satisfies the mathematical properties of a probability density function for a continuous distribution on $S$ :

$f n x 0$ for $x S$
$x S f n x 1$

In fact, show that the empirical density function $f n$ corresponds to the distribution for which $P n A j$ is uniformly distributed over $A j$ for each $j J$

Many of the applets in this project are simulations of experiments which result in variables with continuous distributions. When you run the simulation, you are performing independent replications of the experiment. In most cases, the applet displays the true density visually as a blue graph. When you run the simulation, an empirical density function is also shown visually as a red bar graph.

In the simulation of the gamma experiment, vary the parameters and note the shape and location of the probability density function. For selected values of the parameters, run the experiment 1000 times with an update frequency of 10. Note the apparent convergence of the empirical density function to the true density function.

In the simulation of the random variable experiment, select the normal distribution. Run the experiment 1000 times with an update frequency of 10, and note the apartment convergence of the empirical density function to the true density function.

Exploratory Data Analysis

Many of the concepts discussed above are frequently used in exploratory data analysis. Specifically, suppose that

x

is a variable for a population (generally vector valued), and that

x 1 x 2 x n

are the observed data from a sample of size

n

, corresponding to this variable. For example,

x

might be encode the color counts and net weight for a bag of M&Ms. Now let

A j j J

be a partition of the data set, where

J

is a finite index set. The sets in the partition are generally known as classes. Just as above, we define the frequency of

A j

Finally, if

x

is a continuous variable, taking values in

k

, we define the density of

A j

The mapping that assigns frequencies to classes is known as a frequency distribution for the data set. The mapping that assigns relative frequencies to classes is known as a relative frequency distribution for the data set. Finally, in the case of a continuous variable, the mapping that assigns densities to classes is known as a density distribution for the data set. When

k 1

k 2

, the bar graph of any of these distributions is known as a histogram.

The whole purpose of constructing and graphing one of these empirical distributions is to summarize and display the data in a meaningful way. Thus, there are some general guidelines in choosing the classes:

In the interactive histogram, click on the $x$ -axis at various points to generate a data set with 20 values. Vary the class width over the five values from 0.1 to 5.0 and then back again. For each choice of class width, switch between the frequency histogram and the relative frequency histogram. Note how the shape of the histogram changes as you perform these operations.

It is important to realize that frequency data is inevitable for a continuous variable. For example, suppose that our variable represents the weight of a bag of M&Ms (in grams) and that our measuring device (a scale) is accurate to 0.01 grams. If we measure the weight of a bag as 50.32, then we are really saying that the weight is in the interval

50.315 50.324

(or perhaps some other interval, depending on how the measuring device works). Similarly, when two bags have the same measured weight, the apparent equality of the weights is really just an artifact of the imprecision of the measuring device; actually the two bags almost certainly do not have the exact same weight. Thus, two bags with the same measured weight really give us a frequency count of 2 for a certain interval.

Again, there is a tradeoff between the number of classes and the size of the classes; these determine the resolution of the empirical distribution. At one extreme, when the class size is smaller than the accuracy of the recorded data, each class contains a single datum or no datum. In this case, there is no loss of information and we can recover the original data set from the frequency distribution (except for the order in which the data values were obtained). On the other hand, it can be hard to discern the shape of the data when we have many classes with small frequency. At the other extreme is a frequency distribution with one class that contains all of the possible values of the data set. In this case, all information is lost, except the number of the values in the data set. Between these two extreme cases, an empirical distribution gives us partial information, but not complete information. These intermediate cases can organize the data in a useful way.

In the interactive histogram, set the class width to 0.1. click on the $x$ -axis to generate a data set with 10 distinct values and 20 values total.

From the frequency distribution, explicitly write down the 20 values in the data set.
Now increase the class width to 0.2, 0.5, 1.0, and 5.0. Note how the histogram loses resolution; that is, how the frequency distribution loses information about the original data set.

In Michelson's data, construct a frequency distribution for the velocity of light variable. Use 10 classes of equal width. Draw the histogram and describe the shape of the distribution.

In Cavendish's data, construct a relative frequency distribution for the density of the earth variable . Use 5 classes of equal width. Draw the histogram and describe the shape of the distribution.

In the M&M data, construct a frequency distribution and histogram for the total count variable and for the net weight variable.

In the Cicada data, construct a density distribution and histogram for the body weight variable for the cases given below . Note any differences.

All cases
Each species individually
Male and female individually.

In the interactive histogram, set the class width in the to 0.1 and click on the axis to generate a distribution of the given type with 30 points. Now increase the class width to each of the other four values and describe the type of distribution.

A uniform distribution
A symmetric unimodal distribution
A unimodal distribution that is skewed right.
A unimodal distribution that is skewed left.
A symmetric bimodal distribution
A $u$ -shaped distribution.

2. The Sample Mean and the Law of Large Numbers

The Sample Mean

Moments

Linearity

The Law of Large Numbers

Weak Laws

The Strong Law

Simulation Exercises

Extensions and Special Cases

Relative Frequency

The Empirical Distribution Function

Empirical Density for a Discrete Variable

Empirical Density for a Continuous Variable

Exploratory Data Analysis