Estimators

As usual, our starting point is a random experiment with an underlying sample space and a probability measure

. In the basic statistical model, we have an observable random variable

X

taking values in a set

S

. Recall that in general, this variable can have quite a complicated structure. For example, if the experiment is to sample

n

objects from a population and record various measurements of interest, then the data vector has the form

where

X i

is the vector of measurements for the

i

object. The most important special case is when

X 1 X 2 X n

are independent and identically distributed (IID). In this case

X

is a random sample of size

n

from the distribution of an underlying measurement variable

X

Statistics

Recall also that a statistic is an observable function of the outcome variable of the random experiment:

Thus, a statistic is simply a random variable derived from the observation variable

X

, with the assumption that

W

is also observable. As the notation indicates,

W

is typically also vector-valued.

Parameters

In the general sense, a parameter

θ

is a function of the distribution of

X

, taking values in a parameter space

Θ

. Typically, the distribution of

X

will have

k

real parameters of interest, so that

θ

has the form

and thus

Θ k

. In many cases, one or more of the parameters are unknown, and must be estimated from the data variable

X

. This is one of the of the most important and basic of all statistical problems, and is the subject of this chapter.

Estimators

Suppose now that we have an unknown real parameter

θ

taking values in a parameter space

Θ

. A real-valued statistic

W W X

that is used to estimate

θ

is called, appropriately enough, an estimator of

θ

. Thus, the estimator is a random variable and hence has a distribution, a mean, a variance, and so on. When we actually run the experiment and observe the data

x

, the observed value

w W x

(a single number) is the estimate of the parameter

θ

Basic Properties

The (random) error is the difference between the estimator and the parameter:

W θ

. The expected value of the error is known as the bias:

Use basic properties of expected value to show that $bias W W θ$

Thus, the estimator is said to be unbiased if the bias is 0 for all

θ Θ

, equivalently if the expected value of the estimator is the parameter being estimated:

W θ

for all

θ Θ

. The quality of the estimator is usually measured by computing the mean square error:

Use basic properties of expected value and variance to show that

MSE W W bias 2 W

In particular, if the estimator is unbiased, then the mean square error of

W

is simply the variance of

W

Ideally, we would like to have unbiased estimators with small mean square error. However, this is not always possible, and Exercise 2 shows the delicate relationship between bias and mean square error. In the next section we will see an example with two estimators of a parameter that are multiples of each other; one is unbiased, but the other has smaller mean square error. However, if we have two unbiased estimators of

θ

, denoted

U

and

V

, we naturally prefer the one with the smaller variance (mean square error). The relative efficiency of

V

U

is simply the ratio of the variances:

Asymptotic Properties

Often we have a general formula that defines an estimator of

θ

for any sample size

n

. Technically, this gives a sequence of real-valued estimators of

θ

In this case, we can discuss the asymptotic properties of the estimators as

n

. Most of the definitions are natural generalizations of the ones above. First, the sequence of estimators

W n

is said to be asymptotically unbiased if

Show that $W n$ is asymptotically unbiased if and only if $W n θ$ as $n$ for any $θ Θ$ .

Suppose now that

U n

and

V n

are two sequences of estimators that are asymptotically unbiased for

θ

. The asymptotic relative efficiency of

V n

U n

is the following limit, if it exists:

Naturally, we expect our estimators to improve, in some sense, as the sample size

n

increases. Specifically, the sequence of estimators

W n

is said to be consistent for

θ

W n θ

n

in probability.:

Suppose that $MSE W n 0$ as $n$ for any $θ Θ$ . Show that $W n$ is consistent for $θ$ . Hint: Use Markov's inequality.

The condition in Exercise 4 is known as mean-square consistency. Thus, mean-square consistency implies simple consistency. This is simply a statistical version of the theorem that states that mean-square convergence implies convergence in probability.

Estimation Problems

In the next several subsections, we will review several basic estimation problems that were studied in the chapter on Random Samples.

Estimating the Mean

Suppose that

X X 1 X 2 X n

is a random sample of size

n

from the distribution of a real-valued random variable

X

that has mean

μ

and standard deviation

σ

. A natural estimator of the distribution mean

μ

is the sample mean, defined by

Show or recall that

$M μ$ so $M$ is an unbiased estimator of $μ$ .
$M σ 2 n$ so $M$ is a consistent estimator of $μ$ .

In the sample mean experiment, set the sampling distribution to gamma. Increase the sample size with the scroll bar and note graphically and numerically the unbiased and consistent properties. Run the experiment 1000 times updating every 10.

Run the normal estimation experiment 1000 times, updating every 10 runs, for several values of the parameters. In each case, compare the empirical bias and mean square error of $M n$ with the theoretical values.

The consistency of the sample mean

M n

as an estimator of the distribution mean

μ

is simply the weak law of large numbers. Moreover, there are a number of important special cases of the results in Exercise 5. See the section on Sample Mean for the details.

In matching experiment, the random variable is the number of matches. Run the simulation 1000 times updating every 10 runs and note the apparent convergence of

the sample mean to the distribution mean.
the empirical density function to the distribution density function.

Estimating the Variance

As in the last subsection, suppose that

X X 1 X 2 X n

is a random sample of size

n

from the distribution of a real-valued random variable

X

that has mean

μ

and standard deviation

σ

. We will also assume that the fourth central moment

d 4 X μ 4

is finite.

μ

is known (usually an artificial assumption), then a natural estimator of

σ 2

is a special version of the sample variance, defined by

Show or recall that

$W 2 σ 2$ so $W 2$ is an unbiased estimator of $σ 2$
$W 2 1 n d 4 σ 4$ so $W 2$ is a consistent estimator of $σ 2$

μ

is unknown (the more reasonable assumption), then a natural estimator of the distribution variance is the standard version of the sample variance, defined by

Show or recall that

$S 2 σ 2$ so $S 2$ is an unbiased estimator of $σ 2$
$S 2 1 n d 4 n 3 n 1 σ 4$ so $S 2$ is a consistent estimator of $σ 2$

Run the exponential experiment 1000 times with an update frequency of 10. Note the apparent convergence of the sample standard deviation to the distribution standard deviation.

Show that

$W 2 S 2$ . Thus, $W 2$ is better than $S 2$ , assuming that $μ$ is known so that we can actually use $W 2$ .
The asymptotic relative efficiency of $S 2$ to $W 2$ is 1.

Run the normal estimation experiment 1000 times, updating every 10 runs, for several values of the parameters. In each case, compare the empirical bias and mean square error of $S 2$ and of $W 2$ to their theoretical values. Which estimator seems to work better?

The Poisson Distribution

For an example of the ideas in the last two subsections, suppose that

X

has the Poisson distribution with unknown parameter

a 0

. Then

X X a

, so that we could use either the sample mean

M

or the sample variance

S 2

as an estimator of

a

. Both are unbiased, so which is better? Naturally, we use mean square error as our criterion.

Show that

$X a$
$X 2 a 2 a$
$X 3 a 3 3 a 2 a$
$X 4 a 4 6 a 3 7 a 2 a$
$d 4 3 a 2 a$

Show that

$M a n$
$S 2 a n 1 2 a n n 1$
$M S 2$ , so the sample mean $M$ is a better estimator of the parameter $a$ than the sample variance $S 2$ .
The asymptotic relative efficiency of $M$ to $S 2$ is $1 2 a$

Run the Poisson experiment 100 times, updating every run, for several values of the parameter. In each case, compute the estimators $M$ and $S 2$ . Which estimator seems to work better?

Estimating the Covariance

Suppose that

X 1 Y 1 X 2 Y 2 X n Y n

is a random sample of size

n

from the distribution of

X Y

, where

X

is a real-valued random variable with mean

μ

and standard deviation

σ

, and where

Y

is a real-valued random variable with mean

ν

and standard deviation

τ

. Let

δ

denote the covariance of

X Y

. As usual, we will let

X X 1 X 2 X n

and

Y Y 1 Y 2 Y n

; these are random samples of size

n

from the distributions of

X

and

Y

, respectively.

μ

and

ν

are known (usually an artificial assumption), then a natural estimator of the distribution covariance

δ

is a special version of the sample covariance, defined by

Show or recall that

$W δ$ so $W$ is an unbiased estimator of $δ$ .
$W$ is consistent estimator of $δ$ .

μ

and

ν

are unknown (usually the more reasonable assumption), then a natural estimator of the distribution covariance

δ

is the usual version of the sample covariance, defined by

Show or recall that

$S δ$ so $S$ is an unbiased estimator of $δ$ .
$S$ is consistent estimator of $δ$ .

Chapter Topics

The estimators of the mean, variance, and covariance that we have considered in this section have been natural in a sense. However, for other parameters, it is not clear how to even find a reasonable estimator in the first place. In the next several sections, we will consider the problem of constructing estimators. Then we return to the study of the mathematical properties of estimators, and consider the question of when we can know that an estimator is the best possible, given the data.

1. Estimators

The Basic Statistical Model

Statistics

Parameters

Estimators

Basic Properties

Asymptotic Properties

Estimation Problems

Estimating the Mean

Estimating the Variance

The Poisson Distribution

Estimating the Covariance

Chapter Topics