The Method of Moments

Suppose that we have a basic random experiment with an observable, real-valued random variable

X

. The distribution of

X

has

k

unknown parameters, or equivalently, a parameter vector

θ θ 1 θ 2 θ k

taking values in a parameter space

Θ k

. As usual, we repeat the experiment

n

times to generate a random sample of size

n

from the distribution of

X

Thus,

X

is a sequence of independent random variables, each with the distribution of

X

. The method of moments is a technique for constructing estimators of the parameters that is based on matching the sample moments with the corresponding distribution moments. First, let

so that

μ i θ

is the

i

moment of

X

about 0. Note that we are emphasizing the dependence of these moments on the vector of parameters

θ

. Note also that

μ 1 θ

is just the mean of

X

, which we usually denote simply by

μ

. Next, let

so that

M i X

is the

i

sample moment about 0. Equivalently,

M i X

is the sample mean for the random sample

X 1 i X 2 i X n i

from the distribution of

X i

. Note that we are emphasizing the dependence of the sample moments on the sample

X

. Note also that

M 1 X

is just the ordinary sample mean, which we usually just denote by

M X

From our previous work, we know that

M i X

is an unbiased and consistent estimator of

μ i θ

for each

i

. Thus, to construct estimators

W 1 W 2 W k

for our parameters

θ 1 θ 2 θ k

respectively, we attempt to solve the set of simultaneous equations

for

W 1 W 2 W k

in terms of

X 1 X 2 X n

. Note that we have

k

equations in

k

unknowns, so there is hope that the equations can be solved.

Estimates for the Mean and Variance

Suppose that $X X 1 X 2 X n$ is a random sample of size $n$ from a distribution with unknown mean $μ$ and variance $σ 2$ . Show that the method of moments estimators for $μ$ and $σ 2$ are, respectively

M 1 n i 1 n X i, T 2 1 n i 1 n X i M 2

Of course,

M

is just the ordinary sample mean, but

T 2 n 1 n S 2

where

S 2

is the usual sample variance. In the remainder of this subsection, we will compare the estimators

S 2

and

T 2

. Recall that

S 2

is unbiased and consistent, with

S 2 1 n d 4 n 3 n 1 σ 4

Show that $bias T 2 σ 2 n$ . Thus, $T 2$ is negatively biased, and so on average underestimates $σ 2$ .

Show that $T 2$ is asymptotically unbiased.

Let $d 4 X μ 4$ denote the $4$ central moment. Show that

MSE T 2 n 1 2 n 3 d 4 n 3 n 1 σ 4 σ 4 n 2

Show that the asymptotic relative efficiency of $T 2$ to $S 2$ is 1.

Suppose that the sampling distribution is normal, so that $d 4 3 σ 4$ . Show that in this case

$MSE T 2 2 n 1 n 2 σ 4$
$MSE S 2 2 n 1 σ 4$
$MSE T 2 MSE S 2$ for $n 23$

Thus,

S 2

and

T 2

are multiplies of one another;

S 2

is unbiased, but at least when the sampling distribution is normal,

T 2

has smaller mean square error. Next, recall that under the (artificial) assumption that

μ

is known, a natural estimator of

σ 2

Moreover,

W 2

is unbiased and consistent, with

W 2 d 4 σ 4 n

. Amazingly, when the sampling distribution is normal,

T 2

has smaller mean square error even than

W 2

Suppose again that the sampling distribution is normal. Show that

$MSE W 2 2 σ 4 n$
$MSE T 2 MSE W 2$ for $n 23$

Run the normal estimation experiment 1000 times, updating every 10 runs, for several values of the sample size $n$ and the parameters $μ$ and $σ$ . Compare the empirical bias and mean square error of $S 2$ and of $T 2$ to their theoretical values. Which estimator is better in terms of bias? Which estimator is better in terms of mean square error?

There are several important one-parameter families of distributions for which the parameter is the mean, including the Bernoulli distribution with parameter

p

and the Poisson distribution with parameter

a

. For these families, the method of moments estimator of the parameter is the sample mean

M

. Similarly, the parameters of the normal distribution are the mean

μ

and the variance

σ 2

, so the method of moments estimators are

M

and

T 2

, respectively.

Additional Exercises

Suppose that $X X 1 X 2 X n$ is a random sample from the gamma distribution with shape parameter $k$ and scale parameter $b$ . Show that the method of moments estimators of $k$ and $b$ are respectively

U M 2 T 2, V T 2 M

Run the gamma estimation experiment 1000 times, updating every 10 runs for several different values of the sample size $n$ , the shape parameter $k$ and scale parameter $b$ . Note the empirical bias and mean square error of the estimators $U$ and $V$ .

Suppose that $X X 1 X 2 X n$ is a random sample of size $n$ from the beta distribution with left parameter $a$ and right parameter 1. Show that the method of moments estimator of $a$ is

U M 1 M

Run the beta estimation experiment 1000 times, updating every 10 runs, for several different values of the sample size $n$ and the parameter $a$ . Note the empirical bias and mean square error of the estimator $U$ .

Suppose that $X X 1 X 2 X n$ is a random sample of size $n$ from the Pareto distribution with shape parameter $a 1$ . Show that the method of moments estimator of $a$ is

U M M 1

Run the Pareto estimation experiment 1000 times, updating every 10 runs, for several different values of the sample size $n$ and the parameter $a$ . Note the empirical bias and mean square error of the estimator $U$ .

2. The Method of Moments

The Method

Estimates for the Mean and Variance

Additional Exercises