Likelihood Ratio Tests

As usual, our starting point is a random experiment with an underlying sample space, and a probability measure

. In the basic statistical model, we have an observable random variable

X

taking values in a set

S

. In general,

X

can have quite a complicated structure. For example, if the experiment is to sample

n

objects from a population and record various measurements of interest, then

where

X i

is the vector of measurements for the

i

object. The most important special case occurs when

X 1 X 2 X n

are independent and identically distributed. In this case, we have a random sample of size

n

from the common distribution.

In the previous sections, we developed tests for parameters based on natural test statistics. However, in other cases, the tests may not be parametric, or there may not be an obvious statistic to start with. Thus, we need a more general method for constructing test statistics. Moreover, we do not yet know if the tests constructed so far are the best, in the sense of maximizing the power for the set of alternatives. In this and the next section, we investigate both of these ideas. Likelihood functions, similar to those used in maximum likelihood estimation, will play a key role.

Tests of Simple Hypotheses

Suppose that

X

has one of two possible distributions. Our simple hypotheses are

We will use subscripts on the probability measure

to indicate the two hypotheses. The test that we will construct is based on the following simple idea: if we observe

X x

, then the condition

f 1 x f 0 x

is evidence in favor of the alternative; the opposite inequality is evidence against the alternative. Thus, let

The function

L

is the likelihood ratio function for the hypotheses and

L X

is the likelihood ratio statistic. Restating our earlier observation, note that small values of

L

are evidence in favor of

. Thus it seems reasonable that the likelihood ratio statistic may be a good test statistic, and that we should consider tests of the following form, where

k

is a constant:

Show that the significance level of the test is $α 0 L k$

As usual, we can try to construct a test by choosing

k

so that

α

is a prescribed value. If

X

has a discrete distribution, this will only be possible when

α

is a value of the distribution function of

L X

An important special case of this model occurs when the distribution of

X

depends on a parameter

θ

that has two possible values. Thus, the parameter space is

Θ θ 0 θ 1

, and

f 0

denotes the probability density function of

X

when

θ θ 0

and

f 1

denotes the probability density function of

X

when

θ θ 1

. In this case, the hypotheses are equivalent to

The Neyman-Pearson Lemma

The following exercises establish the Neyman-Pearson Lemma, named for Jerzy Neyman and Egon Pearson. This result shows that the test given above is most powerful. Let

Use the definitions of $L$ and $R$ to show that

$0 X A k 1 X A$ for $A R$
$0 X A k 1 X A$ for $A R c$

Show that if $A S$ then

1 X R 1 X A 1 k 0 X R 0 X A

Hint: Write $R R A R A$ and $A A R A R$ . Use the additivity of probability and the results in Exercise 2.

Consider the tests with rejection regions $R$ and $A$ . Recall that the size of a rejection region is the significance of the test with that rejection region. Use Exercise 3 to show that if the size of $R$ is at least as large as the size of $A$ then the test with rejection region $R$ is more powerful than the test with rejection region $A$ :

0 X R 0 X A 1 X R 1 X A

The Neyman-Pearson lemma is a beautiful result, and is more useful than might be first apparent. In many important cases, the same most powerful test works for a range of alternatives, and thus is a uniformly most powerful test for this range. In the following subsections, we will consider some of these special cases.

Tests for the Exponential Model

Suppose that

X X 1 X 2 X n

is a random sample from the exponential distribution with scale parameter

b

. The sample variables might represent the lifetimes from a sample of devices of a certain type. We are interested in testing the simple hypotheses

b b 0

versus

b b 1

, where

b 0 0

and

b 1 0

are distinct specified values.

Recall also that

Y

has the gamma distribution with shape parameter

n

and scale parameter

b

. For

α 01

, we will denote the quantile of order

α

for the this distribution by

γ n b α

Show that the likelihood ratio statistic is

L b 1 b 0 n 1 b 1 1 b 0 Y

Show that the following tests are most powerful test at the $α$ level

Suppose that $b 1 b 0$ . Reject $b b 0$ versus $b b 1$ if and only if $Y γ n b 0 1 α$ .
Suppose that $b 1 b 0$ . Reject $b b 0$ versus $b b 1$ if and only if $Y γ n b 0 α$ .

Note that the tests in Exercise 6 do not depend on the value of

b 1

. This fact, together with the monotonicity of the power function can be used to shows that the tests are uniformly most powerful for the usual one-sided tests.

Show that

The test in Exercise 6 (a) is uniformly most powerful for the hypotheses $b b 0$ versus $b b 0$
The test in Exercise 6 (b) is uniformly most powerful for the hypotheses $b b 0$ versus $b b 0$

Tests for the Bernoulli Model

Suppose that

X X 1 X 2 X n

is a random sample of size

n

from the Bernoulli distribution with success parameter

p

. The sample could represent the results of tossing a coin

n

times, where

p

is the probability of heads. We wish to test the simple hypotheses

p p 0

versus

p p 1

, where

p 0 01

and

p 1 01

are distinct specified values. In the coin tossing model, we know that the probability of heads is either

p 0

p 1

, but we don't know which.

Recall also that

Y

has the binomial distribution with parameters

n

and

p

. For

α 01

, we will denote the quantile of order

α

for the this distribution by

b n p α

; although since the distribution is discrete, only certain values of

α

are possible.

Show that the likelihood ratio statistic is

L 1 p 0 1 p 1 n p 0 1 p 1 p 1 1 p 0 Y

Show that the following tests are most powerful test at the $α$ level

Suppose that $p 1 p 0$ . Reject $p p 0$ versus $p p 1$ if and only if $Y b n p 0 1 α$ .
Suppose that $p 1 p 0$ . Reject $p p 0$ versus $p p 1$ if and only if $Y b n p 0 α$ .

Note that the tests in Exercise 9 do not depend on the value of

p 1

. This fact, together with the monotonicity of the power function can be used to shows that the tests are uniformly most powerful for the usual one-sided tests.

Show that

The test in Exercise 9 (a) is uniformly most powerful for the hypotheses $p p 0$ versus $p p 0$
The test in Exercise 9 (b) is uniformly most powerful for the hypotheses $p p 0$ versus $p p 0$

Uniformly Most Powerful Tests

A Nonparametric Example

Suppose that

X X 1 X 2 X n

is a random sample, either from the Poisson distribution with parameter 1 or from the geometric distribution on

with parameter

1 2

. Thus, we wish to test the hypotheses

Show that the likelihood ratio statistic is

L 2 n n 2 Y U where Y i 1 n X i and U i 1 n X i

Show that the most powerful tests have the following form, where $d$ is a constant: reject if and only if $2 Y U d$

Generalized Likelihood Ratio

The likelihood ratio statistic can be generalized to composite hypotheses. Suppose again that the probability density function

f θ

of the data variable

X

depends on a parameter

θ

, taking values in a parameter space

Θ

. Consider the hypotheses

θ Θ 0

versus

θ Θ 0

, where

Θ 0 Θ

. We define

The function

L

is the likelihood ratio function and

L X

is the likelihood ratio statistic. By the same reasoning as before, small values of

L x

are evidence in favor of the alternative hypothesis.

5. Likelihood Ratio Tests

Preliminaries

Tests of Simple Hypotheses

The Neyman-Pearson Lemma

Tests for the Exponential Model

Tests for the Bernoulli Model

Uniformly Most Powerful Tests

A Nonparametric Example

Generalized Likelihood Ratio