Introduction

As usual, our starting point is a random experiment with an underlying sample space and a probability measure

. In the basic statistical model, we have an observable random variable

X

taking values in a set

S

. In general,

X

can have quite a complicated structure. For example, if the experiment is to sample

n

objects from a population and record various measurements of interest, then

where

X i

is the vector of measurements for the

i

object. The most important special case occurs when

X 1 X 2 X n

are independent and identically distributed. In this case, we have a random sample of size

n

from the common distribution.

The purpose of this section is to define and discuss the basic concepts of statistical hypothesis testing. Collectively, these concepts are sometimes referred to as the Neyman-Pearson framework, in honor of Jerzy Neyman and Egon Pearson, who first formalized them.

General Hypothesis Tests

A statistical hypothesis is a statement about the distribution of the data variable

X

. Equivalently, a statistical hypothesis specifies a set of possible distributions of

X

(namely, the set of distributions for which the statement is true). In hypothesis testing, the goal is to see if there is sufficient statistical evidence to reject a presumed null hypothesis in favor of a conjectured alternative hypothesis. The null hypothesis is usually denoted

while the alternative hypothesis is usually denoted

. A hypothesis that specifies a single distribution for

X

is called simple; a hypothesis that specifies more than one distribution for

X

is called composite.

An hypothesis test is a statistical decision; the conclusion will either be to reject the null hypothesis in favor of the alternative, or to fail to reject the null hypothesis. The decision that we make must, of course, be based on the data vector

X

. Thus, we will find a subset

R

of the sample space

S

and reject

if and only if

X R

. The set

R

is known as the rejection region or the critical region. Note the asymmetry between the null and alternative hypotheses. This asymmetry is due to the fact that we assume the null hypothesis, in a sense, and then see if there is sufficient evidence in

X

to overturn this assumption in favor of the alternative.

Often, the critical region is defined in terms of a statistic

W X

, known as a test statistic. As usual, the use of a statistic allows data reduction when the dimension of the statistic is much smaller than the dimension of the data vector.

Errors

The ultimate decision may be correct or may be in error. There are two types of errors, depending on which of the hypotheses is actually true:

Similarly, there are two ways to make a correct decision: we could reject the null hypothesis when it is false or we could fail to reject the null hypothesis when it is true. The possibilities are summarized in the following table:

Hypothesis Test
State/Decision	Fail to reject	Reject
True	Correct	Type 1 error
False	Type 2 error	Correct

is true (that is, the distribution of

X

is specified by

), then

X R

is the probability of a type 1 error for this distribution. If

is composite, then

specifies a variety of different distributions for

X

and thus there is a set of type 1 error probabilities. The maximum probability of a type 1 error is known as the significance level of the test or the size of the critical region, which we will denote by

α

. Usually, the rejection region is constructed so that the significance level is a prescribed, small value (typically 0.1, 0.05, 0.01).

is true (that is, the distribution of

X

is specified by

). then

X R

is the probability of a type 2 error for this distribution. Again, if

is composite then

specifies a variety of different distributions for

X

. and thus there will be a set of type 2 error probabilities. Generally, there is a tradeoff between the type 1 and type 2 error probabilities. If we reduce the probability of a type 1 error, by making the rejection region

R

smaller, we necessarily increase the probability of a type 2 error because the complementary region

S R

is larger.

Power

is true (that is, the distribution of

X

is specified by

), then

X R

, the probability of rejecting

(and thus making a correct decision), is known as the power of the test for the distribution.

Suppose that we have two tests, corresponding to rejection regions

R 1

and

R 2

, respectively, each having significance level

α

. The test with region

R 1

is uniformly more powerful than the test with region

R 2

Naturally, in this case, we would prefer the first test. Often, however, two tests will not be uniformly ordered; one test will be more powerful for some distributions specified by

while the other test will be more powerful for other distributions specified by

. Finally, if a test has significance level

α

and is uniformly more powerful than any other test with significance level

α

. then the test is said to be a uniformly most powerful test at level

α

. Clearly, such a test is the best we can do.

P

-value

In most cases, we have a general procedure that allows us to construct a test (that is, a rejection region

R α

) for any given significance level

α 01

. Typically,

R α

decreases (in the subset sense) as

α

decreases. In this context, the

P

-value of the data variable

X

. denoted

P X

is defined to be the smallest

α

for which

X R α

; that is, the smallest significance level for which

is rejected, given

X

. Knowing

P X

allows us to test

at any significance level, for the given data: If

P X α

then we would reject

at significance level

α

; if

P X α

then we fail to reject

at significance level

α

. Note that

P X

is a statistic.

Tests of an Unknown Parameter

Hypothesis testing is a very general concept, but an important special class occurs when the distribution of the data variable

X

depends on a parameter

θ

. taking values in a parameter space

Θ

. The parameter may be vector-valued, so that

θ θ 1 θ 2 θ k

and

Θ k

for some

k

. The hypotheses generally take the form

where

Θ 0

is a prescribed subset of the parameter space

Θ

. In this setting, the probabilities of making an error or a correct decision depend on the true value of

θ

. If

R

is the rejection region, then the power function is given by

Show that

$Q θ$ is the probability of a type 1 error when $θ Θ o$
$Q θ : θ Θ 0$ is the significance level of the test.

Show that

$1 Q θ$ is the probability of a type 2 error when $θ Θ 0$ .
$Q θ$ is the power of the test when $θ Θ 0$ .

Suppose that we have two tests, corresponding to rejection regions

R 1

and

R 2

, respectively, each having significance level

α

. The test with rejection region

R 1

is uniformly more powerful than the test with rejection region

R 2

Most hypothesis tests of an unknown real parameter

θ

fall into three special cases:

where

θ 0

is a specified value. Case 1 is known as the two-sided test; case 2 is known as the left-tailed test, and case 3 is known as the right-tailed test (named after the conjectured alternative). There may be other unknown parameters besides

θ

(known as nuisance parameters).

Equivalence Between Hypothesis Test and Confidence Sets

There is an equivalence between hypothesis tests and confidence sets for a parameter

θ

Suppose that $C X$ is a $1 α$ level confidence set for $θ$ . Show that the test below has significance level $α$ for the hypothesis $θ θ 0$ versus $θ θ 0$ :

Reject if and only if θ 0 C X

equivalently, we fail to reject

at significance level

α

if and only if

θ 0

is in the corresponding

1 α

level confidence set.

In particular, show that this equivalence applies to interval estimates of a real parameter $θ$ and the common tests for $θ$ . In each case below, the confidence interval has confidence level $1 α$ and the test has significance level $α$

Suppose that $L X U X$ is a two-sided confidence interval for $θ$ . Reject $θ θ 0$ versus $θ θ 0$ if and only if $θ 0 L X$ or $θ 0 U X$
Suppose that $L X$ is a confidence lower bound for $θ$ . Reject $θ θ 0$ versus $θ θ 0$ if and only if $θ 0 L X$
Suppose that $U X$ is a confidence upper bound for $θ$ . Reject $θ θ 0$ versus $θ θ 0$ if and only if $θ 0 U X$

Pivot Variables and Test Statistics

Recall that confidence sets of an unknown parameter

θ

are often constructed through a pivot variable, that is, a random variable

W X θ

that depends on the data vector

X

and the parameter

θ

. but whose distribution does not depend on

θ

. In this case, a natural test statistic is

W X θ 0

1. Introduction

The Basic Statistical Model

General Hypothesis Tests

Errors

Power

$P$ -value

Tests of an Unknown Parameter

Equivalence Between Hypothesis Test and Confidence Sets

Pivot Variables and Test Statistics