\(\newcommand{\R}{\mathbb{R}}\) \(\newcommand{\N}{\mathbb{N}}\) \(\newcommand{\Z}{\mathbb{Z}}\) \(\newcommand{\P}{\mathbb{P}}\) \(\newcommand{\E}{\mathbb{E}}\) \(\newcommand{\var}{\text{var}}\) \(\newcommand{\sd}{\text{sd}}\) \(\newcommand{\bs}{\boldsymbol}\)
  1. Virtual Laboratories
  2. 8. Hypothesis Testing
  3. 1
  4. 2
  5. 3
  6. 4
  7. 5
  8. 6

5. Likelihood Ratio Tests

Preliminaries

As usual, our starting point is a random experiment with an underlying sample space, and a probability measure \(\P\). In the basic statistical model, we have an observable random variable \(\bs{X}\) taking values in a set \(S\). In general, \(\bs{X}\) can have quite a complicated structure. For example, if the experiment is to sample \(n\) objects from a population and record various measurements of interest, then

\[ \bs{X} = (X_1, X_2, \ldots, X_n) \]

where \(X_i\) is the vector of measurements for the \(i\)th object. The most important special case occurs when \((X_1, X_2, \ldots, X_n)\) are independent and identically distributed. In this case, we have a random sample of size \(n\) from the common distribution.

In the previous sections, we developed tests for parameters based on natural test statistics. However, in other cases, the tests may not be parametric, or there may not be an obvious statistic to start with. Thus, we need a more general method for constructing test statistics. Moreover, we do not yet know if the tests constructed so far are the best, in the sense of maximizing the power for the set of alternatives. In this and the next section, we investigate both of these ideas. Likelihood functions, similar to those used in maximum likelihood estimation, will play a key role.

Tests of Simple Hypotheses

Suppose that \(\bs{X}\) has one of two possible distributions. Our simple hypotheses are

\(H_0: \bs{X}\) has probability density function \(f_0\) versus \(H_1: \bs{X}\) has probability density function \(f_1\)

We will use subscripts on the probability measure \(\P\) to indicate the two hypotheses. The test that we will construct is based on the following simple idea: if we observe \(\bs{X} = \bs{x}\), then the condition \(f_1(\bs{x}) \gt f_0(\bs{x})\) is evidence in favor of the alternative; the opposite inequality is evidence against the alternative. Thus, let

\[ L(\bs{x}) = \frac{f_0(\bs{x})}{f_1(\bs{x})}, \quad \bs{x} \in S \]

The function \(L\) is the likelihood ratio function for the hypotheses and \(L(\bs{X})\) is the likelihood ratio statistic. Restating our earlier observation, note that small values of \(L\) are evidence in favor of \(H_1\). Thus it seems reasonable that the likelihood ratio statistic may be a good test statistic, and that we should consider tests of the following form, where \(k\) is a constant to be determined:

Reject \(H_0\) if and only if \(L \le k\)

The significance level of the test is \(\alpha = \P_0(L \le k)\).

As usual, we can try to construct a test by choosing \(k\) so that \(\alpha\) is a prescribed value. If \(\bs{X}\) has a discrete distribution, this will only be possible when \(\alpha\) is a value of the distribution function of \(L(\bs{X})\).

An important special case of this model occurs when the distribution of \(\bs{X}\) depends on a parameter \(\theta\) that has two possible values. Thus, the parameter space is \(\Theta = \{\theta_0, \theta_1\}\), and \(f_0\) denotes the probability density function of \(\bs{X}\) when \(\theta = \theta_0\) and \(f_1\) denotes the probability density function of \(\bs{X}\) when \(\theta = \theta_1\). In this case, the hypotheses are equivalent to

\(H_0: \theta = \theta_0\) versus \(H_1: \theta = \theta_1\)

The Neyman-Pearson Lemma

The following exercises establish the Neyman-Pearson Lemma, named for Jerzy Neyman and Egon Pearson. This result shows that the test given above is most powerful. Let

\[ R = \{\bs{x} \in S: L(\bs{x}) \le k\} \]

The following inequalities hold:

  1. \(\P_0(\bs{X} \in A) \le k \, \P_1(\bs{X} \in A)\) for \(A \subseteq R\)
  2. \(\P_0(\bs{X} \in A) \ge k \, \P_1(\bs{X} \in A)\) for \(A \subseteq R^c\)
Proof:

Use the definitions of \(L\) and \(R\)

If \(A \subseteq S\) then

\[ \P_1(\bs{X} \in R) - \P_1(\bs{X} \in A) \ge \frac{1}{k} [\P_0(\bs{X} \in R) - \P_0(\bs{X} \in A)] \]
Proof:

Write \(R = (R \cap A) \cup (R \setminus A)\) and \(A = (A \cap R) \cup (A \setminus R)\). Use the additivity of probability and the results in Exercise 2.

Recall that the size of a rejection region is the significance of the test with that rejection region.

Consider the tests with rejection regions \(R\) and \(A\). If the size of \(R\) is at least as large as the size of \(A\) then the test with rejection region \(R\) is more powerful than the test with rejection region \(A\):

\[ \P_0(\bs{X} \in R) \ge \P_0(\bs{X} \in A) \implies \P_1(\bs{X} \in R) \ge \P_1(\bs{X} \in A) \]
Proof:

Use Exercise 3

The Neyman-Pearson lemma is a beautiful result, and is more useful than might be first apparent. In many important cases, the same most powerful test works for a range of alternatives, and thus is a uniformly most powerful test for this range. In the following subsections, we will consider some of these special cases.

Tests for the Exponential Model

Suppose that \(\bs{X} = (X_1, X_2, \ldots, X_n)\) is a random sample from the exponential distribution with scale parameter \(b \gt 0\). The sample variables might represent the lifetimes from a sample of devices of a certain type. We are interested in testing the simple hypotheses \(H_0: b = b_0\) versus \(H_1: b = b_1\), where \(b_0 \gt 0\) and \(b_1 \gt 0\) are distinct specified values.

Recall that the sum of the variables is a sufficient statistic for \(b\):

\[ Y = \sum_{i=1}^n X_i \]

Recall also that \(Y\) has the gamma distribution with shape parameter \(n\) and scale parameter \(b\). For \(\alpha \gt 0\), we will denote the quantile of order \(\alpha\) for the this distribution by \(\gamma_{n, \, b}(\alpha)\).

The likelihood ratio statistic is

\[ L = \left(\frac{b_1}{b_0}\right)^n \exp\left[\left(\frac{1}{b_1} - \frac{1}{b_0}\right) Y \right] \]

The following tests are most powerful test at the \(\alpha\) level

  1. Suppose that \(b_1 \gt b_0\). Reject \(H_0: b = b_0\) versus \(H_1: b = b_1\) if and only if \(Y \ge \gamma_{n, \, b_0}(1 - \alpha)\).
  2. Suppose that \(b_1 \lt b_0\). Reject \(H_0: b = b_0\) versus \(H_1: b = b_1\) if and only if \(Y \le \gamma_{n, \, b_0}(\alpha)\).

Note that the tests in Exercise 6 do not depend on the value of \(b_1\). This fact, together with the monotonicity of the power function can be used to shows that the tests are uniformly most powerful for the usual one-sided tests.

  1. The test in Exercise 6 (a) is uniformly most powerful for the hypotheses \(H_0: b \le b_0\) versus \(H_1: b \gt b_0\).
  2. The test in Exercise 6 (b) is uniformly most powerful for the hypotheses \(H_0: b \ge b_0\) versus \(H_1: b \lt b_0\).

Tests for the Bernoulli Model

Suppose that \(\bs{X} = (X_1, X_2, \ldots, X_n)\) is a random sample of size \(n\) from the Bernoulli distribution with success parameter \(p\). The sample could represent the results of tossing a coin \(n\) times, where \(p\) is the probability of heads. We wish to test the simple hypotheses \(H_0: p = p_0\) versus \(H_1: p = p_1\), where \(p_0 \in (0, 1)\) and \(p_1 \in (0, 1)\) are distinct specified values. In the coin tossing model, we know that the probability of heads is either \(p_0\) or \(p_1\), but we don't know which.

Recall that the number of successes is a sufficient statistic for \(p\):

\[ Y = \sum_{i=1}^n X_i \]

Recall also that \(Y\) has the binomial distribution with parameters \(n\) and \(p\). For \(\alpha \in (0, 1)\), we will denote the quantile of order \(\alpha\) for the this distribution by \(b_{n, \, p}(\alpha)\); although since the distribution is discrete, only certain values of \(\alpha\) are possible.

The likelihood ratio statistic is

\[ L = \left(\frac{1 - p_0}{1 - p_1}\right)^n \left(\frac{p_0 (1 - p_1)}{p_1 (1 - p_0)}\right)^Y\]

The following tests are most powerful test at the \(\alpha\) level

  1. Suppose that \(p_1 \gt p_0\). Reject \(H_0: p = p_0\) versus \(H_1: p = p_1\) if and only if \(Y \ge b_{n, \, p_0}(1 - \alpha)\).
  2. Suppose that \(p_1 \lt p_0\). Reject \(p = p_0\) versus \(p = p_1\) if and only if \(Y \le b_{n, \, p_0}(\alpha)\).

Note that the tests in Exercise 9 do not depend on the value of \(p_1\). This fact, together with the monotonicity of the power function can be used to shows that the tests are uniformly most powerful for the usual one-sided tests.

  1. The test in Exercise 9 (a) is uniformly most powerful for the hypotheses \(H_0: p \le p_0\) versus \(H_1: p \gt p_0\).
  2. The test in Exercise 9 (b) is uniformly most powerful for the hypotheses \(H_0: p \ge p_0\) versus \(H_1: p \lt p_0\).

Uniformly Most Powerful Tests

The one-sided tests that we derived in the normal model, for \(\mu\) with \(\sigma\) known, for \(\mu\) with \(\sigma\) unknown, and for \(\sigma\) with \(\mu\) unknown are all uniformly most powerful. On the other hand, none of the two-sided tests are uniformly most powerful.

A Nonparametric Example

Suppose that \(\bs{X} = (X_1, X_2, \ldots, X_n)\) is a random sample, either from the Poisson distribution with parameter 1 or from the geometric distribution on \(\N\) with parameter \(p = \frac{1}{2}\). Thus, we wish to test the hypotheses

\[ \begin{align} H_0: & X \text{ has probability density function } f_0(x) = e^{-1} \frac{1}{x!}, \quad x \in \N \\ H_1: & X \text{ has probability density function } f_1(x) = \left(\frac{1}{2}\right)^{x+1}, \quad x \in \N \end{align} \]

The likelihood ratio statistic is

\[ L = 2^n e^{-n} \frac{2^Y}{U} \text{ where } Y = \sum_{i=1}^n X_i \text{ and } U = \prod_{i=1}^n X_i ! \]

The most powerful tests have the following form, where \(d\) is a constant: reject \(H_0\) if and only if \(\ln(2) \, Y - \ln(U) \le d\).

Generalized Likelihood Ratio

The likelihood ratio statistic can be generalized to composite hypotheses. Suppose again that the probability density function \(f_\theta\) of the data variable \(\bs{X}\) depends on a parameter \(\theta\), taking values in a parameter space \(\Theta\). Consider the hypotheses \(\theta \in \Theta_0\) versus \(\theta \notin \Theta_0\), where \(\Theta_0 \subseteq \Theta\). We define

\[ L(\bs{x}) = \frac{\max\{f_\theta(\bs{x}): \theta \in \Theta_0\}}{\max\{f_\theta(\bs{x}): \theta \in \Theta\}} \]

The function \(L\) is the likelihood ratio function and \(L(\bs{X})\) is the likelihood ratio statistic. By the same reasoning as before, small values of \(L(\bs{x})\) are evidence in favor of the alternative hypothesis.