Order Statistics

We perform

n

independent replications of the basic experiment to generate a random sample

X X 1 X 2 X n

of size

n

from the distribution of

X

. Recall that this is a sequence of independent random variables, each with the distribution of

X

Let

X n k X

denote the

k

smallest of element of the sample

X

. This statistics is called the order statistic of order

k

. Often the first step in a statistical study is to order the data; thus order statistics occur naturally. Our goal in this section is to study the distribution of the order statistics in terms of the sampling distribution. Note in particular that the extreme order statistics are the minimum and maximum values:

In the order statistic experiment, use the default settings and run the experiment a few times. Note the following:

The table on the left shows the values of the order statistics.
The graph on the left shows the density function of the sampling distribution in blue and the sample values in red.
The graph on the right shows the density function of the selected order statistic in blue and the empirical density function in red. The mean/standard deviation bar of the distribution is shown in blue while the empirical mean/standard deviation bar is shown in red.
The table on the right gives numerical values of the density function and moments and the empirical density function and moments.

Distributions

The Distribution of the

k

Order Statistic

Show that $N n y$ has the binomial distribution with parameters $n$ and $F y$ for each $y$ .

Show that $X n k y$ if and only if $N n y k$ for $y$ and $k 12 n$ .

Use the results of Exercises 2 and 3 to show that

G n k y j k n n j F y j 1 F y n j, y

In particular, show that $G n 1 y 1 1 F y n, y$ .

In particular, show that $G n n y F y n, y$ .

Suppose now that $X$ has a continuous distribution. Show that $X n k$ has a continuous distribution with probability density function

g n k y n k 1 1 n k F y k 1 1 F y n k f y, y

Hint: Differentiate the expression in Exercise 4 with respect to $y$ .

In the order statistic experiment, select the uniform distribution on $01$ and $n 5$ . Vary $k$ from 1 to 5 and note the shape of the density function of $X n k$ . For each value of $k$ , run the simulation 1000 times with and update frequency of 10. Note the apparent convergence of the empirical density function to the true density function.

There is a simple heuristic argument for the result in Exercise 7. First,

g n k y y

is the probability that

X n k

is in an infinitesimal interval of size

y

about

y

. On the other hand, this event means that one of sample variables is in the infinitesimal interval,

k 1

sample variables are less than

y

, and

n k

sample variables are greater than

y

. The number of ways of choosing these variables is the multinomial coefficient

By independence, the probability that the chosen variables are in the specified intervals is

Consider a random sample of size $n$ from the exponential distribution with rate parameter $r$ . Compute the probability density function of the $k$ order statistic $X n k$ . In particular, note that the minimum of the variables $X n 1$ has the exponential distribution with rate parameter $n r$ .

In the order statistic experiment, select the exponential (1) distribution and $n 5$ . Vary $k$ from 1 to 5 and note the shape of the probability density function of $X n k$ . For each value of $k$ , run the simulation 1000 times with and update frequency of 10. Note the apparent convergence of the empirical density function to the true density function.

Consider a random sample of size $n$ from the uniform distribution on the interval $01$ .

Show that $X n k$ has beta distribution with parameters $k$ and $n k 1$ .
Given the mean and variance of $X n k$ .

In the order statistic experiment, select the uniform distribution on $01$ and $n 6$ . Vary $k$ from 1 to 6 and note the size and location of the mean/standard deviation bar. For each value of $k$ , run the simulation 1000 times with and update frequency of 10. Note the apparent convergence of the empirical moments to the distribution moments.

Four fair dice are rolled. Find the probability density function of each of the order statistics.

In the dice experiment, select the following order statistic and die distribution. Increase the number of dice from 1 to 20, noting the shape of the probability density function at each stage. Now with $n 4$ , run the simulation 1000 times, updating every 10 runs. Note the apparent convergence of the relative frequency function to the density function.

Maximum score with fair dice.
Minimum score with fair dice.
Maximum score with ace-six flat dice.
Minimum score with ace-six flat dice.

Joint Distributions

Suppose that $j k$ . Use an heuristic argument to show that the joint density of $X n j X n k$ is

g n j k y z n j 1 1 k j 1 1 n k F y j 1 f y F z F y k j 1 f z 1 F z n k, y z

Similar arguments can be used to obtain the joint probability density function of any number of the order statistics. Of course, we are particularly interested in the joint probability density function of all of the order statistics; the following exercise gives this joint probability density function, which has a remarkably simple form.

Show that $X n 1 X n 2 X n n$ has joint probability density function given by

g n y 1 y 2 y n n f y 1 f y 2 f y n, y 1 y 2 y n

For each permutation $i i 1 i 2 i n$ of $12 n$ , let $S i x n x i 1 x i 2 x i n$ .
On $S i$ , the mapping $x 1 x 2 x n x i 1 x i 2 x i n$ is one-to-one, has continuous first partial derivatives, and has Jacobian 1.
The sets $S i$ where $i$ ranges over the $n$ permutations of $12 n$ are disjoint.
The probability that $X 1 X 2 X n$ is not in one of these sets is 0.
Now use the multivariate change of variables formula.

Again, there is a simple heuristic argument for the formula in Exercise 16. For each

y n

with

y 1 y 1 y n

, there are

n

permutations of the coordinates of

y

. The probability density of

X 1 X 2 X n

at each of the this points is

f y 1 f y 2 f y n

. Hence the probability density of

X n 1 X n 2 X n n

y

n

times this product.

Consider a random sample of size $n$ from the exponential distribution with rate parameter $r$ . Compute the joint probability density function of the order statistics $X n 1 X n 2 X n n$ .

Suppose that $X 1 X 2 X n$ is a random sample of size $n$ from the uniform distribution on the interval $a b$ , where $a b$ . Show that

$X 1 X 2 X n$ is uniformly distributed on $a b n$ .
$X n 1 X n 2 X n n$ is uniformly distributed on $x a b n a x 1 x 2 x n b$ .

Four fair dice are rolled. Find the joint probability density function of the order statistics.

Derived Statistics

Sample Range

This statistic gives a simple measure of the dispersion of the sample. Note the distribution of the sample range can be obtained from the joint distribution of

X n 1 X n n

given earlier.

Consider a random sample of size $n$ from the exponential distribution with rate parameter $r$ . Show that the sample range $R$ has the same distribution as the maximum of a random sample of size $n 1$ from this exponential distribution.

Consider a random sample of size $n$ from the uniform distribution on $01$ .

Show that $R$ has the beta distribution with left parameter $n 1$ and right parameter 2.
Give the mean and variance of $R$ .
What happens as $n$ ?

Four fair dice are rolled. Find the probability density function of the sample range.

The Sample Median

n

is odd, the sample median is the middle of the ordered observations, namely

n

is even, there is not a single middle observation, but rather two middle observations. Thus, the median interval is

In this case, the sample median is defined to be the midpoint of the median interval

In a sense, this definition is a bit arbitrary because there is no compelling reason to prefer one point in the median interval over another. For more on this issue, see the discussion of error functions in the section on Variance. In any event, sample median is a natural statistic that is analogous to the median of the distribution. Moreover, the distribution of the sample median can be obtained from our results on order statistics.

Sample Quantiles

We can generalize the sample median discussed above to other sample quantiles. Suppose that

p 01

. Let

k n 1 p

, the integer part of

n 1 p

, and let

q n 1 p k

, the fractional part of

n 1 p

. Using linear interpolation, we define the sample quantile of order

p

to be

Once again, the sample quantile of order

p

is a natural statistic that is analogous to the distribution quantile of order

p

. Moreover, the distribution of a sample quantile can be obtained from our results on order statistics.

The sample quantile of order

1 4

is known as the first sample quartile and is frequently denoted

Q 1

. The the sample quantile of order

3 4

is known as the third sample quartile and is frequently denoted

Q 3

. Note that the sample median is the quartile of order

1 2

and is sometimes denoted

Q 2

. The interquartile range is defined to be

The IQR is a statistic that measures the spread of the distribution about the median, but of course this number gives less information than the interval

Q 1 Q 3

Exploratory Data Analysis

The five statistics

X n 1 Q 1 Q 2 Q 3 X n n

are often referred to as the five-number summary. Together, these statistics give a great deal of information about the distribution in terms of the center, spread, and skewness. Graphically, the five numbers are often displayed as a boxplot, which consists of a line extending from the minimum

X n 1

to the maximum

X n n

, with a rectangular box from the first quartile

Q 1

to the third quartile

Q 3

and tick marks at the minimum, the median

Q 2

, and the maximum.

In the interactive histogram, select boxplot. Construct a frequency distribution with at least 6 classes and at least 10 values. Compute the statistics in the five-number summary by hand and verify that you get the same results as the applet.

In the interactive histogram, select boxplot. Set the class width to 0.1 and construct a distribution with at least 30 values of each of the types indicated below. Then increase the class width to each of the other four values. As you perform these operations, note the shape of the boxplot and the relative positions of the statistics in the five-number summary:

A uniform distribution.
A symmetric, unimodal distribution.
A unimodal distribution that is skewed right.
A unimodal distribution that is skewed left.
A symmetric bimodal distribution.
A $u$ -shaped distribution.

In the interactive histogram, select boxplot. Start with a distribution and add additional points as follows. Note the effect on the boxplot:

Add a point below $X n 1$ .
Add a point between $X n 1$ and $Q 1$ .
Add a point between $Q 1$ and $Q 2$ .
Add a point between $Q 2$ and $Q 3$ .
Add a point between $Q 3$ and $X n n$ .
Add a point above $X n n$ .

In the last problem, you may have noticed that when you add an additional point to the distribution, one or more of the five statistics does not change. In general, quantiles can be relatively insensitive to changes in the data.

Compute the five number summary and sketch the boxplot for the velocity of light variable in Michelson's data. Compare the median with the true value of the velocity of light.

Compute the five number summary and sketch the boxplot for the density of the earth variable in Cavendish's data. Compare the median with the true value of the density of the earth.

Compute the five number summary and sketch the boxplot for the net weight variable in the M&M data.

Compute the five number summary for the sepal length variable in Fisher's iris data, using the cases indicated below. Plot the boxplots on parallel axes, so you can compare.

All cases
Type Setosa only
Type Verginica only
Type Versicolor only

6. Order Statistics

Definitions