Convergence in Distribution

Suppose that

X n, n

and

X

are real-valued random variables with distribution functions

F n, n

and

F

, respectively. We say that the distribution of

X n

converges to the distribution of

X

n

for all

x

at which

F

is continuous. The first fact to notice is that convergence in distribution, as the name suggests, only involves the distributions of the random variables. Thus, the random variables need not even be defined on the same probability space (that is, they need not be defined for the same random experiment). This is in sharp contrast to the other modes of convergence we have studied:

We will show, in fact, that convergence in distribution is the weakest of all of these modes of convergence. It is nonetheless very important. The central limit theorem, one of the two fundamental theorems of probability, is a theorem about convergence in distribution.

Preliminary Examples

The examples below show why the definition is given in terms of distribution functions, rather than density functions, and why convergence is only required at the points of continuity of the limiting distribution function.

Let $X n 1 n$ for $n$ and let $X 0$ . Let $f n$ and $f$ be the corresponding density functions and let $F n$ and $F$ be the corresponding distribution functions. Show that

$f n x 0$ as $n$ for all $x$
$F n x 0 x 0 1 x 0$ as $n$
$F n x F x$ as $n$ for all $x 0$

Suppose that $X n$ has the discrete uniform distribution on $1 n 2 n n 1 n 1$ for each $n$ . Let $X$ have the continuous uniform distribution on the interval $01$

Show that the distribution of $X n$ converges to the distribution of $X$ as $n$ .
Recall that denotes the set of rational numbers. Show that $X n 1$ for each $n$ but $X 0$ .
Let $f n$ denote the (discrete) probability density function of $X n$ . Show that $f n x 0$ as $n$ for all $x 01$

Probability Density Functions

As Exercise 2 shows, it is quite possible to have a sequence of discrete distributions converge to a continuous distribution (or the other way around). Recall that probability density functions have very different meanings in the discrete and continuous cases: density with respect to counting measure in the first case, and density with respect to Lebesgue measure in the second case. This is another indication that distribution functions, rather than density functions, are the correct objects of study. However, if probability density functions of a fixed type converge then the distributions converge. The following results are a consequence of Scheffe's theorem, which is given in advanced topics below.

Suppose that

f n, n

and

f

are probability density functions for discrete distributions on a countable set

S

, and that

f n x f x

n

for each

x S

. Then the distribution defined by

f n

converges to the distribution defined by

f

n

. Similarly, suppose that

f n, n

and

f

are probability density functions for continuous distributions on

, and that

f n x f x

n

for all

x

(except perhaps on a set with Lebesgue measure 0). Then the distribution defined by

f n

converges to the distribution defined by

f

n

Relation to Convergence in Probability

Suppose that $X 1 X 2$ and $X$ are random variables (defined on the same probability space) with distribution functions $F 1 F 2$ and $F$ , respectively. Show that if $X n X$ as $n$ in probability, then the distribution of $X n$ converges to the distribution of $X$ as $n$ .

Show that $X n x X n x X x ε X n x X x ε$ for $ε 0$ .
Conclude that $F n x F x ε X n X ε$ for $ε 0$ ,
Show that $X x ε X x ε X n x X x ε X n x$ for $ε 0$ .
Conclude that $F x ε F n x X n X ε$ for $ε 0$ .
Combine (b) and (d) to conclude that $F x ε X n X ε F n x F x ε X n X ε$ for $ε 0$ .
Let $n$ in (e) to show that $F x ε n F n x n F n x F x ε$ for $ε 0$ .
Let $ε 0$ in (f) to show that if $F$ is continuous at $x$ then $F n x F x as n$

Our next example shows that even when the variables are defined on the same probability space, a sequence can converge in distribution, but not in any other way.

Let $X$ be an indicator variable with $X 1 X 0 1 2$ , and let $X n X$ for $n$ . Show that

$1 X$ has the same distribution as $X$ .
The distribution of $X n$ converges to the distribution of $1 X$ as $n$ .
$X n 1 X 1$ for any $n$
$X n does not converge to 1 X as n 1$
$X n 1 X 1 2 1$ for each $n$ so $X n$ does not converge to $1 X$ as $n$ in probability.
$X n 1 X 1$ for each $n 12$ so $X n$ does not converge to $1 X$ as $n$ in mean.

To summarize, we have the following implications for the various modes of convergence; no other implications hold in general.

However, the following exercise gives an important converse to the last implication in the summary above, when the limiting variable is a constant. Of course, a constant can be viewed as a random variable defined on any probability space.

Suppose that $X 1 X 2$ is a sequence of random variables (defined on the same probability space) and that the distribution of $X n$ converges to the distribution of the constant $c$ as $n$ . Show that $X n c$ as $n$ in probability:

$X n x 0 x c 1 x c$ as $n$
$X n c ε 1$ as $n$ for any $ε 0$

Examples and Applications

There are several important cases where a special distribution converges to another special distribution as a parameter approaches a limiting value. Indeed, such convergence results are part of the reason why such distributions are special in the first place.

Convergence of the Hypergeometric Distribution to the Binomial

Recall that the hypergeometric distribution with parameters

m

r

, and

n

is the distribution that governs the number of type 1 objects in a sample of size

n

, drawn without replacement from a population of

m

objects with

r

of type 1. It has discrete probability density function

Suppose that $r m$ depends on $m$ , and that $r m m p$ as $m$ . Show that for fixed $n$ , the hypergeometric distribution with parameters $m$ , $r m$ , and $n$ converges to the binomial distribution with parameters $n$ and $p$ as $, m$ .

From a practical point of view, the result in the last exercise means that if the population size

m

is large compared to sample size

n

, then the hypergeometric distribution with parameters

m

r

, and

n

(which corresponds to sampling without replacement) is well approximated by the binomial distribution with parameters

n

and

p r m

(which corresponds to sampling with replacement). This is often a useful result, because the binomial distribution has fewer parameters than the hypergeometric distribution (and often in real problems, the parameters may only be known approximately). Specifically, in the limiting binomial distribution, we do not need to know the population size

m

and the number of type 1 objects

r

individually, but only in the ratio

r m

In the ball and urn experiment, set $m 100$ and $r 30$ For each of the following values of $n$ (the sample size), switch between sampling without replacement (the hypergeometric distribution) and sampling with replacement (the binomial distribution). Note the difference in the density function. Run the simulation 1000 times with an update frequency of 10 for each sampling mode.

Convergence of the Binomial Distribution to the Poisson

Recall that the binomial distribution with parameters

n

and

p 01

is the distribution of the number successes in

n

Bernoulli trials, when

p

is the probability of success on a trial. This distribution has probability density function

Recall also that the Poisson distribution with rate parameter

r 0

has probability density function;

Suppose now that the success parameter $p$ in the binomial distribution depends on the trial parameter $n$ and that $n p n r$ as $n$ where $r 0$ . Show that this binomial distribution converges to the Poisson distribution with parameter $r$ as $n$ .

From a practical point of view, the result in the last exercise means that if the number of trials

n

is large and the probability of success

p

small, so that the product

r n p

is of moderate size, then the binomial distribution with parameters

n

and

p

is well approximated by the Poisson distribution with parameter

r

. This is often a useful result, because the Poisson distribution has fewer parameters than the binomial distribution (and often in real problems, the parameters may only be known approximately). Specifically, in the limiting Poisson distribution, we do not need to know the number of trials

n

and the probability of success

p

individually, but only in the product

n p

The Geometric Distribution

Suppose that $U$ is a random variable with probability density function $U k p 1 p k 1, k$ , where $p 01$ is a parameter. Thus, $U$ has the geometric distribution on with parameter $p$ . Random variable $U$ can be interpreted as the trial number of the first success in a sequence of Bernoulli trials.

Find the conditional density function of $U$ given $U n$ .
Show that the distribution in (a) converges to the uniform distribution on $12 n$ as $p 0$ .

Suppose that $U n$ has the geometric distribution on with success parameter $p n$ . Moreover, suppose that $n p n r$ as $n$ where $r 0$ . Show that the distribution of $U n n$ converges to the exponential distribution with parameter $r$ as $n$ .

Convergence of the Matching Distribution to the Poisson

Consider a random permutation

X 1 X 2 X n

of the elements in the set

12 n

. We say that a match occurs at position

i

X i i

Show that $X i i 1 n$ for each $i 12 n$ . Thus, the matching events all have the same probability, which varies inversely with the number of trials.

Show, however, that $X i i X j j 1 n n 1$ for each $i 12 n$ , $j 12 n$ with $i j$ . Thus, the matching events are dependent, and in fact are positively correlated. In particular, the matching events do not form a sequence of Bernoulli trials.

The matching problem is studied in detail in the chapter on Finite Sampling Models. In particular, the number of matches

N

has the following density function

Show that the distribution of $N$ converges to the Poisson distribution with parameter 1 as $n$ .

The Extreme Value Distribution

Suppose that

X 1 X 2

is a sequence of independent random variables, each with the standard exponential distribution. Thus, the common distribution function is

Find the distribution function of

$V n X 1 X 2 X n$
$Y n V n n$

Show that the distribution of $Y n$ converges to the distribution with the following distribution function as $n$ :

F x x

The limiting distribution in the last exercise is the type 1 extreme value distribution, also known as the Gumbel distribution in honor of Emil Gumbel. Extreme value distributions are studied in detail in the chapter on Special Distributions.

The Pareto Distribution

Suppose that $X n$ takes values in $1$ for $n$ with distribution function $F n x 1 1 x n, x 1$ . Thus, $X n$ has the Pareto distribution with shape parameter $n$ , named for Vilfredo Pareto. The Pareto distribution is studied in more detail in the chapter on Special Distributions.

Find the limiting distribution of $X n$ as $n$ .
Find the limiting distribution of $Y n n X n n$ as $n$ .

Fundamental Theorems

The two fundamental theorems of basic probability theory, the law of large numbers and the central limit theorem, are studied in detail in the chapter on Random Samples. For this reason we will simply state the results in this section.

Suppose that

X 1 X 2

is a sequence of independent, identically distributed, real-valued random variables (defined on the same probability space) with mean

μ

and standard deviation

σ

. Let

denote the sum of the first

n

variables. A weak version of the law of large numbers states that the distribution of the average

M n 1 n Y n

converges to the point mass distribution at

μ

n

. From Exercise 5, the convergence is also in probability. In fact the convergence is with probability 1 (much stronger), assuming that

σ

is finite. The central limit theorem states that the distribution of the standard score

Advanced Topics

The Skorohod Representation

Suppose that

F 1 F 2

and

F

are distribution functions, and that

F n F

n

in the sense of convergence of distribution. In this subsection we will prove the Skorohod representation theorem: there exist random variables

X 1 X 2

and

X

(defined on the same probability space) such that

Prove the Skorohod representation theorem using the following steps:

Let $U$ be uniformly distributed on the interval $01$ .
Define $X n F n U$ and $X F U$ where $F n$ and $F$ are the quantile functions of $F n$ and $F$ respectively.
Recall that $X n$ has distribution function $F n$ for each $n$ , and $X$ has distribution function $F$ .
Let $ε 0$ and let $u 01$ . Pick a continuity point $x$ of $F$ such that $F u ε x F u$ .
Show that $F x u$ .
Show that $F n x u$ for $n$ sufficiently large.
Conclude that $F u ε x F n u$ for $n$ sufficiently large.
Let $n$ and $ε 0$ to conclude that $F u n F n u$ for any $u 01$ .
Next, let $v$ satisfy $0 u v 1$ and let $ε 0$ . Pick a continuity point $x$ of $F$ such that $F v x F v ε$
Show that $u v F x$ .
Show that $u F n x$ for $n$ sufficiently large.
conclude that $F n u x F v ε$ for $n$ sufficiently large,
Let $n$ and $ε 0$ to conclude that $n F n u F v$ for any $u$ , $v$ with $0 u v 1$ .
Let $v u$ to show that $n F n u F u$ if $u$ is a point of continuity of $F$ .
Conclude that $F n u F u as n$ if $u$ is a point of continuity of $F$ .
Recall from analysis that since $F$ is increasing, the set $D 01$ of discontinuities of $F$ is countable.
Conclude that $U D 0$
Conclude that $X n X as n 1$

The following important result illustrates the value of the Skorohod representation.

Suppose that $X 1 X 2$ and $X$ are real-valued random variables such that the distribution of $X n$ converges to the distribution of $X$ as $n$ . If $g$ is a continuous function from into , then the distribution of $g X n$ converges to the distribution of $g X$ as $n$ .

Let $Y n, n$ and $Y$ be random variables, defined on the same probability space, such that $Y n$ has the same distribution as $X n$ for each $n$ , $Y$ has the same distribution as $X$ , and $Y n Y$ as $n$ with probability 1.
Argue that $g Y n g Y$ as $n$ with probability 1.
Argue that the distribution of $g Y n$ converges to the distribution of $g Y$ as $n$ .
Argue that $g Y n$ has the same distribution as $g X n$ and that $g Y$ has the same distribution as $g X$ .

Scheffé's Theorem

Suppose that $f n$ is a probability density function for a continuous distribution $n$ on for each $n$ , and that $f$ is a probability density function for a continuous distribution on . Suppose that $f n x f x$ as $n$ for all $x$ , except perhaps on a set of Lebesgue measure 0. Then $n A A$ as $n$ uniformly in (measurable) $A$ .

Use basic properties of integrals to show that $A n A x f x f n x$
Let $g n f f n$ , and let $g n$ denote the positive part of $g n$ and $g n$ the negative part of $g n$ . Show that $g n f$ .
Show that $g n x 0$ as $n$ for all $x$ (except perhaps on a set of Lebesgue measure 0).
Use part (b) and (c) and the dominated convergence theorem to conclude that $x g n x 0$ as $n$ .
Show that $x g n x 0$ .
Conclude that $x g n x x g n x$ .
Conclude that $x g n x 2 x g n x$ .
Use the results of parts (a), (d), and (g) to finish the proof.

Scheffé's theorem is true if the functions are probability density functions with respect to an arbitrary positive measure on

, not just Lebesgue measure. The proof is essentially the same. In particular, if we use counting measure, we get the version of Scheffé's theorem for discrete distributions.

Generating Functions

Generating functions are studied in the chapter on Expected Value. In part, the importance of generating functions stems from the fact that ordinary (pointwise) convergence of a sequence of generating functions corresponds to the convergence of the distributions in the sense of this section. Often it is easier to show convergence in distribution using generating functions than directly from the definition.

8. Convergence in Distribution

Basic Theory

Definition

Preliminary Examples

Probability Density Functions

Relation to Convergence in Probability

Examples and Applications

Convergence of the Hypergeometric Distribution to the Binomial

Convergence of the Binomial Distribution to the Poisson

The Geometric Distribution

Convergence of the Matching Distribution to the Poisson

The Extreme Value Distribution

The Pareto Distribution

Fundamental Theorems

Advanced Topics

The Skorohod Representation

Scheffé's Theorem

Generating Functions