The Multivariate Normal Distribution

Suppose that

U

and

V

are independent random variables each, with the standard normal distribution. We will need the following five parameters:

μ 1

μ 2

σ 1 0

σ 2 0

, and

ρ 11

. Now let

X

and

Y

be new random variables defined by

The joint distribution of

X Y

is called the bivariate normal distribution with parameters

μ 1 μ 2 σ 1 σ 2 ρ

Basic Properties

For the following exercises, use properties of mean, variance, covariance, and the normal distribution.

Show that $X$ is normally distributed with mean $μ 1$ and standard deviation $σ 1$ .

Show that $Y$ is normally distributed with mean $μ 2$ and standard deviation $σ 2$ .

Show that $X$ and $Y$ are independent if and only if $X Y 0$ .

Thus, for two random variables with a joint normal distribution, the random variables are independent if and only if they are uncorrelated.

In the bivariate normal experiment, change the standard deviations of $X$ and $Y$ with the scroll bars. Watch the change in the shape of the probability density functions. Now change the correlation with the scroll bar and note that the probability density functions do not change. For various values of the parameters, run the experiment 2000 times with an update frequency of 10. Observe the cloud of points in the scatterplot, and note the apparent convergence of the empirical density function to the probability density function.

The Probability Density Function

We will now use the change of variables formula to find the joint probability density function of

X Y

Show that inverse transformation is given by

u x μ 1 σ 1, v x μ 1 σ 1 1 ρ 2 ρ y μ 2 σ 2 1 ρ 2

Show that the Jacobian of the transformation in the previous exercise is

∂ u v ∂ x y 1 σ 1 σ 2 1 ρ 2

Note that the Jacobian is a constant; this is because the transformation is linear.

Use the previous exercises, the independence of $U$ and $V$ , and the change of variables formula to show that the joint probability density function of $X Y$ is

f x y 1 2 σ 1 σ 2 1 ρ 2 1 2 1 ρ 2 x μ 1 2 σ 1 2 2 ρ x μ 1 y μ 2 σ 1 σ 2 y μ 2 2 σ 2 2, x y 2

c

is a constant, the set of points

x y 2 f x y c

is called a level curve of

f

(these are curves of constant probability density).

Show that

The level curves of $f$ are ellipses centered at $μ 1 μ 2$ .
The axes of these ellipses are parallel to the coordinate axes if and only if $ρ 0$

In the bivariate normal experiment, run the experiment 2000 times with an update frequency of 10 for selected values of the parameters. Observe the cloud of points in the scatterplot and note the apparent convergence of the empirical density function to the probability density function.

Transformations

The following exercise shows that the bivariate normal distribution is preserved under affine transformations.

Define $W a 1 X b 1 Y c 1$ and $Z a 2 X b 2 Y c 2$ . Use the change of variables formula to show that $W Z$ has a bivariate normal distribution. Identify the means, variances, and correlation.

Show that the conditional distribution of $Y$ given $X x$ is normal with mean and variance given by

$Y X x μ 2 ρ σ 2 x μ 1 σ 1$
$Y X x σ 2 2 1 ρ 2$

Use the definition of $X$ and $Y$ in terms of the independent standard normal variables $U$ and $V$ to show that

Y μ 2 ρ σ 2 X μ 1 σ 1 σ 2 1 ρ 2 V

Now give another proof of the result in Exercise 12 (note that

X

and

V

are independent).

In the bivariate normal experiment, set the standard deviation of $X$ to 1.5, the standard deviation of $Y$ to 0.5, and the correlation to 0.7.

Run the experiment 100 times, updating after each run.
For each run, compute $Y X x$ the predicted value of $Y$ for the given the value of $X$ .
Over all 100 runs, compute the square root of the average of the squared errors between the predicted value of $Y$ and the true value of $Y$ .

The following problem is a good exercise in using the change of variables formula and will be useful when we discuss the simulation of normal variables.

Recall that $U$ and $V$ are independent random variables each with the standard normal distribution. Define the polar coordinates $R Θ$ of $U V$ by the equations $U R Θ$ , $V R Θ$ where $R 0$ and $0 Θ 2$ . Show that

$R$ has probability density function $g r r 1 2 r 2, r 0$ .
$Θ$ is uniformly distributed on $02$ .
$R$ and $Θ$ are independent.

The distribution of $R$ is known as the Rayleigh distribution, named for William Strutt, Lord Rayleigh. It is a member of the family of Weibull distributions, named in turn for Wallodi Weibull.

The General Multivariate Normal Distribution

The general multivariate normal distribution is a natural generalization of the bivariate normal distribution studied above. The exposition is very compact and elegant using expected value and covariance matrices, and would be horribly complex without these tools. Thus, this section requires some prerequisite knowledge of linear algebra. In particular, recall that

A

denotes the transpose of a matrix

A

The Standard Normal Distribution

Suppose that

Z Z 1 Z 2 Z n

is a vector of independent random variables, each with the standard normal distribution. Then

Z

is said to have the

n

-dimensional standard normal distribution.

Show that $Z 0$ (the zero vector in $n$ ).

Show that $VC Z I$ (the $n n$ identity matrix).

Show that $Z$ has probability density function

g z 1 2 n 2 1 2 z z, z n

Show that $Z$ has moment generating function given by

t Z 1 2 t t, t n

The General Normal Distribution

Now suppose that

Z

has the

n

-dimensional standard normal distribution. Suppose that

μ n

and that

A n n

is invertible. The random vector

X μ A Z

is said to have an

n

-dimensional normal distribution.

Show that $VC X A A$ , and that this matrix is invertible and positive definite.

Let $V VC X A A$ . Use the multivariate change of variables theorem to show that $X$ has probability density function

f x 1 2 n 2 V 1 2 x μ V x μ, x n

Show that $X$ has moment generating function given by

t X μ t 1 2 t V t, t n

Note that the matrix

A

that occurs in the transformation is not unique, but of course the variance-covariance matrix

V

is unique. In general, for a given positive definite matrix

V

, there are many invertible matrices

A

such that

A A V

. A theorem in matrix theory states that there is a unique lower triangular matrix

L

with this property.

Identify the lower triangular matrix $L$ for the bivariate normal distribution.

Transformations

The multivariate normal distribution is invariant under two basic types of transformations: affine transformation with an invertible matrix, and the formation of subsequences.

Suppose that $X$ has an $n$ -dimensional normal distribution. Suppose also that $a n$ and that $B n n$ is invertible. Show that $Y a B X$ has a multivariate normal distribution. Identify the mean vector and the variance-covariance matrix of $Y$ .

Suppose that $X$ has an $n$ -dimensional normal distribution. Show that any permutation of the coordinates of $X$ also has an $n$ -dimensional normal distribution. Identify the mean vector and the variance-covariance matrix. Hint: Permuting the coordinates of $X$ corresponds to multiplication of $X$ by a permutation matrix--a matrix of 0's and 1's in which each row and column has a single 1.

Suppose that $X X 1 X 2 X n$ has an $n$ -dimensional normal distribution. Show that if $k n$ , $W X 1 X 2 X k$ has a $k$ -dimensional normal distribution. Identify the mean vector and the variance-covariance matrix.

Use the results of Exercise 26 and Exercise 27 to show that if $X X 1 X 2 X n$ has an $n$ -dimensional normal distribution and if $i 1 i 2 i k$ is a sequence of distinct indices, then $W X i 1 X i 2 X i k$ has a $k$ -dimensional normal distribution.

Suppose that $X$ has an $n$ -dimensional normal distribution, $a m$ , and that $B m n$ has linearly independent rows (thus, $m n$ ). Show that $Y a B X$ has an $m$ -dimensional normal distribution. Hint: There exists an invertible $n n$ matrix $C$ for which the first $m$ rows are the rows of $B$ . Now use Exercise 25 and Exercise 27.

Note that the results in Exercises 25, 26, 27, and 28 are special cases of the result in Exercise 29.

Suppose that $X$ has an $n$ -dimensional normal distribution, $Y$ has an $m$ -dimensional normal distribution, and that $X$ and $Y$ are independent. Show that $X Y$ has an $m n$ dimensional normal distribution. Identify the mean vector and the variance-covariance matrix.

Suppose that $X$ is a random vector in $m$ , $Y$ is a random vector in $n$ , and that $X Y$ has an $m n$ -dimensional normal distribution. Show that $X$ and $Y$ are independent if and only if $X Y 0$ (the $m n$ zero matrix).

7. The Multivariate Normal Distribution

The Bivariate Normal Distribution

Definition

Basic Properties

The Probability Density Function

Transformations

The General Multivariate Normal Distribution

The Standard Normal Distribution

The General Normal Distribution

Transformations