Expected Value and Covariance Matrices

The main purpose of this section is a discussion of expected value and covariance for random matrices and vectors. These topics are particularly important in multivariate statistical models and the multivariate normal distribution. This section requires some prerequisite knowledge of linear algebra.

Basic Theory

We will let

m n

denote the space of all

m n

matrices of real numbers. In particular, we will identify

n

with

n 1

, so that an ordered

n

-tuple can also be thought of as an

n 1

column vector. The transpose of a matrix

A

is denoted

A

. As usual, our starting point is a random experiment with a probability measure

on an underlying sample space.

Expected Value of a Random Matrix

Suppose that

X

is an

m n

matrix of real-valued random variables, whose

i j

entry is denoted

X i j

. Equivalently,

X

can be thought of as a random

m n

matrix. It is natural to define the expected value

X

to be the

m n

matrix whose

i j

entry is

X i j

, the expected value of

X i j

Many of the basic properties of expected value of random variables have analogies for expected value of random matrices, with matrix operation replacing the ordinary ones.

Show that $X Y X Y$ if $X$ and $Y$ are random $m n$ matrices.

Show that $A X A X$ if $A$ is a non-random $m n$ matrix and $X$ is random $n k$ matrix.

Show that $X Y X Y$ if $X$ is a random $m n$ matrix, $Y$ is a random $n k$ matrix, and $X$ and $Y$ are independent.

Covariance Matrices

Suppose now that

X

is a random vector in

m

and

Y

is a random vector in

n

. The covariance matrix of

X

and

Y

is the

m n

matrix

X Y

whose

i j

entry is

X i Y j

the covariance of

X i

and

Y j

Show that $X Y X X Y Y$ .

Show that $X Y X Y X Y$

Show that $Y X X Y$

Show that $X Y 0$ if and only if each coordinate of $X$ is uncorrelated with each coordinate of $Y$ (in particular, this holds if $X$ and $Y$ are independent).

Show that $X Y Z X Z Y Z$ if $X$ and $Y$ are random vectors in $m$ and $Z$ is a random vector in $n$ .

Show that $X Y Z X Y X Z$ if $X$ is a random vector in $m$ and $Y$ and $Z$ are random vectors in $n$ .

Show that $A X Y A X Y$ if $X$ is a random vector in $m$ , $Y$ is a random vector in $n$ , and $A$ is a non-random matrix in $k m$ .

Show that $X A Y X Y A$ if $X$ is a random vector in $m$ , $Y$ is a random vector in $n$ , and $A$ is a non-random matrix in $k n$ .

Variance-Covariance Matrices

Suppose now that

X X 1 X 2 X n

is a random vector in

n

. The covariance matrix of

X

with itself is called the variance-covariance matrix of

X

Show that $VC X$ is a symmetric $n n$ matrix with $X 1 X 2 X n$ on the diagonal.

Show that $VC X Y VC X X Y Y X VC Y$ if $X$ and $Y$ are random vectors in $n$ .

Show that $VC A X A VC X A$ if $X$ is a random vector in $n$ and $A$ is a non-random matrix in $m n$ .

a n

, note that

a X

is simply the inner product or dot product of

a

with

X

, and is a linear combination of the coordinates of

X

Show that $a X a VC X a$ if $X$ is a random vector in $n$ and $a n$ . Thus conclude that $VC X$ is either positive semi-definite or positive definite. In particular, the eigenvalues and the determinant of $VC X$ are nonnegative.

Show that $VC X$ is positive semi-definite (but not positive definite) if and only if there exists $a n$ and $c$ such that, with probability 1,

a X i 1 n a i X i c

Thus, if

VC X

is positive semi-definite, then one of the coordinates of

X

can be written as an affine transformation of the other coordinates (and hence can usually be eliminated in the underlying model). By contrast, if

VC X

is positive definite, then this cannot happen;

VC X

has positive eigenvalues and determinant and is invertible.

Best Linear Predictors

Suppose again that

X X 1 X 2 X m

is a random vector in

m

and that

Y Y 1 Y 2 Y n

is a random vector in

n

. We are interested in finding the linear (technically affine) function of

X

that is closest to

Y

in the mean square sense. This problem is of fundamental importance in statistics when random vector

X

, the predictor vector is observable, but not random vector

Y

, the response vector. Our discussion here generalizes the one-dimensional case, when

X

and

Y

are random variables. That problem was solved in the section on Covariance and Correlation. We will assume that

VC X

is positive definite, so that none of the coordinates of

X

can be written as an affine function of the other coordinates.

Show that $Y A X b 2$ is minimized when $A Y X VC X$ and $b Y Y X VC X X$

Thus, the linear function of

X

that is closest to

X

in the mean square sense is the random vector

is known as the (distribution) linear regression function. If we observe

X x

then

L Y X x

is our prediction of

Y

Non-linear regression with a single, real-valued predictor variable can be thought of as a special case of multiple linear regression. Thus, suppose that

X

is the predictor variable,

Y

is the response variable, and that

g 1 g 2 g n

is a sequence of real-valued functions. We can apply the results of Exercise 17 to find the linear function of

g 1 X g 2 X g n X

that is closest to

Y

in the mean square sense. We just replace

X i

with

g i X

for each

i

Examples and Applications

Suppose that $X Y$ has probability density function $f x y x y, 0 x 1, 0 y 1$ . Find each of the following:

$X Y$
$VC X Y$

Suppose that $X Y$ has probability density function $f x y 2 x y, 0 x y 1$ . Find each of the following:

$X Y$
$VC X Y$

Suppose that $X Y$ has probability density function $f x y 6 x 2 y, 0 x 1, 0 y 1$ . Find each of the following:

$X Y$
$VC X Y$

Suppose that $X Y$ has probability density function $f x y 15 x 2 y, 0 x y 1$ . Find each of the following:

$X Y$
$VC X Y$
$L Y X$
$L Y X X 2$
Sketch the regression curves on the same set of axes.

Suppose that $X Y Z$ is uniformly distributed on the region $x y z 3 0 x y z 1$ Find each of the following:

$X Y Z$
$VC X Y Z$
$L Z X Y$ .
$L Y X Z$ .
$L X Y Z$ .

Suppose that $X$ is uniformly distributed on $01$ , and that given $X$ , $Y$ is uniformly distributed on $0 X$ . Find each of the following:

$X Y$
$VC X Y$

6. Expected Value and Covariance Matrices

Basic Theory

Expected Value of a Random Matrix

Covariance Matrices

Variance-Covariance Matrices

Best Linear Predictors

Examples and Applications