Conditional Expectation

As usual, our starting point is a random experiment with probability measure

Ω

. Suppose that

X

is a random variable taking values in a set

S

and that

Y

is a random variable taking values in

T

. In this section, we will study the conditional expected value of

Y

given

X

, a concept of fundamental importance in probability. As we will see, the expected value of

Y

given

X

is the function of

X

that best approximates

Y

in the mean square sense. Note that in general,

X

will be vector-valued. In this section, we will assume that all real-valued random variables occurring in expected values have finite second moment.

Basic Theory

The Elementary Definition

Note that we can think of

X Y

as a random variable that takes values in a subset of

S T

. Suppose first that the that

X Y

has a (joint) continuous distribution with probability density function

f

. Recall that the (marginal) probability density function

g

X

is given by

Thus, the conditional expected value of

Y

given

X x

is simply the mean computed relative to the conditional distribution:

Of course, the conditional mean of

Y

depends on the given value

x

X

. Temporarily, let

v

denote the function from

S

into

defined by

The function

v

is sometimes refereed to as the regression function of

Y

based on

X

. The random variable

v X

is called the conditional expected value of

Y

given

X

and is denoted

Y X

. The results and definitions above would be exactly the same if

X Y

has a joint discrete distribution, except that sums would replace the integrals. Intuitively, we treat

X

as known, and therefore not random, and we then average

Y

with respect to the probability distribution that remains.

The General Definition

The random variable

Y X

satisfies a critical property that characterizes it among all functions of

X

Suppose that $r$ is a function from $S$ into . Use the change of variables theorem for expected value to show that

r X Y X r X Y

In fact, the result in Exercise 1 can be used as a definition of conditional expected value, regardless of the type of the distribution of

X Y

. Thus, generally we define

Y X

to be the random variable that satisfies the condition in Exercise 1 and is of the form

Y X v X

for some function

v

from

S

into

. Then we define

Y X x

to be

v x

for

x S

. (More technically,

Y X

is required to be measurable with respect to

X

Properties

Our first consequence of Exercise 1 is a formula for computing the expected value of

Y

By taking $r$ to be the constant function 1 in Exercise 1, show that $Y X Y$

Aside from the theoretical interest, the result in Exercise 2 is often a good way to compute

Y

when we know the conditional distribution of

Y

given

X

. We say that we are computing the expected value of

Y

by conditioning on

X

Show that, in light of Exercise 2, the condition in Exercise 1 can be restated as follows: For any function $r S$ , the random variables $Y Y X$ and $r X$ are uncorrelated.

Suppose that $u X$ and $v X$ satisfy the condition in Exercise 1 and hence also the results in Exercise 2 and Exercise 3. Show that $u X$ and $v X$ are equivalent:

First show that $u X v X 0$
Then show that $u X v X 1$ .

Suppose that $s S$ . Use the characterization in Exercise 1 to show that

s X Y X s X Y X

This result makes intuitive sense: if we know

X

, then we know any (deterministic) function of

X

. Any such function acts like a constant in terms of the conditional expected value with respect to

X

. The following rule generalizes this result and is sometimes referred to as the substitution rule for conditional expected value.

Suppose that $s S T$ . Show that

s X Y X x s x Y X x

In particular, it follows from Exercise 5 or Exercise 6 that

s X X s X

. At the opposite extreme, we have the result in the next exercise. If

X

and

Y

are independent, then knowledge of

X

gives no information about

Y

and so the conditional expected value with respect to

X

is the same as the ordinary (unconditional) expected value of

Y

Suppose that $X$ and $Y$ are independent. Use the characterization in Exercise 1 to show that

Y X Y

Use the general definition to establish the properties in the following exercises, where

Y

and

Z

are real-valued random variables and

c

is a constant. Note that these are analogies of basic properties of ordinary expected value. Every type of expected value must satisfy two critical properties: linearity and monotonicity.

Show that $Y Z X Y X Z X$ .

Show that $c Y X c Y X$ .

Show that if $Y 0$ with probability 1, then $Y X 0$ with probability 1.

Show that if $Y Z$ with probability 1, then $Y X Z X$ with probability 1.

Show that $Y X Y X$ with probability 1.

Suppose now that

Z

is real-valued and that

X

and

Y

are random variables (all defined on the same probability space, of course). The following exercise gives a consistency condition of sorts. Iterated conditional expected values reduce to a single conditional expected value with respect to the minimum amount of information:

Show that

Z X Y X Z X X Y Z X

Conditional Probability

The conditional probability of an event

A

, given random variable

X

, is a special case of the conditional expected value. As usual, let

A

denote the indicator random variable of

A

. We define

The properties above for conditional expected value, of course, have special cases for conditional probability.

Again, the result in the previous exercise is often a good way to compute

A

when we know the conditional probability of

A

given

X

. We say that we are computing the probability of

A

by conditioning on

X

. This is a very compact and elegant version of the conditioning result given first in the section on Conditional Probability in the chapter on Probability Spaces and later in the section on Discrete Distributions in the Chapter on Distributions.

The Best Predictor

The next two exercises show that, of all functions of

X

Y X

is the best predictor of

Y

, in the sense of minimizing the mean square error. This is fundamentally important in statistical problems where the predictor vector

X

can be observed but not the response variable

Y

Let $v X Y X$ and let $u S$ . By adding and subtracting $v X$ , expanding, and using the result of Exercise 3, show that

Y u X 2 Y v X 2 v X u X 2

Use the result of the last exercise to show that if $u S$ , then

Y X Y 2 u X Y 2

and equality holds if and only if $u X Y X$ with probability 1.

Suppose now that

X

is real-valued. In the section on covariance and correlation, we found that the best linear predictor of

Y

based on

X

On the other hand,

Y X

is the best predictor of

Y

among all functions of

X

. It follows that if

Y X

happens to be a linear function of

X

then it must be the case that

Y X L Y X

Show that $X Y X X Y$

Show directly that if $Y X a X b$ then

$a X Y X$
$b Y X Y X X$

Conditional Variance

Show that $Y X Y 2 X Y X 2$ .

Show that $Y Y X Y X$ .

Again, the result in the previous exercise is often a good way to compute

Y

when we know the conditional distribution of

Y

given

X

. We say that we are computing the variance of

Y

by conditioning on

X

Let us return to the study of predictors of the real-valued random variable

Y

, and compare the three predictors we have studied in terms of mean square error.

Examples and Applications

Suppose that $X Y$ has probability density function $f x y x y, 0 x 1, 0 y 1$ .

Find $L Y X$ .
Find $Y X$ .
Graph $L Y X x$ and $Y X x$ as functions of $x$ , on the same axes.
Find $Y$ .
Find $Y 1 X Y 2$ .
Find $Y Y X$ .

Suppose that $X Y$ has probability density function $f x y 2 x y, 0 x y 1$ .

Find $L Y X$ .
Find $Y X$ .
Graph $L Y X x$ and $Y X x$ as functions of $x$ , on the same axes.
Find $Y$ .
Find $Y 1 X Y 2$ .
Find $Y Y X$ .

Suppose that $X Y$ has probability density function $f x y 6 x 2 y, 0 x 1, 0 y 1$ .

Find $L Y X$ .
Find $Y X$ .
Graph $L Y X x$ and $Y X x$ as functions of $x$ , on the same axes.
Find $Y$ .
Find $Y 1 X Y 2$ .
Find $Y Y X$ .

Suppose that $X Y$ has probability density function $f x y 15 x 2 y, 0 x y 1$ .

Find $L Y X$ .
Find $Y X$ .
Graph $L Y X x$ and $Y X x$ as functions of $x$ , on the same axes.
Find $Y$ .
Find $Y 1 X Y 2$ .
Find $Y Y X$ .

Suppose that $X$ , $Y$ , and $Z$ are real-valued random variables with $Y X X 3$ and $Z X 1 1 X 2$ . Find $Y X Z X X$ .

Uniform Distributions

Suppose that $X Y$ is uniformly distributed on a rectangular region $S a b c d 2$ . Find $Y X$ .

In the bivariate uniform experiment, select the square in the list box. Run the simulation 2000 times, updating every 10 runs. Note the relationship between the cloud of points and the graph of the regression function.

Suppose that $X Y$ is uniformly distributed on a triangular region $T x y 2 a x y a$ , where $a 0$ is a parameter. Find $Y X$ .

In the bivariate uniform experiment, select the triangle in the list box. Run the simulation 2000 times, updating every 10 runs. Note the relationship between the cloud of points and the graph of the regression function.

Suppose that $X$ is uniformly distributed on $01$ , and that given $X$ , $Y$ is uniformly distributed on $0 X$ . Find each of the following:

$Y X$
$Y$
$Y X$
$Y$

Coins and Dice

A pair of fair dice are thrown, and the scores $X 1 X 2$ recorded. Let $Y X 1 X 2$ denote the sum of the scores and $U X 1 X 2$ the minimum score. Find each of the following:

$Y X 1$
$U X 1$
$Y U$
$X 2 X 1$

A box contains 10 coins, labeled 0 to 9. The probability of heads for coin $i$ is $i 9$ . A coin is chosen at random from the box and tossed. Find the probability of heads. This problem is an example of Laplace's rule of succession,

Random Sums of Random Variables

Suppose that

X X 1 X 2

is a sequence of independent and identically distributed real-valued random variables. We will denote the common mean, variance, and moment generating function, respectively, by:

μ X i

σ 2 X i

, and

G t t X i

. Let

so that

Y Y 0 Y 1

is the partial sum process associated with

X

. Suppose now that

N

is a random variable taking values in

, independent of

X

. Then

is a random sum of random variables; the terms in the sum are random, and the number of terms is random. This type of variable occurs in many different contexts. For example,

N

might represent the number of customers who enter a store in a given period of time, and

X i

the amount spent by the customer

i

Show that

$Y N N N μ$
$Y N N μ$

Wald's equation, named for Abraham Wald, is a generalization of the result in the previous exercise to the case where

N

is not necessarily independent of

X

, but rather is a stopping time for

X

. Wald's equation is discussed in the section on Partial Sums in the chapter on Random Samples.

Show that

$Y N N N σ 2$
$Y N N σ 2 N μ 2$

Let $H$ denote the probability generating function of $N$ . Show that the moment generating function of $Y N$ is $H G$ .

$t Y N N G t N$
$t Y N H G t$

In the die-coin experiment, a fair die is rolled and then a fair coin is tossed the number of times showing on the die. Let $N$ denote the die score and $Y$ the number of heads.

Find the conditional distribution of $Y$ given $N$ .
Find $Y N$ .
Find $Y N$ .
Find $Y$ .
Find $Y$ .

Run the die-coin experiment 1000 times, updating every 10 runs. Note the apparent convergence of the empirical mean and standard deviation to the distribution mean and standard deviation.

The number of customers entering a store in a given hour is a random variable with mean 20 and standard deviation 3. Each customer, independently of the others, spends a random amount of money with mean $50 and standard deviation $5. Find the mean and standard deviation of the amount of money spent during the hour.

A coin has a random probability of heads $V$ and is tossed a random number of times $N$ . Suppose that $V$ is uniformly distributed on $01$ ; $N$ has the Poisson distribution with parameter $a 0$ ; and $V$ and $N$ are independent. Let $Y$ denote the number of heads. Compute the following:

$Y N V$
$Y N$
$Y V$
$Y$
$Y N V$
$Y$

Mixtures of Distributions

Suppose that

X X 1 X 2

is a sequence of real-valued random variables. Let

μ i X i

σ i 2 X i

, and

M i t t X i

for

i

. Suppose also that

N

is a random variable taking values in

, independent of

X

. Denote the probability density function of

N

p i N i

for

i

. The distribution of the random variable

X N

is a mixture of the distributions of

X X 1 X 2

, with the distribution of

N

as the mixing distribution.

Show that $X N N μ N$

Show that $X N i 1 p i μ i$ .

Show that $X N i 1 p i σ i 2 μ i 2 i 1 p i μ i 2$ .

Show that $t X N i 1 p i M i t$

In the coin-die experiment, a biased coin is tossed with probability of heads $1 3$ . If the coin lands tails, a fair die is rolled; if the coin lands heads, an ace-six flat die is rolled (faces 1 and 6 have probability $1 4$ each, and faces 2, 3, 4, 5 have probability $1 8$ each). Find the mean and standard deviation of the die score.

Run the coin-die experiment 1000 times, updating every 10 runs. Note the apparent convergence of the empirical mean and standard deviation to the distribution mean and standard deviation.

Vector Space Concepts

Conditional expectation can be interpreted in terms vector of space concepts. This connection can help illustrate many of the properties of conditional expectation from a different point of view.

Recall that the vector space

V 2

consists of all real-valued random variables defined on a fixed sample space

Ω

(that is, relative to the same random experiment) that have finite second moment. Recall that two random variables are equivalent if they are equal with probability 1. We consider two such random variables as the same vector, so that technically, our vector space consists of equivalence classes under this equivalence relation. The addition operator corresponds to the usual addition of two real-valued random variables, and the operation of scalar multiplication corresponds to the usual multiplication of a real-valued random variable by a real (non-random) number.

Projections

Suppose now that

X

a general random variable defined on the sample space

S

, and that

Y

is a real-valued random variable in

V 2

Show that the set below is a subspace of $V 2$ :

U U V 2 u S U u X >

Reconsider Exercise 3 to show that $Y X$ is the projection of $Y$ on to the subspace $U$ .

is also a subspace of

V 2

, and in fact is clearly also a subspace of

U

. We showed that the

L Y X

is the projection of

Y

onto

W

5. Conditional Expected Value

Basic Theory

The Elementary Definition

The General Definition

Properties

Conditional Probability

The Best Predictor

Conditional Variance

Examples and Applications

Uniform Distributions

Coins and Dice

Random Sums of Random Variables

Mixtures of Distributions

Vector Space Concepts

Projections