Joint Distributions

As usual, we start with a random experiment with probability measure

on an underlying sample space. Suppose now that

X

and

Y

are random variables for the experiment, and that

X

takes values in

S

while

Y

takes values in

T

. We can think of

X Y

as a random variable taking values in the product set

S T

. The purpose of this section is to study how the distribution of

X Y

is related to the distributions of

X

and

Y

individually. In this context, the distribution of

X Y

is called the joint distribution, while the distributions of

X

and of

Y

are referred to as marginal distributions. Note that

X

and

Y

themselves may be vector valued.

The first simple, but very important point, is that the marginal distributions can be obtained from the joint distribution, but not conversely.

Show that

$X A X Y A T$ for $A S$
$Y B X Y S B$ for $B T$

and as we have noted before, this completely determines the distribution

X Y

S T

. However, if

X

and

Y

are dependent, the joint distribution cannot be determined from the marginal distributions. Thus in general, the joint distribution contains much more information than the marginal distributions individually.

Joint and Marginal Densities

In the discrete case, note that

S T

is countable if and only if

S

and

T

are countable.

Suppose that $X Y$ has a discrete distribution with probability density function $f$ on a countable set $S T$ . Show that $X$ and $Y$ have density functions $g$ and $h$ , respectively, given by

$g x y T f x y, x S$
$h y x S f x y, y T$

Suppose that $X Y$ has a continuous distribution on $S T$ with probability density function $f$ . Show that $X$ and $Y$ have continuous distributions with probability density functions $g$ and $h$ , respectively, given by

$g x y T f x y, x S$
$h y x S f x y, y T$

In the context of Exercises 2 and 3,

f

is called the joint probability density function of

X Y

, while

g

and

h

are called the marginal density functions of

X

and of

Y

, respectively.

Independence

When the variables are independent, the joint density is the product of the marginal densities.

Suppose that $X$ and $Y$ are independent, either both with discrete distributions or both with continuous distributions. Let $g$ and $h$ denote the probability density functions of $X$ and $Y$ respectively. Show that $X Y$ has probability density function $f$ given by

f x y g x h y, x y S T

The following exercise gives a converse to Exercise 4. If the joint probability density factors into a function of

x

only and a function of

y

only, then

X

and

Y

are independent.

Suppose that $X Y$ has either a discrete or continuous distribution, with probability density function $f$ . Suppose that

f x y u x v y, x y S T

where $u S 0$ and $v T 0$ . Show that $X$ and $Y$ are independent and that there exists a nonzero constant $c$ such that $g x c u x, x S$ is a probability density function for $X$ and $h y 1 c v y, y T$ is a probability density function for $Y$ .

Random Variables with Mixed Coordinates

The results of this section have natural analogies in the case that

X Y

has coordinates with different distribution types, as discussed in the section on mixed distributions. For example, suppose

X

has a discrete distribution,

Y

has a continuous distribution, and

X Y

has joint probability density function

f

S T

. Then the results in exercises 2(a), 3(b), 4, and 5 hold.

Examples and Applications

Dice

Suppose that two standard, fair dice are rolled and the sequence of scores $X 1 X 2$ recorded. Let $Y X 1 X 2$ and $Z X 1 X 2$ denote the sum and difference of the scores, respectively.

Find the probability density function of $Y Z$ .
Find the probability density function of $Y$
Find the probability density function of $Z$ .
Are $Y$ and $Z$ independent?

Suppose that two standard, fair dice are rolled and the sequence of scores $X 1 X 2$ recorded. Let $U X 1 X 2$ and $V X 1 X 2$ denote the minimum and maximum scores, respectively.

Find the probability density function of $U V$ .
Find the probability density function of $U$
Find the probability density function of $V$ .
Are $U$ and $V$ independent?

Suppose that $X Y$ has probability density function $f x y x y$ for $0 x 1, 0 y 1$ .

Find the probability density function of $X$ .
Find the probability density function of $Y$ .
Are $X$ and $Y$ independent?

Suppose that $X Y$ has probability density function $f x y 2 x y$ for $0 x y 1$ .

Find the probability density function of $X$ .
Find the probability density function of $Y$ .
Are $X$ and $Y$ independent?

Suppose that $X Y$ has probability density function $f x y 6 x 2 y$ for $0 x 1, 0 y 1$ .

Find the probability density function of $X$ .
Find the probability density function of $Y$ .
Are $X$ and $Y$ independent?

Suppose that $X Y$ has probability density function $f x y 15 x 2 y$ for $0 x y 1$ .

Find the probability density function of $X$ .
Find the probability density function of $Y$ .
Are $X$ and $Y$ independent?

Suppose that $X Y Z$ has probability density function $f x y z 2 x y z$ for $0 x 1, 0 y 1, 0 z 1$ .

Find the probability density function of each pair of variables.
Find the probability density function of each variable.
Determine the dependency relationships between the variables.

Suppose that $X Y$ has probability density function $f x y 2 x y$ for $0 x y$ .

Find the probability density function of $X$ .
Find the probability density function of $Y$ .
Are $X$ and $Y$ independent?

Multivariate Uniform Distributions

Multivariate uniform distributions give a geometric interpretation of some of the concepts in this section. Recall first that the standard Lebesgue measure on

n

In particular,

1

is the length measure on

2

is the area measure on

2

, and

3

is the volume measure on

3

Suppose now that

X

takes values in

j

Y

takes values in

k

, and that

X Y

is uniformly distributed on a set

R j k

. Thus, by definition, the joint probability density function of

X Y

Let

S

and

T

be the projections of

R

onto

j

and

k

respectively, defined as follows:

Note that

R S T

. Next we define the cross-sections at

x S

and at

y T

, respectively by

Show that $X$ takes values in $S$ and that the probability density function $g$ of $X$ is proportional to the cross-sectional measure:

g x k T x j k R, x S

Show that $Y$ takes values in $T$ and that the probability density function $h$ of $Y$ is proportional to the cross-sectional measure:

h y j S y j k R, y T

In particular, note from previous exercises that

X

and

Y

are not in general either independent nor uniformly distributed.

Suppose that $R S T$ . Show that

$X$ is uniformly distributed on $S$ .
$Y$ is uniformly distributed on $T$ .
$X$ and $Y$ are independent.

In each of the following cases, find the joint and marginal densities, and determine if $X$ and $Y$ are independent.

$X Y$ is uniformly distributed on the square $R 662$ .
$X Y$ is uniformly distributed on the triangle $R x y 6 y x 6$ .
$X Y$ is uniformly distributed on the circle $R x y x 2 y 2 36$ .

In the bivariate uniform experiment, run the simulation 5000 times, updating every 10 runs for each of the following cases. Watch the points in the scatter plot and the graphs of the marginal distributions. Interpret what you see in the context of the discussion above.

square
triangle
circle

Suppose that $X Y Z$ is uniformly distributed on the cube $013$ .

Give the joint probability density function of $X Y Z$ .
Find the probability density function of each pair of variables.
Find the probability density function of each variable
Determine the dependency relationships between the variables.

Suppose that $X Y Z$ is uniformly distributed on $x y z 0 x y z 1$ .

Give the joint density function of $X Y Z$ .
Find the probability density function of each pair of variables.
Find the probability density function of each variable
Determine the dependency relationships between the variables.

The following exercise shows how an arbitrary continuous distribution can be obtained from a uniform distribution. This result is useful for simulating certain continuous distributions.

Suppose that $g$ is a probability density function for a continuous distribution on $S n$ . Let

R x y x S 0 y g x n 1

Show that if $X Y$ is uniformly distributed on $R$ , then $X$ has probability density function $g$ . A picture in the case $n 1$ is given below:

The Multivariate Hypergeometric Distribution

Suppose that a population consists of

m

objects, and that each object is one of four types. There are

a

type 1 objects,

b

type 2 objects,

c

type 3 objects and

m a b c

type 0 objects. The parameters

a

b

, and

c

are nonnegative integers with

a b c m

. We sample

n

objects from the population at random, and without replacement. Denote the number of type 1, 2, and 3 objects in the sample by

U

V

, and

W

, respectively. Hence, the number of type 0 objects in the sample is

n U V W

. In the problems below, the variables

i

j

, and

k

take values in the set

01 n

Use a combinatorial argument to show that $U V W$ has a (multivariate) hypergeometric distribution, with probability density function:

U i V j W k a i b j c k m a b c n i j k m n, i j k n

Use both a combinatorial argument and an analytic argument to show that $U V$ also has a (multivariate) hypergeometric distribution, with the probability density function given below. The essence of the combinatorial argument is that we are selecting a random sample of size $n$ from a population of $m$ objects, with $a$ objects of type 1, $b$ objects of type 2, and $m a b$ objects of other types.

U i V j a i b j m a b n i j m n, i j n

Use both a combinatorial argument and an analytic argument to show that $U$ has an ordinary hypergeometric distribution, with the probability density function given below. The essence of the combinatorial argument is that we are selecting a random sample of size $n$ from a population of size $m$ , with $a$ objects of type 1 and $m a$ objects of other types.

U i a i m a n i m n, i 01 n

These results generalize in a straightforward way to a population with any number of types. In brief, if a random vector has a hypergeometric distribution, then any sub-vector also has a hypergeometric distribution. In other terms, all of the marginal distributions of a hypergeometric distribution are themselves hypergeometric. The hypergeometric distribution and the multivariate hypergeometric distribution are studied in detail in the chapter on Finite Sampling Models.

Recall that a bridge hand consists of 13 cards selected at random and without replacement from a standard deck of 52 cards. Let $U$ , $V$ , and $W$ denote the number of spades, hearts, and diamonds, respectively, in the hand. Find the density function of each of the following:

$U V W$
$U V$
$U$

Multinomial Trials

Suppose that we have a sequence of independent trials, each with 4 possible outcomes. On each trial, outcome 1 occurs with probability

p

, outcome 2 with probability

q

, outcome 3 with probability

r

, and outcome 0 occurs with probability

1 p q r

. The parameters

p

q

, and

r

are nonnegative numbers with

p q r 1

. Denote the number of times that outcome 1, outcome 2, and outcome 3 occurred in the

n

trials by

U

V

, and

W

respectively. Of course, the number of times that outcome 0 occurs is

n U V W

. In the problems below, the variables

i

j

, and

k

take values in the set

01 n

Use a probability argument (based on combinatorics and independence) to show that $U V W$ has a multinomial distribution, with probability density function given by

U i V j W k n i j k p i q j r k 1 p q r n i j k, i j k n

Use a probability argument and an analytic argument to show that $U V$ also has a multinomial distribution, with the probability density function given below. The essence of the probability argument is that we have $n$ independent trials, and on each trial, outcome 1 occurs with probability $p$ , outcome 2 with probability $q$ , and some other outcome with probability $1 p q$ .

U i V j n i j p i q j 1 p q n i j, i j n

Use a probability argument and an analytic argument to show that $U$ has a binomial distribution, with the probability density function given below. The essence of the probability argument is that we have $n$ independent trials, and on each trial, outcome 1 occurs with probability $p$ and some other outcome with probability $1 p$

U i n i p i 1 p n i, i n

These results generalize in a completely straightforward way to multinomial trials with any number of trial outcomes. In brief, if a random vector has a multinomial distribution, then any sub-vector also has a multinomial distribution. In other terms, all of the marginal distributions of a multinomial distribution are themselves multinomial. The binomial distribution and the multinomial distribution are studied in detail in the chapter on Bernoulli Trials.

Recall that an ace-six flat die is a standard 6-sided die in which faces 1 and 6 have probability $1 4$ each, while faces 2, 3, 4, and 5 have probability $1 8$ each. Suppose that an ace-six flat die is thrown 10 times; let $X i$ denote the number of times that score $i$ occurred for $i 123456$ . Find the density function of each of the following:

$X 1 X 2 X 3 X 4 X 5$
$X 1 X 3 X 4 X 5$
$X 3 X 5 X 6$
$X 1 X 3$
$X 3$

Bivariate Normal Distributions

Suppose that $X Y$ has probability the density function given below:

f x y 112 x 2 8 y 2 18, x y 2

Find the probability density function of $X$ .
Find the probability density function of $Y$ .
Are $X$ and $Y$ independent?

Suppose that $X Y$ has the probability density function given below:

f x y 13 2 3 x 2 x y y 2, x y 2

Find the density function of $X$ .
Find the density function of $Y$ .
Are $X$ and $Y$ independent?

The joint distributions in the last two exercises are examples of bivariate normal distributions. Normal distributions are widely used to model physical measurements subject to small, random errors. The bivariate normal distribution is studied in more detail in the chapter on Special Distributions.

The Exponential Distribution

where

r 0

is the rate parameter. The exponential distribution is widely used to model random times, and is studied in more detail in the chapter on the Poisson Process.

Suppose $X$ and $Y$ have exponential distributions with parameters $a$ and $b$ , respectively, and are independent. Show that $X Y a a b$

Suppose $X$ , $Y$ , and $Z$ have exponential distributions with parameters $a$ , $b$ , and $c$ , respectively, and are independent. Show that

$X Y Z a a b c b b c$
$X Y X Z a a b c$

X

Y

, and

Z

are the lifetimes of devices that act independently, then the results in the previous two exercises give probabilities of various failure orders. Results of this type are also very important in the study of continuous-time Markov processes. We will continue this discussion in the section on transformations of random variables.

Suppose $X$ takes values in the finite set $123$ , $Y$ takes values in the interval $03$ , and that the joint density function $f$ is given by

f x y 1 3 x 1, 0 y 1 1 6 x 2, 0 y 2 1 9 x 3, 0 y 3

Find the probability density function of $X$ .
Find the probability density function of $Y$ .
Are $X$ and $Y$ independent?

Suppose that $V$ takes values in the interval $01$ , $X$ takes values in the finite set $0123$ , and that $V X$ has joint probability density function $f$ given by

f p k 6 3 k p k 1 1 p 4 k, p k 010123

Find the probability density function of $V$ .
Find the probability density function of $X$ .
Are $V$ and $X$ independent?

As we will see in the section on conditional distributions, the distribution in the last exercise models the following experiment: a random probability

V

is selected, and then a coin with this probability of heads is tossed 3 times;

X

is the number of heads.

Data Analysis Exercises

For the cicada data, $G$ denotes gender and $S$ denotes species type.

Find the empirical density of $G S$ .
Find the empirical density of $G$ .
Find the empirical density of $S$ .
Do you believe that $S$ and $G$ are independent?

For the cicada data, $W B$ denotes body weight and $L B$ denotes body length (in mm).

Construct an empirical density for $W B L B$ .
Find the corresponding empirical density for $W B$ .
Find the corresponding empirical density for $L B$ .
Do you believe that $W B$ and $L B$ are independent?

For the cicada data, $G$ denotes gender and $W B$ denotes body weight (in grams).

Construct and empirical density for $G W B$ .
Find the empirical density for $G$ .
Find the empirical density for $W B$ .
Do you believe that $G$ and $W B$ are independent?

4. Joint Distributions

Basic Theory

Joint and Marginal Densities

Independence

Random Variables with Mixed Coordinates

Examples and Applications

Dice

Multivariate Uniform Distributions

The Multivariate Hypergeometric Distribution

Multinomial Trials

Bivariate Normal Distributions

The Exponential Distribution

Data Analysis Exercises