Independence

As usual, suppose that we have a random experiment with sample space

S

and probability measure

. In this section, we will discuss independence, one of the fundamental concepts in probability theory. Independence is frequently invoked as a modeling assumption, and moreover, probability itself is based on the idea of independent replications of the experiment.

Basic Theory

We will define independence on increasingly complex structures, from two events, to arbitrary collections of events, and then to collections of random variables. In each case, the basic idea is the same.

Independence of Two Events

If both of the events have positive probability, then independence is equivalent to the statement that the conditional probability of one event given the other is the same as the unconditional probability of the event:

This is how you should think of independence: knowledge that one event has occurred does not change the probability assigned to the other event.

The terms independent and disjoint sound vaguely similar but they are actually very different. First, note that disjointness is purely a set-theory concept while independence is a probability (measure-theoretic) concept. Indeed, two events can be independent relative to one probability measure and dependent relative to another. But most importantly, two disjoint events can never be independent, except in the trivial case that one of the events is null.

Suppose that $A$ and $B$ are disjoint events for an experiment, each with positive probability. Show that $A$ and $B$ are negatively correlated and hence dependent.

A

and

B

are independent events in an experiment, it seems clear that any event that can be constructed from

A

should be independent of any event that can be constructed from

B

. This is the case, as the next exercise shows.

Suppose that $A$ and $B$ are independent events in an experiment. Show that each of the following pairs of events is independent:

$A, B$
$B, A$
$A, B$

Suppose that $A$ and $B$ are events in a random experiment. Show that

If $A 0$ or $A 1$ , then $A$ and $B$ are independent.
$A$ is independent of itself if and only if $A 0$ or $A 1$ .

General Independence of Events

To extend the definition of independence to more than two events, we might think that we could just require pairwise independence, the independence of each pair of events. The exercise below shows that three events can be pairwise independent, but a combination of two of the events can be related to a third event in the strongest possible sense.

Consider the dice experiment that consists of rolling 2 standard, fair dice and recording the sequence of scores. Let $A$ denote the event that first score is 3, $B$ the event that the second score is 4, and $C$ the event that the sum of the scores is 7.

Show that the events $A$ , $B$ , $C$ are pairwise independent.
Show that $A B$ implies (is a subset of) $C$ .

In the dice experiment, set $n 2$ . Run the experiment 500 times. For each pair of events in the previous exercise, compute the product of the empirical probabilities and the empirical probability of the intersection. Compare the results.

Another possible generalization would be to simply require the probability of the intersection of the events to be the product of the probabilities of the events. This is also not the right way to go, as the following exercise illustrates.

Suppose that we throw a standard, fair die one time. Let $A 1234$ , $B C 456$ .

Show that $A B C A B C$
But note that $B$ and $C$ are the same event, and hence are dependent in the strongest possbile sense.

However, the definition of independence for two events does generalize in a natural way to an arbitrary collection of events. Specifically, a collection of events

A

is said to be independent if for every finite subcollection

A 1 A 2 A k A

General independence of a collection of events is much stronger than mere pairwise independence of the events in the collection. The basic inheritance property in the following exercise is essentially equivalent to the definition.

Suppose that $A$ is a collection of events

If $A$ is independent, then $B$ is independent for every $B A$
If $B$ is independent for every finite $B A$ then $A$ is independent.

Show that there are $2 n n 1$ non-trivial conditions in the definition of the independence of $n$ events.

Explicitly give the 4 conditions that must be satisfied for events $A$ , $B$ , and $C$ to be independent.
Explicitly give the 11 conditions that must be satisfied for events $A$ , $B$ , $C$ , and $D$ to be independent.

If the finite collection of events

A 1 A 2 A n

is independent, then it follows immediately from the definition that

This is known as the multiplication rule for independent events. Compare this with the general multiplication rule for conditional probability.

Show that the collection of essentially deterministic events $D A S A 0 A 1$ is an independent collection of events.

Suppose that $A$ , $B$ , $C$ , and $D$ are independent events in an experiment. Show directly that $A B$ and $C D$ are independent.

Compare the previous exercise with Exercise 2. The complete generalization of these results is a bit complicated, but goes roughly as follows: Suppose that

A

is a independent collection of events and that

A i i I

is a pairwise disjoint set of subcollections of

A

(where

I

is an index set). That is,

A i A

for each

i

and

A i A j

for distinct

i

and

j

. Suppose now that for each

i I

, an event

A i

is constructed from the events in

A i

using a countable number of set operations. Then the collection of events

A i i I

is independent.

The following exercise gives a formula for the probability of the union of a collection of independent events that is much nicer than the inclusion-exclusion formula.

Suppose that $A 1 A 2 A n$ is a finite collection of independent events. Show that

i 1 n A i 1 i 1 n 1 A i

Suppose that $A$ is an independent collection of events, each with the same probability. Show that $A$ is also an exchangeable collection of events. The converse is not true, as Pólya's urn model so dramatically illustrates.

Independence of Random Variables

Suppose now that

X i

is a random variable taking values in

T i

for each

i

in a nonempty index set

I

. Intuitively, the random variables are independent if knowledge of the values of some of the variables tells us nothing about the values of the other variables. Mathematically, independence of random variables can be reduced to the independence of events. Formally, the collection of random variables

X i i I

is independent if every collection of events of the following form is independent:

Equivalently then, the collection of random variables is independent if for every finite subset

J

I

, and for every choice of

B j T j

for

j J

we have

Suppose that $A$ is a collection of random variables. Show that

If $A$ is independent, then $B$ is independent for every $B A$
If $B$ is independent for every finite $B A$ then $A$ is independent.

Suppose that $X i i I$ is a collection of independent random variables, as above, and suppose that for each $i I$ , $g i$ is a function from $T i$ into a set $U i$ . Show that the collection of random variables $g i X i i I$ is also independent.

Independence of random variables subsumes independence of events. Show that a collection of events $A$ is independent if and only if the corresponding collection of indicator variables $A A A$ is independent.

Many of the concepts that we have been using informally can now be made precise. A compound experiment that consists of independent stages is essentially just an experiment whose outcome is a sequence of independent random variables

X 1 X 2

where

X i

is the outcome of the

i

stage:

In particular, suppose that we have a basic experiment with outcome variable

X

. By definition, the outcome of the experiment that consists of independent replications of the basic experiment is a sequence of independent random variables

X X 1 X 2

each with the distribution of

X

. This is fundamental to the very concept of probability, as expressed in the law of large numbers. From a statistical point of view, suppose that we have a population of objects and a vector of measurements

X

of interest for the objects in the sample. The sequence

X

above corresponds to sampling from the distribution of

X

; that is,

X i

is the vector of measurements for the

i th

object drawn from the sample. When we sample from a finite population, sampling with replacement generates independent random variables while sampling without replacement generates dependent random variables.

Conditional Independence

As noted at the beginning of our discussion, independence of events or random variables depends on the underlying probability measure. Thus, suppose that

B

is an event in a random experiment with positive probability. A collection of events or a collection of random variables is conditionally independent given

B

if the collection is independent relative to the conditional probability measure

A A B

. Note that the definitions and theorems of this section would still be true, but with all probabilities conditioned on

B

An Interpretation of Conditional Probability

The following exercises gives an important interpretation of conditional probability. Suppose that we start with a basic experiment that has sample space

S

and outcome variable

X

. In particular, recall that

X

is simply the identity function on

S

so that if

A

is an event, then trivially,

X A A

Suppose now that we replicate the experiment independently. This results in a new, compound experiment with a sequence of independent random variables

X 1 X 2

, each with the same distribution as

X

. Suppose now that

A

and

B

are events in the basic experiment (that is, subsets of

S

) with

B 0

Show that, in the compound experiment, the event that when $B$ occurs for the first time, $A$ also occurs is

n 1 X 1 B X 2 B X n 1 B X n A B

Show that the probability of the event in the last exercise is

A B B A B

Argue the result in the last exercise directly. Specifically, suppose that we create a new experiment by repeating the basic experiment until $B$ occurs for the first time, and then record the outcome of just the last repetition of the basic experiment. Argue that the appropriate probability measure on the new experiment is $A A B$ .

Suppose that $A$ and $B$ are disjoint events in a basic experiment with $A 0$ and $B 0$ . In the compound experiment obtained by replicating the basic experiment, show that the event that $A$ occurs before $B$ has probability

A A B

Examples and Applications

Suppose that $A$ , $B$ , and $C$ are independent events in an experiment with $A 0.3$ , $B 0.5$ , and $C 0.8$ . Express each of the following events in set notation and find its probability:

All three events occur.
None of the three events occurs.
At least one of the three events occurs.
Exactly one of the three events occurs.
Exactly two of the three events occurs.

Suppose that $A$ , $B$ and $C$ are independent events for an experiment with $A 1 2$ , $B 1 3$ , $C 1 4$ . Find the probability of each of the following events:

$A B C$
$A B C$
$A B C$

A small company has 100 employees; 40 are men and 60 are women. There are 6 male executives. How many female executives should there be if gender and rank are independent? (The underlying experiment is to choose an employee at random.)

Suppose that 3 students, who ride together, miss a mathematics exam. They decide to lie to the instructor by saying that the car had a flat tire. The instructor separates the students and asks each of them which tire was flat. The students, who did not anticipate this, select their answers independently and at random. Find the probability that the students get away with their deception. For a more extensive treatment of the lying students problem, see The Number of Distinct Sample Values in the Chapter on Finite Sampling Models.

Bernoulli Trials

A Bernoulli trials sequence is a sequence

X 1 X 2

of independent, identically distributed indicator variables. Random variable

X i

is the outcome of trial

i

, where in the usual terminology of reliability theory, 1 denotes success and 0 denotes failure. The canonical example is the sequence of scores when a coin (not necessarily fair) is tossed repeatedly. The process is named for Jacques Bernoulli, and has a single basic parameter

p X i 1

. This random process is studied in detail in the chapter on Bernoulli trials.

Show that

X 1 x 1 X 2 x 2 X n x n p x 1 x 2 x n 1 p n x 1 x 2 x n for x 1 x 2 x n 01 n

Let $Y$ denote the number of successes in the first $n$ trials. Show that

Y k n k p k 1 p n k for k 01 n

The distribution of

Y

is called the binomial distribution with parameters

n

and

p

. The binomial distribution is studied in detail in the chapter on Bernoulli Trials.

More generally, a multinomial trials sequence is a sequence

X 1 X 2

of independent, identically distributed random variables, each taking values in a finite set

S

. The canonical example is the sequence of scores when a

k

-sided die (not necessarily fair) is thrown repeatedly. Multinomial trials are studied in detail in the chapter on Bernoulli trials.

Cards

Consider the experiment that consists of dealing 2 cards from a standard deck and recording the sequence of cards dealt. For $i 12$ , let $Q i$ be the event that card $i$ is a queen and $H i$ the event that card $i$ is a heart. Compute the appropriate probabilities to verify the following results. Reflect on these results.

$Q 1$ and $H 1$ are independent.
$Q 2$ and $H 2$ are independent.
$Q 1$ and $Q 2$ are negatively correlated..
$H 1$ and $H 2$ are negatively correlated..
$Q 1$ and $H 2$ are independent.
$H 1$ and $Q 2$ are independent.

In the card experiment, set $n 2$ . Run the simulation 500 times. For each pair of events in the previous exercise, compute the product of the empirical probabilities and the empirical probability of the intersection. Compare the results.

Dice

Suppose that a standard, fair die is thrown 5 times. Find the probability of getting at least one six.

Suppose that a pair of standard, fair dice are thrown 10 times. Find the probability of getting at least one double six.

Consider the dice experiment that consists of rolling $n$ , $k$ -sided dice and recording the sequence of scores $X 1 X 2 X n$ . Show that the following conditions are equivalent (and correspond to the assumption that the dice are fair):

$X 1 X 2 X n$ is uniformly distributed on $12 k n$ .
$X 1 X 2 X n$ is an independent sequence and $X i$ is uniformly distributed on $12 k .$ for each $i$ .

A pair of standard, fair dice are rolled. Find the probability that a sum of 4 occurs before a sum of 7. Problems of this type are important in the game of craps.

Coins

A biased coin with probability of heads $1 3$ is tossed 5 times. Let $Y$ denote the number of heads. Explicitly compute $Y k$ for $k 012345$ .

A box contains a fair coin and a two-headed coin. A coin is chosen at random from the box and tossed repeatedly. Let $F$ denote the event that the fair coin is chosen, and let $X i$ denote the outcome of the $i th$ toss (where 1 encodes heads and 0 encodes tails).

Argue that $X 1 X 2$ are conditionally independent given $F$ , with $X i 1 F 1 2$ for each $i$ .
Argue that $X 1 X 2$ are conditionally independent given $F$ , with $X i 1 F 1$ for each $i$ .
Show that $X i 1 3 4$ for each $i$ . (Condition on the type of coin and use parts (a) and (b).)
Show that $X 1 1 X 2 1 X n 1 1 2 n 1 1 2$ . (Condition on the type of coin and use parts (a) and (b).)
Show that $X 1 X 2$ are dependent. (Use parts (c) and (d).)
Show that $F X 1 1 X 2 1 X n 1 1 2 n 1$ . (Use Bayes' theorem and part (d).)
Note that $F X 1 1 X 2 1 X n 1 0$ as $n$ . Think about this result.

Consider again the box in the previous exercise, but we change the experiment as follows: a coin is chosen at random from the box and tossed and the result recorded. The coin is returned to the box and the process is repeated. As before, let $X i$ denote the outcome of toss $i$ . Show that $X 1 X 2$ is a Bernoulli trials sequence with parameter $p 3 4$ . Specifically,

Argue that $X 1 X 2$ are independent.
Show that $X i 1 3 4$ for each $i$ . (Condition on the type of the $i$ coin.)
Show that $X 1 1 X 2 1 X n 1 3 4 n$ .

Buffon's Coin

Recall that Buffon's coin experiment consists of tossing a coin with radius $r 1 2$ randomly on a floor covered with square tiles of side length 1. The coordinates $X Y$ of the center of the coin are recorded relative to axes through the center of the square in which the coin lands. Show that the following conditions are equivalent:

$X Y$ is uniformly distributed on $1 2 1 2 2$ .
$X$ and $Y$ are independent and each is uniformly distributed on $1 2 1 2$ .

In Buffon's coin experiment, set $r 0.3$ . Run the simulation 500 times. For the events $X 0$ and $Y 0$ , compute the product of the empirical probabilities and the empirical probability of the intersection. Compare the results.

The arrival time $X$ of the $A$ train is uniformly distributed on the interval $030$ , while the arrival time $Y$ of the $B$ train is uniformly distributed on the interval $1560$ . (The arrival times are in minutes, after 8:00 AM). Moreover, the arrival times are independent.

Find the probability that the $A$ train arrives first.
Find the probability that the both trains arrive sometime after 20 minutes.

Reliability

Recall the simple model of structural reliability in which a system is composed of

n

components. Suppose in addition that the components operate independently of each other. As before, let

X i

denote the state of component

i

, where 1 means working and 0 means failure. Thus, our basic assumption is that

X 1 X 2 X n

is a vector of independent indicator random variables that specifies the states of all of the components. We assume that the state of the system (either working or failed) depends only on the states of the components, according to a structure function. Thus, the state of the system is an indicator random variable

Generally, the probability that a device is working is the reliability of the device. Thus, we will denote the reliability of component

i

p i X i 1

so that the vector of component reliabilities is

p 1 p 2 p n

. By independence, the system reliability

r

is a function of the component reliabilities:

Appropriately enough, this function is known as the reliability function. Our challenge is usually to find the reliability function, given the structure function. When the components all have the same probability

p

then of course the system reliability

r

is just a function of

p

. In this case, the component states

X 1 X 2 X n

. forms a sequence of Bernoulli trials.

Comment on the independence assumption for real systems, such as your car or your computer.

Recall that a series system is working if and only if each component is working. Show that

The state of the system is $U X 1 X 2 ··· X n X 1 X 2 X n$ .
The reliability is $U 1 p 1 p 2 ··· p n$ .

Recall that a parallel system is working if and only if at least one component is working. Show that

The state of the system is $V 1 1 X 1 1 X 2 ··· 1 X n X 1 X 2 X n$ .
The reliability is $V 1 1 1 p 1 1 p 2 ··· 1 p n$ .

Recall that a

k

out of

n

system is working if and only if at least

k

of the

n

components are working. Thus, a parallel system is a 1 out of

n

system and a series system is an

n

out of

n

system. A

k

out of

2 k 1

system is a majority rules system. The reliability function of a general

k

out of

n

system is a mess. However, if the component reliabilities are the same, the function has a reasonably simple form.

Show that for a $k$ out of $n$ system with common component reliability $p$ , the system reliability is

r p i k n n i p i 1 p n i

Consider a system of 3 components with reliabilities $p 1 0.8$ , $p 2 0.9$ , $p 3 0.7$ . Find the reliability of each of the following:

The series system.
The 2 out of 3 system.
The parallel system.

Consider an airplane with an odd number of engines, each with reliability $p$ . Suppose that the airplane is a majority rules system, so that the airplane needs a majority of working engines in order to fly.

Find the reliability of a 3 engine plane as a function of $p$ .
Find the reliability of a 5 engine plane as a function of $p$ .
For what values of $p$ is a 5 engine plane preferable to a 3 engine plane?

The graph below is known as the Wheatstone bridge network and is named for Charles Wheatstone. The edges represent components, and the system works if and only if there is a working path from vertex $a$ to vertex $b$ .

Find the structure function.
Find the reliability function.

A system consists of 3 components, connected in parallel. Because of environmental factors, the components do not operate independently, so our usual assumption does not hold. However, we will asume that under low stress conditions, the components are independent, each with reliability 0.9; under medium stress conditions, the components are independent with reliability 0.8; and under high stress conditions, the components are independent, each with reliability 0.7. The probability of low stress is 0.5, of medium stress is 0.3, and of high stress is 0.2.

Find the reliability of the system. (Condition on the stress level.)
Given that the system works, find the conditional probability of each stress level. (Use Bayes' theorem and part (a).)

Diagnostic Testing

Please recall the discussion of diagnostic testing in the section on Conditional Probability. Thus, we have an event

A

for a random experiment whose occurrence or non-occurrence we cannot observe directly. Suppose now that we have

n

tests for the occurrence of

A

, labeled from 1 to

n

. We will let

T i

denote the event that test

i

is positive for

A

. The tests are independent in the following sense:

A

occurs, then

T 1 T 2 T n

are (conditionally) independent and test

i

has sensitivity

A

does not occur, then

T 1 T 2 T n

are (conditionally) independent and test

i

has specificity

We can form a new, compound test by giving a decision rule in terms of the individual test results. In other words, the event

T

that the compound test is positive for

A

is a function of

T 1 T 2 T n

. The typical decision rules are very similar to the reliability structures discussed above. A special case of interest is when the

n

tests are independent applications of a given basic test. In this case, the

a i a

and

b i b

for each

i

Consider the compound test that is positive for $A$ if and only if each of the $n$ tests is positive for $A$ . Show that

$T T 1 T 2 T n$
The sensitivity is $T A a 1 a 2 a n$
The specificity is $T A 1 1 b 1 1 b 2 1 b n >$

Consider the compound test that is positive for $A$ if and only if each at least one of the $n$ tests is positive for $A$ . Show that

$T T 1 T 2 T n$
The sensitivity is $T A 1 1 a 1 1 a 2 1 a n >$
The specificity is $T A b 1 b 2 b n$

More generally, we could define the compound

k

out of

n

test that is positive for

A

if and only if at least

k

of the individual tests are positive for

A

. The test in Exercise 46 is the

n

out of

n

test, while the test in Exercise 47 is the 1 out of

n

test. The

k

out of

2 k 1

test is the majority rules test.

Suppose that a woman initially believes that there is an even chance that she is pregnant or not pregnant. She buys three identical pregnancy tests with sensitivity 0.95 and specificity 0.90. Tests 1 and 3 are positive and test 2 is negative. Find the probability that the woman is pregnant.

Suppose that 3 independent, identical tests for an event $A$ are applied, each with sensitivity $a$ and specificity $b$ . Find the sensitivity and specificity of the following tests:

1 out of 3 test
2 out of 3 test
3 out of 3 test

In a criminal trial, the defendant is convicted if and only if all 6 jurors vote guilty. Assume that if the defendant really is guilty, the jurors vote guilty, independently, with probability 0.95, while if the defendant is really innocent, the jurors vote not guilty, independently with probability 0.8. Suppose that 70% of defendants brought to trial are guilty.

Find the probability that the defendant is convicted.
Given that the defendant is convicted, find the probability that the defendant is guilty.
Comment on the assumption that the jurors act independently.

Genetics

Recall our discussion of genetics in the section on Probability Measure and our discussion of genetics in the section on Conditional Probability. For a given genetic trait (such as eye color or the presence of a disorder), it's usually reasonable to assume that the genotypes of the children are conditionally independent, given the genotypes of the parents. Unconditionally, however, the state of a child (for the given trait) gives information about the states of the parents, which in turn give information about the states of other children.

In the following exercise, suppose that a certain type of pea plant has either green pods or yellow pods, and that the green-pod gene is dominant. Thus, a plant with genotype

g g

g y

has green pods, while a plant with genotype

y y

has yellow pods.

Suppose that 2 green-pod plants are bred together. Suppose further that each plant, independently, has the recessive yellow-pod gene with probability $1 4$

Find the probability that 3 offspring plants will have green pods.
Given that the 3 offspring plants have green pods, find the updated probability that both parents have the recessive gene.

In the following exercise, consider a sex-linked hereditary disorder associated with a gene on the X chromosome. As before, let

n

denote the dominant normal gene and

d

the recessive defective gene. Thus, a woman of genotype

n n

is normal; a woman of genotype

n d

is free of the disease, but is a carrier; and a woman of genotype

d d

has the disease. A man of genotype

n

is normal and a man of genotype

d

has the disease.

Suppose that a healthy woman initially has a $1 2$ chance of being a carrier. (From our discussion above, this would be the case, for example, if her mother and father are healthy but she has a brother with the disorder, so that her mother must be a carrier).

Find the probability that the first two sons of the women will be healthy.
Given that the first two sons are healthy, compute the updated probability that she is a carrier.
Given that the first two sons are healthy, compute the conditional probability that the third son will be healthy.

Laplace's Rule of Succession

Suppose that we have

m 1

coins, labeled 0, 1, ...,

m

. Coin

i

lands heads with probability

i m

for each

i

. In particular, note that, coin 0 is two-tailed and coin

m

is two-headed. Our experiment is to choose a coin at random (so that each coin is equally likely to be chosen) and then toss the chosen coin repeatedly.

Show that the probability that the first $n$ tosses are all heads is

p m n 1 m 1 i 0 m i m n

Show that the conditional probability that toss $n 1$ is heads given that the previous $n$ tosses were all heads is

p m n 1 p m n

Interpret $p m n$ as an approximating sum for the integral $x 01 x n$ to show that

p m n 1 n 1 as m

Conclude that

p m n 1 p m n n 1 n 2 as m

The limiting conditional probability in the last exercise is called Laplace's Rule of Succession, named after Simon Laplace. This rule was used by Laplace and others as a general principle for estimating the conditional probability that an event will occur on time

n 1

, given that the event has occurred

n

times in succession.

Suppose that a missile has had 10 successful tests in a row. Compute Laplace's estimate that the $11 th$ test will be successful. Does this make sense?

5. Independence

Basic Theory

Independence of Two Events

General Independence of Events

Independence of Random Variables

Conditional Independence

An Interpretation of Conditional Probability

Examples and Applications

Bernoulli Trials

Cards

Dice

Coins

Buffon's Coin

Reliability

Diagnostic Testing

Genetics

Laplace's Rule of Succession