The Multinomial Distribution

A multinomial trials process is a sequence of independent, identically distributed random variables

X X 1 X 2

each taking

k

possible values. Thus, the multinomial trials process is a simple generalization of the Bernoulli trials process (which corresponds to

k 2

). For simplicity, we will denote the set of outcomes by

12 k

, and we will denote the common probability density function of the trial variables by

Of course

p i 0

for each

i

and

i 1 k p i 1

. In statistical terms, the sequence

X

is formed by sampling from the distribution.

As with our discussion of the binomial distribution, we are interested in the random variables that count the number of times each outcome occurred. Thus, let

Note that

i 1 k Y n i n

so if we know the values of

k 1

of the counting variables, we can find the value of the remaining counting variable. As with any counting variable, we can express

Y n i

as a sum of indicator variables:

Show that $Y n i j 1 n X j i$ .

Basic arguments using independence and combinatorics can be used to derive the joint, marginal, and conditional densities of the counting variables. In particular, recall the definition of the multinomial coefficient: for positive integers

j 1 j 2 j k

with

i 1 k j i n

Joint Distribution

Show that for positive integers $j 1 j 2 j k$ with $i 1 k j i n$ ,

Y n 1 j 1 Y n 2 j 2 Y n k j k n j 1 j 2 j k p 1 j 1 p 2 j 2 p k j k

The distribution of

Y n Y n 1 Y n 2 Y n k

is called the multinomial distribution with parameters

n

and

p p 1 p 2 p k

. We also say that

Y n 1 Y n 2 Y n k 1

has this distribution (recall that the values of

k 1

of the counting variables determine the value of the remaining variable). Usually, it is clear from context which meaning of the term multinomial distribution is intended. Again, the ordinary binomial distribution corresponds to

k 2

Marginal Distributions

Show that $Y n i$ has the binomial distribution with parameters $n$ and $p i$ :

Y n i j n j p i j 1 p i n j, j 01 n

Give a probabilistic proof, by defining an appropriate sequence of Bernoulli trials.
Give an analytic proof, using the joint probability density function.

Grouping

The multinomial distribution is preserved when the counting variables are combined. Specifically, suppose that

A 1 A 2 A m

. is a partition of the index set

12 k

into nonempty subsets. For

j 12 m

let

Show that $Z n Z n 1 Z n 2 Z n m$ has the multinomial distribution with parameters $n$ and $q 1 q 2 q m$ .

Give a probabilistic proof, by defining an appropriate sequence of multinomial trials.
Give an analytic proof, using the joint probability density function.

Conditional Distribution

The multinomial distribution is also preserved when some of the counting variables are observed. Specifically, suppose that

A B

is a partition of the index set

12 k

into nonempty subsets. Suppose that

j i i B

is a sequence of nonnegative integers, indexed by

B

such that

Show that the conditional distribution of $Y n i i A$ . given $Y n i j i i B$ . is multinomial with parameters $n j$ and $p i p i A$ .

Give a probabilistic proof, by defining an appropriate sequence of Bernoulli trials.
Give an analytic proof, based on the joint probability density function.

Combinations of the basic results in Exercise 4 and Exercise 5 can be used to compute any marginal or conditional distributions.

Moments

We will compute the mean, variance, covariance, and correlation of the counting variables. Results from the binomial distribution and the representation in terms of indicator variables are the main tools.

Show that

$Y n i n p i$ .
$Y n i n p i 1 p i$

Show that for distinct $i$ and $j$ ,

$Y n i Y n j n p i p j$ .
$Y n i Y n j p i p j 1 p i 1 p j$ .

From Exercise 7, note that the number of times outcome

i

occurs and the number of times outcome

j

occurs are negatively correlated, but the correlation does not depend on

n

k

. Does this seem reasonable?

Use the result of Exercise 7 to show that if $k 2$ , then the number of times outcome 1 occurs and the number of times outcome 2 occurs are perfectly correlated. Does this seem reasonable?

Examples and Applications

In the dice experiment, select the number of aces. For each die distribution, start with a single die and add dice one at a time, noting the shape of the density function and the size and location of the mean/standard deviation bar. When you get to 10 dice, run the simulation with an update frequency of 10. Note the apparent convergence of the relative frequency function to the density function, and the empirical moments to the distribution moments.

Suppose that we throw 10 standard, fair dice. Find the probability of each of the following events:

scores 1 and 6 occur once each and the other scores occur twice each.
scores 2 and 4 occur 3 times each.
there are 4 even scores and 6 odd scores.
scores 1 and 3 occur twice each given that score 2 occurs once and score 5 three times.

Suppose that we roll 4 ace-six flat dice (faces 1 and 6 have probability $1 4$ each; faces 2, 3, 4, and 5 have probability $1 8$ each). Find the joint probability density function of the number of times each score occurs.

In the dice experiment, select 4 ace-six flats. Run the experiment 500 times, updating after each run. Compute the joint relative frequency function of the number times each score occurs. Compare the relative frequency function with the true probability density function.

Suppose that we roll 20 ace-six flat dice. Find the covariance and correlation of the number of 1's and the number of 2's.

In the dice experiment, select 20 ace-six flat dice. Run the experiment 500 times, updating after each run. Compute the empirical covariance and correlation of the number of 1's and the number of 2's. Compare the results with the theoretical results computed in Problem 13.

5. The Multinomial Distribution

Basic Theory

Multinomial trials