Variance and Higher Moments

Recall that by taking the expected value of various transformations of a random variable, we can measure many interesting characteristics of the distribution of the variable. In this section, we will study expected values that measure spread, skewness and other properties.

Variance

Definitions

As usual, we start with a random experiment with probability measure

on an underlying sample space. Suppose that

X

is a random variable for the experiment, taking values in

S

. Recall that the expected value or mean of

X

gives the center of the distribution of

X

. The variance of

X

is a measure of the spread of the distribution about the mean and is defined by

Recall that the second moment of

X

about

a

X a 2

. Thus, the variance is the second moment of

X

about

μ X

, or equivalently, the second central moment of

X

. Second moments have a nice interpretation in physics, if we think of the distribution of

X

as a mass distribution in

. Then the second moment of

X

about

a

is the moment of inertia of the mass distribution about

a

. This is a measure of the resistance of the mass distribution to any change in its rotational motion about

a

. In particular, the variance of

X

is the moment of inertia of the mass distribution about the center of mass

μ

Suppose that $X$ has a discrete distribution with probability density function $f$ . Use the change of variables theorem to show that

X x S x X 2 f x

Suppose that $X$ has a continuous distribution with probability density function $f$ . Use the change of variables theorem to show that

X x S x X 2 f x

The standard deviation of

X

is the square root of the variance. It also measures dispersion about the mean but has the same physical units as the variable

X

Properties

The following exercises give some basic properties of variance, which in turn rely on basic properties of expected value:

Show that $X X 2 X 2$ .

Show that $X 0$ if and only if $X c 1$ for some constant $c$ .

Show that if $a$ and $b$ are constants then $a X b a 2 X$ .

Show that the random variable given below has mean 0 and variance 1:

Z X X X

The random variable

Z

in Exercise 7 is sometimes called the standard score associated with

X

. Since

X

and its mean and standard deviation all have the same physical units, the standard score

Z

is dimensionless. It measures the directed distance from

X

X

in terms of standard deviations.

On the other hand, when

X 0

, the ratio of standard deviation to mean is called the coefficient of variation. This quantity also is dimensionless, and is sometimes used to compare variability for random variables with different means.

Chebyshev's Inequality

Chebyshev's inequality (named after Pafnuty Chebyshev) gives an upper bound on the probability that a random variable will be more than a specified distance from its mean. This is often useful in applied problems where the distribution is unknown, but the mean and variance are at least approximately known. In the following two exercises, suppose that

X

is a real-valued random variable with mean

μ X

and standard deviation

σ X

Use Markov's inequality to prove Chebyshev's inequality:

X μ t σ 2 t 2 for t 0

Establish the following equivalent version of Chebyshev's inequality:

X μ k σ 1 k 2 for k 0

The usefulness of the Chebyshev inequality comes from the fact that it holds for any distribution (assuming only that the mean and variance exist). The tradeoff is that for many specific distributions, the Chebyshev bound is rather crude. Note in particular that in the last exercise, the bound is useless when

k 1

, since 1 is an upper bound for the probability of any event.

Examples and Applications

Indicator Variables

Suppose that $X$ is an indicator variable with $X 1 p$ .

Recall that $X p$ .
Show that $X p 1 p$ .
Sketch the graph of $X$ as a function of $p$ .

Note that the minimum value of

X

is 0, and occurs when

p 0

and

p 1

. The maximum value is

1 4

and occurs when

p 1 2

Uniform Distributions

Suppose that $X$ has the discrete uniform distribution on the integer interval $m m 1 n$ where $m n$ .

Recall that $X 1 2 m n$ .
Show that $X 1 12 n m n m 2$ .

Suppose that $X$ has the continuous uniform distribution on the interval $a b$ .

Recall that $X 1 2 a b$ .
Show that $X 1 12 b a 2$ .

Note that in both the discrete and continuous cases, the variance depends only on the length of the interval.

Dice

Recall that a standard die is a six-sided die. A fair die is one in which the faces are equally likely. An ace-six flat die is a standard die in which faces 1 and 6 have probability

1 4

each, and faces 2, 3, 4, and 5 have probability

1 8

each.

A standard, fair die is thrown. Find the mean, variance, and standard deviation of the score.

In the dice experiment, select one fair die. Run the experiment 1000 times, updating every 10 runs, and note the apparent convergence of the empirical mean and standard deviation to the distribution mean and standard deviation.

An ace-six flat die is thrown. Find the mean, variance and standard deviation.

In the dice experiment, select one ace-six flat die. Run the experiment 1000 times, updating every 10 runs, and note the apparent convergence of the empirical mean and standard deviation to the distribution mean and standard deviation.

The Poisson Distribution

where

a 0

is a parameter. The Poisson distribution is named after Simeon Poisson and is widely used to model the number of random points in a region of time or space; the parameter

a

is proportional to the size of the region. The Poisson distribution is studied in detail in the chapter on the Poisson Process.

Suppose that $N$ has the Poisson distribution with parameter $a$ .

Recall that $N a$ .
Show that $N a$ .

In the Poisson experiment, the parameter is $a r t$ . Vary the parameter and note the size and location of the mean-standard deviation bar. For selected values of the parameter, run the experiment 1000 times updating every 10 runs. Note the apparent convergence of the empirical mean and standard deviation to the distribution mean and standard deviation.

The Geometric Distribution

Recall that the geometric distribution on

is a discrete distribution with density function

where

p 01

is a parameter. The geometric distribution governs the trial number of the first success in a sequence of Bernoulli trials with success parameter

p

Suppose that $W$ has the geometric distribution with success parameter $p$ .

Recall that $W 1 p$ .
Show that $W 1 p p 2$ .

In the negative binomial experiment, set $k 1$ to get the geometric distribution . Vary $p$ with the scroll bar and note the size and location of the mean-standard deviation bar. For selected values of $p$ , run the experiment 1000 times updating every 10 runs. Note the apparent convergence of the empirical mean and standard deviation to the distribution mean and standard deviation.

Suppose that $W$ has the geometric distribution with parameter $p 3 4$ . Compute the true value and the Chebyshev bound for the probability that $Y$ is at least 2 standard deviations away from the mean.

The Exponential Distribution

Recall that the exponential distribution is a continuous distribution with probability density function

where

r 0

is the with rate parameter. This distribution is widely used to model failure times and other arrival times. The exponential distribution is studied in detail in the chapter on the Poisson Process.

Suppose that $X$ has the exponential distribution with rate parameter $r$ .

Recall that $X 1 r$ .
Show that $X 1 r$ .

Thus, for the exponential distribution, the mean and standard deviation are the same.

In the gamma experiment, set $k 1$ to get the exponential distribution. Vary $r$ with the scroll bar and note the size and location of the mean-standard deviation bar. For selected values of $r$ , run the experiment 1000 times updating every 10 runs. Note the apparent convergence of the empirical mean and standard deviation to the distribution mean and standard deviation.

Suppose that $X$ has the exponential distribution with rate parameter $r 0$ . Compute the true value and the Chebyshev bound for the probability that $X$ is at least $k$ standard deviations away from the mean.

The Pareto Distribution

Recall that the Pareto distribution is a continuous distribution with density function

where

a 0

is a parameter. The Pareto distribution is named for Vilfredo Pareto. It is a heavy-tailed distribution that is widely used to model financial variables such as income. The Pareto distribution is studied in detail in the chapter on Special Distributions.

Suppose that $X$ has the Pareto distribution with shape parameter $a$ .

Recall that $X a 01 a a 1 a 1$
Show that $X a 01 a 12 a a 1 2 a 2 a 2$

In the random variable experiment, select the Pareto distribution. Vary $a$ with the scroll bar and note the size and location of the mean/standard deviation bar. For each of the following values of $a$ , run the experiment 1000 times updating every 10 runs and note the behavior of the empirical mean and standard deviation.

$a 1$
$a 2$
$a 3$

The Normal Distribution

Recall that the standard normal distribution is a continuous distribution with density function

Normal distributions are widely used to model physical measurements subject to small, random errors and are studied in detail in the chapter on Special Distributions.

Suppose that $Z$ has the standard normal distribution.

Recall that $Z 0$ .
Show that $Z 1$ . Hint: Integrate by parts in the integral for $Z 2$ .

Suppose again that $Z$ has the standard normal distribution and that $μ$ , $σ 0$ . Recall that $X μ σ Z$ has the normal distribution with location parameter $μ$ and scale parameter $σ$ .

Recall that $X μ$
Show that $X σ 2$

Thus, as the notation suggests, the location parameter

μ

is also the mean and the scale parameter

σ

is also the standard deviation.

In the random variable experiment, select the normal distribution. Vary the parameters and note the shape and location of the mean-standard deviation bar. For selected parameter values, run the experiment 1000 times updating every 10 runs and note the apparent convergence of the empirical mean and standard deviation to the distribution mean and standard deviation.

Beta Distributions

The distributions in this subsection belong to the family of beta distributions, which are widely used to model random proportions and probabilities. The beta distribution is studied in detail in the chapter on Special Distributions.

Graph the density functions below and compute the mean and variance of each.

$f x 6 x 1 x, 0 x 1$
$f x 12 x 2 1 x, 0 x 1$
$f x 12 x 1 x 2, 0 x 1$
$f x 1 x 1 x, 0 x 1$ .

The particular beta distribution in part (d) is also known as the arcsine distribution.

Miscellaneous Exercises

Suppose that $X$ is a real-valued random variable with $X 5$ and $X 4$ . Find each of the following:

$3 X 2$
$X 2$

Suppose that $X 1$ and $X 2$ are independent, real-valued random variables with $X i μ i$ and $X i σ i$ for $i 12$ . Show that

X 1 X 2 σ 1 2 μ 1 2 σ 2 2 μ 2 2 μ 1 2 μ 2 2

Marilyn Vos Savant has an IQ of 228. Assuming that the distribution of IQ scores has mean 100 and standard deviation 15, find Marilyn's standard score.

Higher Moments

Suppose that

X

is a real-valued random variable. Recall again that the variance of

X

is the second moment of

X

about the mean, and measures the spread of the distribution of

X

about the mean. The third and fourth moments of

X

about the mean also measure interesting features of the distribution. The third moment measures skewness, the lack of symmetry, while the fourth moment measures kurtosis, the degree to which the distribution is peaked. The actual numerical measures of these characteristics are standardized to eliminate the physical units, by dividing by an appropriate power of the standard deviation. As usual, we assume that all expected values given below exist, and we will let

μ X

and

σ X

Skewness

Suppose that $X$ has a continuous distribution with probability density $f$ that is symmetric about $a$ : $f a t f a t, t$ .

Recall that $X a$ .
Show that $skew X 0$ .

Show that

skew X X 3 3 μ X 2 2 μ 3 σ 3

Kurtosis

Show that

kurt X X 4 4 μ X 3 6 μ 2 X 2 3 μ 4 σ 4

Exercises

Suppose that $X$ has uniform distribution on the interval $a b$ . Compute each of the following:

$skew X$
$kurt X$

Suppose that $X$ has the exponential distribution with rate parameter $r 0$ . Compute each of the following:

$skew X$
$kurt X$

Suppose that $X$ has the Pareto distribution with shape parameter $a 4$ . Compute each of the following:

$skew X$
$kurt X$

Suppose that $Z$ has the standard normal distribution. Compute each of the following:

$skew Z$
$kurt Z$

Graph the following density functions and compute the mean, variance, skewness and kurtosis of each. (The corresponding distributions are all members of the family of beta distributions).

$f x 6 x 1 x, 0 x 1$
$f x 12 x 2 1 x, 0 x 1$
$f x 12 x 1 x 2, 0 x 1$
$f x 1 x 1 x, 0 x 1$ , the arcsine distribution.

Vector Space Concepts

Variance and higher moments are related to the concept of norm and distance in the theory of vector spaces. This connection can help unify and illuminate some of the ideas.

Our vector space

V

consists of all real-valued random variables defined on a fixed probability space

Ω F

(that is, relative to a given random experiment). Recall that two random variables are equivalent if they are equal with probability 1. We consider two such random variables as the same vector, so that technically, our vector space consists of equivalence classes under this equivalence relation. The addition operator corresponds to the usual addition of two real-valued random variables, and the operation of scalar multiplication corresponds to the usual multiplication of a real-valued random variable by a real (non-random) number.

Norm

Let

X

be a real-valued random variable. For

k 1

, we define the

k

-norm by

Thus,

X k

is a measure of the size of

X

in a certain sense. The following exercises establish the fundamental properties.

Show that $X k 0$ for any $X$ .

Show that $X k 0$ if and only if $X 0 1$ (so that $X$ is equivalent to 0).

Show that $c X k c X k$ for any constant $c$ .

The next exercise gives Minkowski's inequality, named for Hermann Minkowski. It is also known as the triangle inequality.

Show that $X Y k X k Y k$ for any $X$ and $Y$ .

Show that $S x y 2 x 0 y 0$ is a convex set and that $g x y x 1 k y 1 k k$ is concave on $S$ .
Use (a) and Jensen's inequality to conclude that if $U$ and $V$ are nonnegative random variables then $U 1 k V 1 k k U 1 k V 1 k k$
In (b) let $U X k$ and $V Y k$ and then do some algebra.

It follows from Exercises 42-45 that the set of random variables with finite

k

moment forms a subspace of our parent vector space

V

, and that the

k

-norm really is a norm on this vector space:

Our next exercise gives Lyapunov's inequality, named for Aleksandr Lyapunov. This inequality shows that the

k

-norm of a random variable is increasing in

k

Show that if $j k$ then $X j X k$ .

Show that $S x x 0$ is convex, and $g x x k j$ is convex on $S$ .
Use part (a) and Jensen's inequality to conclude that if $U$ is a nonnegative random variable then $U k j U k j$ .
In (b), let $U X j$ and do some algebra.

Lyapunov's inequality shows that if

j k

and

X

has a finite

k

moment, then

X

has a finite

j

moment as well. Thus,

V k

is a subspace of

V j

Suppose that $X$ is uniformly distributed on the interval $01$ .

Find $X k$ .
Graph $X k$ as a function of $k$ .
Find $k X k$ .

Suppose that $X$ has probability density function $f x a x a 1, x 1$ , where $a 0$ is a parameter. Thus, $X$ has the Pareto distribution with shape parameter $a$ .

Find $X k$ .
Graph $X k$ as a function of $k a$ .
Find $k k a X k$ .

Suppose that $X Y$ has probability density function $f x y x y, 0 x 1, 0 y 1$ . Verify Minkowski's inequality.

Distance

The

k

-norm, like any norm on a vector space, can be used to measure distance; we simply compute the norm of the difference between two vectors. Thus, we define the

k

-distance (or

k

-metric) between real-valued random variables

X

and

Y

to be

The properties in the following exercises are analogies of the properties in Exercises 42-45 (and thus very little additional work should be required). These properties show that the

k

-metric really is a metric.

Show that $d k X Y 0$ for any $X$ , $Y$ .

Show that $d k X Y 0$ if and only if $Y X 1$ (so that $X$ and $Y$ are equivalent).

Show that $d k X Y d k X Z d k Z Y$ for any $X$ , $Y$ , $Z$ (this is known as the triangle inequality).

and the variance is the square of this. More generally, the

k

moment of

X

about

a

is simply the

k

power of the

k

-distance from

X

a

. The 2-distance is especially important for reasons that will become clear below and in the next section. This distance is also called the root mean square distance.

Center and Spread Revisited

Measures of center and measures of spread are best thought of together, in the context of a measure of distance. For a random variable

X

, we first try to find the constants

t

that are closest to

X

, as measured by the given distance; any such

t

is a measure of center relative to the distance. The minimum distance itself is the corresponding measure of spread.

Let us apply this procedure to the 2-distance. Thus, we define the root mean square error function by

Show that $d 2 X t$ is minimized when $t X$ and that the minimum value is $X$ .

Note that the minimum value of the function occurs at the same points as the minimum value of $X t 2$ (this is the mean square error function).
Expand $X t 2$ and take expected values term by term. The resulting expression is a quadratic function of $t$ .
Now use standard calculus.

The physical interpretation of this result is that the moment of inertia of the mass distribution of

X

about

t

is minimized when

t μ

, the center of mass.

In the histogram applet, construct a discrete distribution each of the types indicated below. Note the position and size of the mean ± standard deviation bar and the shape of the mean square error graph.

A uniform distribution.
A symmetric, unimodal distribution.
A unimodal distribution that is skewed right.
A unimodal distribution that is skewed left.
A symmetric bimodal distribution.
A U-shaped distribution.

Next, let us apply our procedure to the 1-distance. Thus, we define the mean absolute error function by

We will show that

d 1 X t

is minimized when

t

is any median of

X

. We start with a discrete case, because it's easier and has special interest.

Suppose that $X$ has a discrete distribution with values in a finite set $S$ .

Show that $X t t X X t X t X t$
Show that $X t a t t b t$ , where $a t 2 X t 1$ and where $b t X 2 X X t$
Argue that $X t$ is a continuous, piecewise linear function of $t$ , with corners at the values in $S$ . That is, the function is a linear spline.
Let $m$ be the smallest median of $X$ . Argue that if $t m$ and $t S$ , then the slope of the linear piece at $t$ is negative.
Let $M$ be the largest median of $X$ . Argue that if $t M$ and $t S$ , then the slope of the linear piece at $t$ is positive.
Argue that if $t m M$ then the slope of the linear piece at $t$ is 0.
Conclude that $X t$ is minimized for every $t$ in the median interval $m M$ .

The last exercise shows that mean absolute error has a couple of basic deficiencies as a measure of error:

Indeed, when

X

does not have a unique median, there is no compelling reason to choose one value in the median interval, as the measure of center, over any other value in the interval.

In the histogram applet, construct a distribution of each of the types indicated below. In each case, note the position and size of the boxplot and the shape of the mean absolute error graph.

A uniform distribution.
A symmetric, unimodal distribution.
A unimodal distribution that is skewed right.
A unimodal distribution that is skewed left.
A symmetric bimodal distribution
A U-shaped distribution.

Let $X$ be an indicator random variable with $X 1 p$ . Graph $d 1 X t X t$ as a function of $t$ in each of the cases below. In each case, find the minimum value of the function and the values of $t$ where the minimum occurs.

$p 1 2$
$p 1 2$
$p 1 2$

Suppose now that $X$ has a general distribution on . Show that $d 1 X t$ is minimized when $t$ is any median of $X$ .

Suppose that $s t$ . Compute the expected value over the events $X s$ , $s X t$ , and $X t$ , and then add and subtract appropriate stuff to show that $X t X s t s 2 X s 1 2 t X s X t$
Suppose that $t s$ . Use methods similar to those in (a) to show that $X t X s t s 2 X s 1 2 X t t X s$
Note that the last terms on the right in the equations in (a) and (b) are nonnegative. If we take $s$ to be a median of $X$ , then the middle terms on the right in the equations in (a) and (b) are also nonnegative.
Conclude that if $s$ is a median of $X$ and $t$ is any other number then $X t X s$ .

Convergence

Whenever we have a measure of distance, we automatically have a criterion for convergence. Let

X n, n 12

and

X

be real-valued random variables defined on the same sample space (that is, defined for the same random experiment). We say that

X n X

n

k

mean if

When

k 1

, we simply say that

X n X

n

in mean; when

k 2

, we say that

X n X

n

in mean square. These are the most important special cases.

Use Lyapunov's inequality to show that if $j k$ , then $X n X$ as $n$ in $k$ mean implies $X n X$ as $n$ in $j$ mean.

Our next sequence of exercises shows that convergence in mean is stronger than convergence in probability.

Use Markov's inequality to show that if $X n X$ as $n$ in mean, then $X n X$ as $n$ in probability.

The converse is not true. Moreover, convergence with probability 1 does not imply convergence in

k

mean and convergence in

k

mean does not imply convergence with probability 1. The next two exercises give some counterexamples.

Suppose that $X 1 X 2$ is a sequence of independent random variables with

X n n 3 1 n 2, X n 0 1 1 n 2, n

Use the first Borel-Cantelli lemma to show that $X n 0 >$ as $n$ with probability 1.
Show that $X n 0 >$ as $n$ in probability.
Show that $X n$ as $n$ .

Suppose that $X 1 X 2$ is a sequence of independent indicator random variables with

X n 1 1 n, X n 0 1 1 n, n

Use the second Borel-Cantelli lemma to show that $X n 0 for infinitely many n 1$ .
Use the second Borel-Cantelli lemma to show that $X n 1 for infinitely many n 1$ .
Show that $X n does not converge as n 1$
Show that $X n 0$ as $n$ in $k$ mean for every $k 1$ .

The implications in the various modes of convergence are shown below; no other implications hold in general.

2. Variance and Higher Moments

Variance

Definitions

Properties

Chebyshev's Inequality

Examples and Applications

Indicator Variables

Uniform Distributions

Dice

The Poisson Distribution

The Geometric Distribution

The Exponential Distribution

The Pareto Distribution

The Normal Distribution

Beta Distributions

Miscellaneous Exercises

Higher Moments

Skewness

Kurtosis

Exercises

Vector Space Concepts

Norm

Distance

Center and Spread Revisited

Convergence

Related Topics