Introduction

In this chapter, we will study a number of parametric families of distributions that have special importance in probability. In some cases, a distribution may be important because it occurs as the limit of other distributions. In some cases, a parametric family may be important because it can be used to model a wide variety of random phenomena. In turn, this is usually the case because the family has a rich collection of densities with a small number of parameters (usually 1 or 2). As a general philosophical principal, we try to model a random process with as few parameters as possible; this is sometimes referred to as the principal of parsimony of parameters. In turn, this is a special case of Ockham's razor, named in honor of William of Ockham, the principle that states that one should use the simplest model that adequately describes a given phenomenon.

There are several other parametric families of distributions that are studied elsewhere in this project, because the natural home for these distributions are various random processes. These include

Before we begin our study of special parametric families of distributions, we will study two general parametric families. Many of the special parametric families studied in this chapter belong to one or both of these general families.

Location-Scale Families

Suppose that a real-valued random variable $Z$ has a continuous distribution with probability density function $g$ and distribution function $G$ . Let $a$ , $b$ be constants with $b 0$ . Show that $X a b Z$ has probability density function $f$ and distribution function $F$ given by

$F x G x a b, x$
$f x 1 b g x a b, x$

This two-parameter family of distributions is called the location-scale family associated with the given distribution;

a

is called the location parameter and

b

the scale parameter. In the special case that

b 1

, the one-parameter family is called the location family associated with the given distribution, and in the special case that

a 0

, the one-parameter family is called the scale family associated with the given distribution.

Interpret the location and scale parameters graphically:

For the location family associated with $g$ , show that the graph of $f$ is obtained by shifting the graph of $g$ , $a$ units to the right if $a 0$ and $a$ units to the left if $a 0$ .
For the scale family associated with $g$ , show that if $b 1$ , the graph of $f$ is obtained from the graph of $g$ by stretching horizontally and compressing vertically, by a factor of $b$ . If $0 b 1$ , the graph of $f$ is obtained from the graph of $g$ by compressing horizontally and stretching vertically, by a factor of $b$ .

Show that if $Z$ has a mode at $z$ , then $X$ has a mode at $x a b z$ .

Show that

$F p a b G p, p 01$
If $z$ is a quantile of order $p$ for $Z$ then $x a b z$ is a quantile of order $p$ for $X$ .

Show that the uniform distribution on the interval $a a b$ , where $a$ and $b 0$ are parameters, is a location-scale family.

Let $g z z, z 0$ This is the probability density function of the exponential distribution with parameter 1.

Find the location-scale family of probability density functions associated with $g$ .
Sketch the graphs.

The distributions in the previous exercise are the two-parameter exponential distributions.

Let $g z 1 1 z 2, z$ . This is the probability density function of the Cauchy distribution, named after Augustin Cauchy.

Find the location-scale family of probability density functions.
Sketch the graphs.

Show that

$E X a b E Z$
$var X b 2 var Z$

Suppose that $Z$ has moment generating function $M$ . Show that $X$ has moment generating function $N$ given by

N t a t M b t

Two probability distributions on

are said to be of the same type if they are related by a location-scale transformation. Specifically, if the distributions have distribution functions

F

and

G

, respectively, then the distributions are of the same type if there exist constants

a

b

with

b 0

, such that

Show that being of the same type is an equivalence relation on the collection of probability distributions on .

Exponential Families

Suppose that

X

is random variable taking values in

S

, and that the distribution of

X

depends on an unspecified parameter

θ

taking values in a parameter space

Θ

. In general, both

X

and

θ

may be vector-valued. Let

f θ

denote the probability density function of

X

S

, corresponding to

θ Θ

The distribution of

X

is a

k

-parameter exponential family if

S

does not depend on

θ

and if the probability density function can be written as

where

α

and

β 1 β 2 β k

are real-valued functions on

Θ

, and where

g

and

h 1 h 2 h k

are real-valued functions on

S

. Moreover,

k

is assumed to be the smallest such integer. The parameters

β 1 θ β 2 θ β k θ

are sometimes called natural parameters of the distribution, and the random variables

h 1 X h 2 X h k X

are sometimes called natural statistics of the distribution. Although the definition may look intimidating, exponential families are useful because they have many nice mathematical properties, and because many special parametric families turn out to be exponential families.

Suppose that $X$ has the binomial distribution with parameters $n$ and $p$ , where $n$ is fixed and $p 01$ . Show that the distribution is a one-parameter exponential family with natural parameter $p 1 p$ and natural statistic $X$ . Note that the natural parameter is the logarithm of the odds ratio corresponding to $p$ . This function is sometimes called the logit function.

Suppose that $X$ has the Poisson distribution with parameter $a 0$ . Show that the distribution is a one-parameter exponential family with natural parameter $a$ and natural statistic $X$ .

Suppose that $X$ has the negative binomial distribution with parameters $k$ and $p$ , where $k$ is fixed and $p 01$ . Show that the distribution is a one-parameter exponential family with natural parameter $1 p$ and natural statistic $X$ .

In many cases, the distribution of a random variable

X

will fail to be an exponential family if the support set

x S f θ x 0

depends on the parameter

θ

Suppose that $X$ has the uniform distribution on $0 a$ where $a 0$ . Show that the distribution of $X$ is not an exponential family.

The next exercise shows that if we sample from the distribution of an exponential family, then the distribution of the random sample is itself an exponential family with the same natural statistics.

Suppose that the distribution of random variable $X$ is a $k$ -parameter exponential family with natural parameters $β 1 β 2 β k$ , and natural statistics $h 1 X h 2 X h k X$ . Let $X X 1 X 2 X n$ be a sequence of $n$ independent random variables, each with the same distribution as $X$ . Show that $X$ is a $k$ -parameter exponential family with natural parameters $β 1 β 2 β k$ , and natural statistics

u j X i 1 n h j X i, j 12 k

1. Introduction

Location-Scale Families

Exponential Families