Sample Covariance and Correlation

Suppose again that we have a basic random experiment, and that

X

and

Y

are real-valued random variables for the experiment. Equivalently,

X Y

is a random vector taking values in

2

. Please recall the basic properties of the means,

X

and

Y

, the variances,

X

and

Y

and the covariance

X Y

. In particular, recall that the correlation is

Now suppose that we run the basic experiment

n

times. This creates a compound experiment with a sequence of independent random vectors

X 1 Y 1 X 2 Y 2 X n Y n

each with the same distribution as

X Y

. In statistical terms, this is a random sample of size

n

from the distribution of

X Y

. As usual, we will let

X X 1 X 2 X n

denote the sequence of first coordinates; this is a random sample of size

n

from the distribution of

X

. Similarly, we will let

Y Y 1 Y 2 X n

denote the sequence of second coordinates; this is a random sample of size

n

from the distribution of

Y

Recall that the sample means and sample variances for

X

are defined as follows (and of course analogous definitions hold for

Y

):.

In this section, we will define and study statistics that are natural estimators of the distribution covariance and correlation. These statistics will be measures of the linear relationship of the sample points in the plane. As usual, the definitions depend on what other parameters are known and unknown.

A Special Sample Covariance

Suppose first that the distribution means

X

and

Y

are known. This is usually an unrealistic assumption, of course, but is still a good place to start because the analysis is very simple and the results we obtain will be useful below. A natural estimator of

X Y

in this case is

Show that $W X Y$ is the sample mean for a random sample of size $n$ from the distribution of $X X Y Y$ .

Use the result of Exercise 1 to show that

$W X Y X Y$
$W X Y 1 n d X Y 2 X Y$
$W X Y X Y$ as $n$ with probability 1.

Properties

The formula in the following exercise is sometimes better than the definition for computational purposes.

With $X Y$ defined to be the sequence $X 1 Y 1 X 2 Y 2 X n Y n$ , show that

W X Y M X Y - M X Y - M Y X + X Y

The properties established in the following exercises are analogies of properties for the distribution covariance

Show that $W X X W 2 X$

Show that $W X Y W Y X$

Show that if $a$ is a constant then $W a X Y a W X Y$

Show that $W X Y Z W X Z W Y Z$

The following exercise gives a formula for the sample variance of a sum. The result extends naturally to larger sums.

Show that $W 2 X Y W 2 X W 2 Y 2 W X Y$

The Standard Sample Covariance

Consider now the more realistic assumption that the distribution means

X

and

Y

are unknown. A natural approach in this case is to average

X i M X Y i M Y

over

i 12 n

. But rather than dividing by

n

in our average, we should divide by whatever constant gives an unbiased estimator of

X Y

Interpret the sign of $X i M X Y i M Y$ geometrically, in terms of the scatterplot of points and its center.

Derivation

Use the bilinearity of the covariance operator to show that

M X M Y X Y n

Expand and sum term by term to show that

i 1 n X i M X Y i M Y i 1 n X i Y i n M X M Y

Use the result of Exercises 10 and 11, and basic properties of expected value, to show that

i 1 n X i M X Y i M Y n 1 X Y

Therefore, to have an unbiased estimator of

X Y

, we should define the sample covariance to be the random variable

As with the sample variance, when the sample size

n

is large, it makes little difference whether we divide by

n

n 1

Properties

The formula in the following exercise is sometimes better than the definition for computational purposes.

With $X Y$ defined as in Exercise 3, show that

S X Y 1 n 1 i 1 n X i Y i n n 1 M X M Y n n 1 M X Y M X M Y

Use the result of the previous exercise and the strong law of large numbers to show that $S X Y X Y$ as $n$ with probability 1.

The properties established in the following exercises are analogies of properties for the distribution covariance

Show that $S X X S 2 X$

Show that $S X Y S > Y X$

Show that if $a$ is a constant then $S a X Y a S X Y$

Show that $S X Y Z S X Z S Y Z$

Show that

S X Y n n 1 W X Y M X X M Y Y

The following exercise gives a formula for the sample variance of a sum. The result extends naturally to larger sums.

Show that $S 2 X Y S 2 X S 2 Y 2 S X Y$

Variance

In this subsection we will derive the following formuala for the variance of the sample covariance. The derivation was contributed by Ranjith Unnikrishnan, and is similar to the derivation of the variance of the sample variance.

Verify the following result. Hint: Start with the expression on the right. Expand the product $X i X j Y i Y j$ , and take the sums term by term.

S X Y 1 2 n n 1 i 1 n j 1 n X i X j Y i Y j

It follows that

S X Y

is the sum of all of the pairwise covariances of the terms in the expansion of Exercise 21.

Now, derive the formula for $S X Y$ by showing that

$X i X j Y i Y j X k X l Y k Y l 0$ if $i j$ or $k l$ or $i$ , $j$ , $k$ , $l$ are distinct.
$X i X j Y i Y j X i X j Y i Y j 2 d X Y 2 X Y$ if $i j$ , and there are $2 n n 1$ such terms in the sum of covariances.
$X i X j Y i Y j X k X j Y k Y j d X Y cov 2 X Y$ if $i$ , $j$ , $k$ are distinct, and there are $4 n n 1 n 2$ such terms in the sum of covariances.

Show that $S X Y W X Y$ . Does this seem reasonable?

Show that $S X Y 0$ as $n$ . Thus, the sample covariance is a consistent estimator of the distribution covariance.

Sample Correlation

By analogy with the distribution correlation, the sample correlation is obtained by dividing the sample covariance by the product of the sample standard deviations:

Use the strong law of large numbers to show that $R X Y X Y$ as $n$ with probability 1.

Click in the interactive scatterplot to define 20 points and try to come as close as possible to the following conditions: sample means 0, sample standard deviations 1, sample correlation as follows: 0, 0.5, −0.5, 0.7, −0.7, 0.9, −0.9.

Click in the interactive scatterplot to define 20 points and try to come as close as possible to the following conditions: $X$ sample mean 1, $Y$ sample mean 3, $X$ sample standard deviation 2, $Y$ sample standard deviation 1, sample correlation as follows: 0, 0.5, −0.5, 0.7, −0.7, 0.9, −0.9.

The Best Linear Predictor

The Distribution Version

Recall that in the section on (distribution) correlation and regression, we showed that the best linear predictor of

Y

based on

X

, in the sense of minimizing mean square error, is the random variable

The Sample Version

Of course, in real applications, we are unlikely to know the distribution parameters

X

Y

X

, and

X Y

. Thus, in this section, we are interested in the problem of estimating the best linear predictor of

Y

based on

X

from our random sample

X 1 Y 1 X 2 Y 2 X n Y n

. One natural approach is to find the line

y A x B

that fits the sample points best. This is a basic and important problem in many areas of mathematics, not just statistics. The term best means that we want to find the line (that is, find

A

and

B

) that minimizes the average of the squared errors between the actual

y

values in our data and the predicted

y

values:

Show that MSE is minimized for

A X Y S X Y S 2 X, B X Y M Y S > X Y S 2 X M X

and thus the sample regression line is

y M Y S X Y S 2 X x M X

Show that the minimum mean square error, using the coefficients in the previous exercise, is

MSE A X Y B X Y S 2 Y 1 R 2 X Y

Use the result of the previous exercise to show that

$1 R X Y 1$
$R X Y 1$ if and only if the sample points lie on a line with negative slope.
$R X Y 1$ if and only if the sample points lie on a line with positive slope.

Thus, the sample correlation measures the degree of linearity of the sample points. The results in the previous exercise can also be obtained by noting that the sample correlation is simply the correlation of the empirical distribution. Of course, properties (a), (b), and (c) are known for the distribution correlation.

The fact that the results in Exercise 28 and Exercise 29 are the sample analogies of the corresponding distribution results is beautiful and reassuring. Note that the sample regression line passes through

M X M Y

, the center of the empirical distribution. Naturally, the coefficients of the sample regression line can be viewed as estimators of the respective coefficients in the distribution regression line.

Assuming that the appropriate higher order moments are finite, use the law of large numbers to show that, with probability 1, the coefficients of the sample regression line converge to the coefficients of the distribution regression line:

S X Y S 2 X X Y X as n

M Y S X Y S 2 X M X Y X Y X X as n

As with the distribution regression lines, the choice of predictor and response variables is important.

Show that the sample regression line for $Y$ based on $X$ and the sample regression line for $X$ based on $Y$ are not the same line, except in the trivial case where the sample points all lie on a line.

is the sample mean

M Y

, and the minimum value of the mean square error is the sample variance

S 2 Y

. Thus, the difference between this value of the mean square error and the one in Exercise 29, namely

S 2 Y R 2 X Y

is the reduction in the variability of the

Y

data when the linear term in

X

is added to the predictor. The fractional reduction is

R 2 X Y

, and hence this statistics is called the (sample) coefficient of determination.

Exercises

Simulation Exercises

Click in the interactive scatterplot, in various places, and watch how the regression line changes.

Click in the interactive scatterplot to define 20 points. Try to generate a scatterplot in which the mean of the $x$ values is 0, the standard deviation of the $x$ values is 1, and in which the regression line has

slope 1, intercept 1
slope 3, intercept 0
slope $2$ , intercept 1

Click in the interactive scatterplot to define 20 points with the following properties: the mean of the $x$ values is 1, the mean of the $y$ values is 1, and the regression line has slope 1 and intercept 2.

If you had a difficult time with the previous exercise, it's because the conditions imposed are impossible to satisfy!

Run the bivariate uniform experiment 2000 times, with an update frequency of 10, in each of the following cases. Note the apparent convergence of the sample means to the distribution means, the sample standard deviations to the distribution standard deviations, the sample correlation to the distribution correlation, and the sample regression line to distribution regression line.

The uniform distribution on the square
The uniform distribution on the triangle.
The uniform distribution on the circle.

Run the bivariate normal experiment 2000 times, with an update frequency of 10, in each of the following cases. Note the apparent convergence of the sample means to the distribution means, the sample standard deviations to the distribution standard deviations, the sample correlation to the distribution correlation, and the sample regression line to the distribution regression line.

$X 1$ , $Y 2$ , $X Y 0.5$
$X 1.5$ , $Y 0.5$ , $X Y 0.7$

Data Analysis Exercises

Compute the correlation between petal length and petal width for the following cases in Fisher's iris data. Comment on the differences.

All cases
Setosa only
Verginica only
Versicolor only

Compute the correlation between each pair of color count variables in the M&M data

Consider all cases in Fisher's iris data.

Compute the least squares regression line with petal length as the predictor variable and petal width as the response variable.
Draw the scatterplot and the regression line together.
Predict the petal width of an iris with petal length 40

Consider the Setosa cases only in Fisher's iris data.

Compute the least squares regression line with sepal length as the predictor variable and sepal width as the unknown variable.
Draw the scatterplot and regression line together.
Predict the sepal width of an iris with sepal length 45.

7. Sample Covariance and Correlation

The Bivariate Model

A Special Sample Covariance

Properties

The Standard Sample Covariance

Derivation

Properties

Variance

Sample Correlation

The Best Linear Predictor

The Distribution Version

The Sample Version

Exercises

Simulation Exercises

Data Analysis Exercises