Note: Matlab code for the stuff below can be downloaded here.
Correlation: an overview
Correlation is, simply put, the degree to which you can predict one variable by knowing another variable. This relationship is often interchangeably referred to as an association or a dependency. Two variables that are completely uncorrelated are said to be independent of one another.
A Pearson correlation coefficient is the most common way to quantify a correlation between two continuous variables (i.e., variables expressed on an interval scale). It is usually denoted \(r\), or \(r_{xy}\) for variables \(X\) and \(Y\), and takes on values from -1 to 1, where 0 indicates complete independence, 1 indicates that the two variables are identical, and -1 indicates that they are exactly inverse.
Correlations can best be visualised using a scatterplot, which plots the individual data points of the two variables on opposing axes. A scatterplot shows both the univariate distributions (also shown as histograms below) and the bivariate distribution (how the two variables are related to one another).
As you can probably see from the above figures, a perfect positive correlation (\(r=1.0\)) or negative correlation (\(r=-1.0\)) has all the data points lying on the diagonal line, slanting upwards or downwards, respectively. In these two extreme situations (which you will never see in naturally observed phenomena), knowledge of \(X\) allows us to predict \(Y\) with complete certainly. Put another way, \(X\) shares 100% of its variance with \(Y\). Indeed, if you square \(r_{xy}\), this will give you the percentage of shared variance between the two variables.
We can get a more intuitive understanding of Pearson coefficients if we convert our variables to Z-scores (or, equivalently, standardize them). A Z-score is obtained when we subtract the mean from all the values, and divide by the standard deviation:
\[ X_Z = \frac{X - \bar{X}}{s_x} \]
where \(\bar{X}\) is the mean of \(X\), and \(s_x\) is its standard deviation. When we transform our variables in this way, we now have a variable for which 0 is always the mean, and ±1 is always one standard deviation above or below the mean.
Pearson coefficients make use of an interesting property of Z-scores, which is that if you square them, sum the squares, and divide by the number of data points (\(N\)) less one, you get exactly 1:
\[ \frac{\sum{x_Z^2}}{N-1}=1.0 \]
(\(x_Z\) here is a single data point in \(X_Z\)). This also is what you get when you multiply two identical variables together (\(Y_Z=X_Z\)):
\[ \frac{\sum{x_Z y_Z}}{N-1}=\frac{\sum{x_Z x_Z}}{N-1}=\frac{\sum{x_Z^2}}{N-1}=1.0 \]
When your variables are inverse (\(Y_Z=-X_Z\)), on the other hand, this sum becomes:
\[ \frac{\sum{x_Z y_Z}}{N-1}=\frac{\sum{-x_Z x_Z}}{N-1}=-\frac{\sum{x_Z^2}}{N-1}=-1.0 \]
You hopefully recognize these as the extreme cases in the scatterplots above. Indeed, for any two variables that are not identical or inverse, the sum of their products will lie somewhere in between -1 and 1, and the closer they are to 0, the less similar they are to one another. This sum of products is actually the equation for a Pearson coefficient:
\[ r_{xy} = \frac{1}{N-1} \sum{x_Z y_Z} \]
For those of you who are less than fond of equations, we can instead look at the same plots as above, but this time after converting our variables to Z-scores:
Here, I've also drawn in some squares and rectangles, corresponding to three data points. The (signed) areas of these shapes are the products of the corresponding \(x_Z\) and \(y_Z\) values; green areas are positive and yellow areas are negative. Our Pearson coefficient is proportional to the sum of these areas. Notably, for \(r_{xy}=1.0\), these areas are always positive, because the two variables always have the same sign; conversely, for \(r_{xy}=-1.0\) they are always negative. For \(r_{xy}=0.5\), some of these areas will be negative and some positive.
Intuitively, this shows us that the more we spread our points into all four quadrants, the lower our absolute Pearson coefficient will be. As the positive and negative areas balance out, our correlation will approach zero. This coefficient thus does a good job of capturing the degree to which our two variables are associated with one another.
The spectre of confounding
While a correlation coefficient can inform us how strongly two variables are associated, the interpretation of this association is often tricky, due to the problem of confounding. A confounder is a third variable that is correlated with both your variables of interest. The existence of such a variable (let's call it \(C\)) makes it very difficult to say anything conclusive about the relationship between \(X\) and \(Y\).
Let's consider a less abstract example. Say we were interested in understanding whether performance in school leads to better incomes later in life. We collect data via a questionnaire that records people's household incomes (\(Y\)) along with their average grades (\(X\)) from secondary school. We then compute a Pearson correlation from these values: \(r_{xy} = 0.43\).
If we square this value, we get \(r_{xy}^2 = 0.185\), indicating that school performance explains 18.5% of later household income. Interesting, right? We might be tempted to conclude that performance in school leads (on average) to better incomes.
The problem with this is that there are likely many factors we haven't considered. Socioeconomic status (SES), for instance, can have a strong detrimental impact on the ability of students to develop their academic skills. This means that, for someone from a low-SES community, it is simply much more difficult to perform better in school, and subsequently to attain higher incomes. Compounding this problem, a low-SES background can itself impede one's ability to attain a higher income, for numerous reasons.
Failing to consider SES means that it is difficult for us to determine how much academic performance contributes to income, since much or even all of this association might be attributable to differing SES backgrounds. In other words, SES is a confounder.
So how do I deal with confounding?
To deal with this sort of confounding, we need a way to assess the association between education and income after first accounting for the relationships between SES and education, and SES and income. We can use so-called partial correlations to do this.
Partial correlations can be summed up in the following Venn diagram:
In this diagram, circles represent a variable's variance, and overlapping circles represent variance that is shared between variables (also called covariance). Our objective with partial correlations is to determine how much shared variance (if any) is left between education and income (cyan), after we remove the shared variance with SES (white).
There are several ways to compute partial correlations, which are detailed in the Wikipedia entry linked above. One straightforward way is to use a so-called correlation matrix. This is simply a square matrix (let's call it \(\textbf{K}\)) with one row/column for each variable we are interested in, where each element \(k_{ij}\) is the Pearson coefficient for variables \(i\) and \(j\).
To get our partial correlations, we perform what's called matrix inversion on \(\textbf{K}\), to get a "precision matrix" \(\textbf{P}\):
\[ \textbf{P} = \textbf{K}^{-1} \]
Each element \(p_{ij}\) of \(\textbf{P}\) can be used to compute our partial correlation \(r_{ij}^\prime\) between variables \(i\) and \(j\), which accounts (or "adjusts") for the shared variance of the other variables:
\[ r_{ij}^\prime = -p_{ij} / \sqrt{p_{ii} p_{jj}} \]
An example
So let's try this out with our example. Say that we also recorded SES (which we'll call \(C\)) in our questionnaire, and we compute the correlations \(r_{xc} = 0.5\) and \(r_{yc} = 0.39\). (Disclaimer: these are completely fabricated values!).
We can plug these values into a correlation matrix (note that \(k_{ij}\) is the same as \(k_{ji}\)), where the rows/columns are ordered \(X\), \(Y\), \(C\):
\[ \begin{bmatrix} 1.00 & 0.43 & 0.50\\ 0.43 & 1.00 & 0.39\\ 0.50 & 0.39 & 1.00 \end{bmatrix} \]
If we invert this matrix, we get the precision matrix \(\textbf{P}\):
\[ \begin{bmatrix} 1.46 & -0.41 & -0.57\\ -0.41 & 1.29 & -0.30\\ -0.57 & -0.30 & 1.40\\ \end{bmatrix} \]
And using our equation above, we get partial correlations \(r_{xy}^\prime=0.29\), \(r_{xc}^\prime=0.40\), and \(r_{yc}^\prime=0.22\). This tells us that, after accounting for SES, education now explains 8% of later income — \((r_{xy}^\prime)^2\) — compared to the 18.5% we estimated earlier.
Summing up
Correlations are a way for us to quantify how strongly two variables are associated with one another, but our ability to infer from this relationship is hampered by the existence of numerous confounders. If we can determine what these confounders are, we can measure them and use partial correlations to get a more accurate interpretation of the relationship between our two variables of interest.
Another way to deal with this problem is through multiple linear regression. In this framework, a single variable of interest (called a criterion or outcome variable) is predicted via a linear model by one or more other variables (called predictor variables). In our example above, the linear regression model would look like:
\[ Y = \beta_0 + \beta_1 X + \beta_2 C + \epsilon \]
Because multiple linear regression fits all \(\beta\) coefficients simultaneously, it has partial correlations built in. But that is a subject for a future blog post :-)
