1. Pearson correlation, beyond the number

Level 4: Correlation, regression and the myth of causality

Two assets with a correlation of 0.8. You take a handful of days of widespread panic out of the calculation and the number collapses to 0.3.

Neither of the two is the wrong number. Both are correct, and they describe different periods.

You already know correlation as a number between -1 and +1 that sums up whether two assets tend to move together. It is time now to open the box and see what it is made of.

The base ingredient: covariance

The base ingredient is covariance. It is built like this, for each pair of observations —the daily return of two shares, for example:

• You compute how far each variable departs from its own mean.
• You multiply those two deviations together.

The sign of the product says everything. If both variables tend to depart in the same direction —both above their mean on the same day, or both below— the product is positive. If they move in opposite directions, the product is negative.

The average of all those products is the covariance. Its sign already tells you whether the relationship is direct or inverse.

From covariance to correlation

The trouble with covariance is that its size depends on the units of each variable, which makes it useless for comparing different pairs.

A covariance of 40 may be enormous or insignificant, depending on whether we are talking about dollars, index points or percentages.

Pearson correlation solves this by dividing the covariance by the product of each variable's standard deviation. That is: correlation = covariance divided by (standard deviation of A times standard deviation of B).

That division normalizes the result so that it always lands between -1 and +1, no matter whether you are comparing the price of gold in dollars with an emerging equity index in its own currency.

It is, at bottom, a covariance without units.

Pearson only sees straight lines

Here comes what an introductory course does not usually explain. Pearson measures linear relationship exclusively: how closely the cloud of points resembles a straight line.

Suppose two variables have a perfectly deterministic but curved relationship, U-shaped or bell-shaped, say. The Pearson coefficient can come out close to 0 even though there is a total and predictable connection between them.

The conclusion is worth committing to memory: a low Pearson value does not certify independence. It only certifies the absence of a linear relationship.

Two ways of inflating a correlation

The first is extreme values, to which Pearson is very sensitive.

A handful of sessions of widespread panic, when everything falls at once, can inflate for months the coefficient computed between two assets that in normal circumstances are barely related. It is the case we opened with: from 0.8 to 0.3 depending on whether or not you include those days.

The second is computing the correlation on prices (levels) instead of on returns.

The prices of almost any asset with a long-term trend rise over time. So any two price series with an upward trend tend to show very high correlations simply because they share that trend, not because their daily movements are genuinely connected.

That is why quantitative finance almost always works with returns. They are far more stable over time, more "stationary", and they reflect the real day-to-day co-oscillation.

How much data is behind the number

As you saw in the statistics level, everything computed on a sample carries uncertainty. Correlation is no exception: its reliability depends on the sample size.

Compare two coefficients that are identical on paper:

• A correlation of 0.30 computed with 8 observations is almost pure noise and should not move any decision.
• The same figure of 0.30 computed with 400 observations is a far more solid signal.

The reason is that the margin of error of the estimate shrinks as the number of data points grows.

It is not a law of physics

Finally, it is worth remembering that correlation is computed over a particular time window and changes over time.

The correlation between shares and bonds, for example, has been negative for long stretches and positive in others, depending on the economic regime.

When you work with many assets at once, you usually build a correlation matrix: a 1 on the diagonal, which is each asset with itself, and the cross coefficients in the remaining cells.

That matrix is the basis for measuring how much real diversification a portfolio provides, beyond looking at isolated pairs.