3. Simple linear regression: finding the best line

Level 4: Correlation, regression and the myth of causality

The sector rose 2%. The model predicted 2.65% for the airline. The airline did 1.9%.

That gap of 0.75 percentage points is not a failure of the method. It is one of the most useful pieces of information a regression produces.

From describing to estimating

Correlation tells you whether two variables move together and how strongly. What it does not give you is a tool for estimating one from the other.

That is exactly what simple linear regression does. It looks for the line that best sums up how one variable —the dependent one— behaves as a function of another —the independent one—, so that you can use it to describe, and to some extent predict, their joint behavior.

The two numbers that define the line

Take a concrete example. You want to describe how the daily return of an airline moves as a function of the daily return of the transport sector index, using 60 sessions of data.

The line the regression looks for has this form, in words:

estimated return of the share = a base value (called the intercept) plus a slope multiplied by the sector return that same day.

The intercept and the slope are the only two numbers that completely define that line.

Least squares

To find that particular line among all the possible ones, regression uses the criterion of least squares.

For each of the 60 days, the line predicts a return. That predicted value almost never coincides exactly with the real observed return, and the difference between the two is called the residual, or error.

The regression chooses, out of all imaginable lines, the one that makes the sum of those residuals squared as small as possible.

Why square them instead of just adding the errors up as they are? For three reasons:

• If you added the errors up without squaring them, the positive and the negative ones would cancel each other out, giving a false sense of good fit even with a dreadful line.
• Squaring penalizes large errors far more than small ones, which pushes the line above all to avoid huge deviations.
• It turns out to be mathematically more tractable to compute.

The line, with numbers

Let us look at a numerical example. Suppose the regression, applied to those 60 sessions, yields this line:

estimated return of the airline = 0.05% + 1.3 times the sector return.

Read it in three scenarios:

• On a day when the sector does not move (0%), the model predicts that the airline returns 0.05% on average. That small base value is the part of the return that the model attributes to the share itself, quite apart from what the sector does.
• On a day when the sector rises 2%, the model predicts 0.05% + 1.3 times 2%, that is, 2.65%.
• If on that day the share actually returned 1.9%, the residual for that session is 1.9% minus 2.65%, that is, -0.75 percentage points.

That residual is the part of that day's move which the model failed to explain.

Residuals are information

It is important to understand that the fitted line almost never passes exactly through every data point. Nor does it need to: its aim is not to memorize the sample, but to sum up the general tendency of the relationship.

And what falls outside the line, the residuals, is not a failure of the method. It is information.

It is the part of the share's behavior that responds to company-specific factors, particular news items or plain noise, and not to the sector's movement.

Look at the chart before believing the line

One last caution. Simple linear regression assumes that the relationship between the two variables is reasonably straight.

If, when you plot the points on a chart, the cloud traces a pronounced curve, the least squares procedure will still hand you "the best possible line" in mathematical terms.

But that line may describe the real underlying relationship badly, and lead to mistaken conclusions if it is used to predict.

Before trusting a regression, it is worth looking at the scatter plot, not just the final number.