7. Overfitting in regression: why more variables do not always help
Level 4: Correlation, regression and the myth of causality
Take a model with an R² of 40% and add twenty series of random numbers to it, with no relation whatsoever to the company. R² can climb to 55% or 60%.
You have learned nothing new about the share. You have merely given the model twenty more ways of memorizing the sample.
The same disease as the backtest
You already saw in the market theory course that a trading strategy can look brilliant in the backtest and fail live. The cause: it has been fitted excessively to the specific noise of the historical data used to design it.
Statistical regression suffers exactly the same disease, with a different symptom. Instead of a trading rule that memorizes the past, it is a mathematical model that memorizes patterns which will not recur.
The mechanical fact
There is a mechanical fact behind this problem that is worth internalizing.
Adding any additional variable to a regression, whether or not it is economically relevant, can never make R² fall. And it almost always makes it rise at least a little.
The reason is purely mathematical: each new variable gives the model one more degree of freedom with which to fit the exact peculiarities of the sample in front of you, whether or not it finds a real relationship.
Let us go back to the opening example. A three-factor model to explain a share's returns reaches an R² of 40%. You add twenty series of numbers with no economic meaning at all, literally random noise.
R² can climb mechanically to 55% or 60%. Not because those twenty series carry real information, but because the model now has twenty additional degrees of freedom with which to fit the particular whims of that historical sample.
Adjusted R²
This mechanical property of R² motivated the creation of adjusted R².
It is a version of the indicator that explicitly penalizes the number of variables included, relative to the number of observations available.
Unlike ordinary R², adjusted R² can indeed fall when a useless variable is added. That makes it a far more honest tool for deciding whether a new variable deserves to stay in the model.
Almost as many parameters as data points
The size of the sample relative to the number of variables matters a great deal.
Imagine you have only 25 days of data and you test 15 candidate factors. The model has almost as many parameters to fit as it has observations with which to fit them.
In that scenario it may end up matching the sample almost perfectly without having captured any genuine relationship. And it will fail completely as soon as it faces new data.
It is the classic trap of overfitting through a small sample.
Two known aggravating factors
Multicollinearity, already seen in module 5, makes this problem worse still when many candidate factors correlated with one another coexist.
The fitting procedure can assign highly unstable coefficients, which change sign at the slightest shift in the data window used. It is a clear symptom that the model is fitting noise instead of signal.
And there is also the statistical version of the data snooping already seen in backtesting: testing dozens of different combinations of factors on the same history until one turns up with a strikingly high R².
By pure probability, among enough combinations tested, one will end up looking excellent in the sample, without that guaranteeing anything about its future behavior.
The discipline that corrects it
The discipline that corrects this problem is the same one you already know from backtesting, now applied to statistical models. There are two steps:
- Setting aside part of the data as a validation sample, completely apart from the process of searching for variables.
- Evaluating the final model once only on that never-seen data.
To that is added a principle of parsimony. Faced with two models of similar explanatory power, always prefer the one that uses fewer variables, each backed by a clear economic reason, over the one that simply maximizes in-sample R².