5. Goodness of fit and the traps of R squared

Level 7: Rigorous Statistical Diagnostics

To predict Y well we want the independent variable X to explain as much of the variation in Y as possible (hence the name explanatory variable). Ideally, the variance of X helps explain the variance of Y. So we need a way to measure how good our model is, a goodness-of-fit measure that quantifies how well the estimated model fits the data.

The best known is the coefficient of determination, R squared: the ratio of explained variation to total variation; equivalently, one minus the ratio of the sum of squared residuals to total variation.

# R squared = ESS / TSS = 1 - RSS / TSS
#   TSS: total variation of Y around its mean
#   ESS: variation explained by the model
#   RSS: sum of squared residuals
model.rsquared
model.rsquared_adj

The closer it is to 1, the closer the observed values of Y are to the fitted values. If it equals exactly 1, every point lies on the estimated line and the fit is perfect. If Y and X have no linear relationship at all, in a simple regression it would be 0. Between 0 and 1 it is read as the proportion of the variation of Y around its mean explained by the model: an R squared of 0.17 means that 17% of the variation of Y is explained by X.

So far, what everyone knows. Now the warnings, which are what really matters:

• It is not valid to compare the R squared of models with different transformations of the dependent variable. A model with Y and one with the logarithm of Y are not comparable by R squared, because the total variation used as a reference is different. It is valid, however, to compare a linear model with a linear-log model when both share the same Y.
• R squared does not measure predictive ability. A linear model can fit the sample very well and produce forecasts with no economic sense at all (for instance, negative wages for few years of education).
• It is computed on the sample data, so it says nothing about whether the model resembles the true data generating process.
• It can be low for reasons unrelated to the model: if the error variance is large, or if the variance of X is small.
• It can be high even when the model is wrong. If the true relationship is non-linear, a linear model may show a higher R squared than the quadratic or logarithmic one.
• It does not depend on the units of measurement. Switching from dollars to thousands of dollars leaves it unchanged.

The professor's conclusion is blunt: under no circumstances should a model be chosen on the basis of R squared alone.

In finance this matters a lot. A CAPM regression on an individual stock usually gives a modest R squared, of the order of 0.2, and that is perfectly normal: almost all the movement of a single stock is idiosyncratic. Conversely, an R squared of 0.99 in a regression on prices (not returns) is not good news but the usual signature of a spurious regression between trending series. Adjusted R squared penalizes the number of regressors and is preferable when comparing models with different numbers of factors, because plain R squared never falls when you add variables, useful or not.