1. The key assumptions of regression

Level 7: Rigorous Statistical Diagnostics

Estimating a regression is trivial: three lines of Python and you have a beta. The hard part, and what separates a serious analyst from someone who just draws lines, is knowing under what conditions that beta means anything at all. Every piece of statistical machinery that comes afterwards (standard errors, t statistics, p-values, confidence intervals) rests on a set of assumptions. If they break, the numbers still appear on screen, but they stop meaning what you think they mean.

We work with random samples of the error, of the explanatory variable and of the dependent variable. The usual assumptions are numbered UR.1 through UR.4.

UR.1 — The data generating process is linear in the coefficients. The population is described as: Y equals beta zero plus beta one times X plus the error. Watch the word linear: it refers to the coefficients, not to the variables. This is also a linear model:

# log(Y) = beta0 + beta1 * (1/X) + error
# If we define U = log(Y) and V = 1/X, we get U = beta0 + beta1*V + error
# It is still linear in beta0 and beta1, and therefore estimable by OLS.
import numpy as np
data['U'] = np.log(data['Y'])
data['V'] = 1 / data['X']

By contrast, a model where the coefficient is squared or sits inside an exponential is no longer linear in the parameters and OLS does not apply. Note something important as well: in a linear model the error and Y are always dependent (Y contains the error), so their covariance is never zero. What may or may not happen is that the error depends on X. That is where the next requirements come from.

UR.2 — The error has zero expectation conditional on any value of the explanatory variable. Whatever the observations of X, the errors average out to zero. This also implies that the expectation of the product of the error and X is zero, and that the unconditional expectation of the error is zero. A typical case is X and the error being independent with zero mean. The practical consequence is beautiful: the expectation of Y given X is exactly beta zero plus beta one times X, that is, the regression line coincides with the conditional mean of Y. And the other way round: if the covariance between X and the error is not zero, then the conditional expectation of the error cannot be zero and UR.2 breaks. In finance this happens constantly: if your explanatory factor is contaminated by something that also moves the asset and that you have left out, the beta you estimate is biased.

UR.3 — Constant variance and no autocorrelation. The variance of the error conditional on the X values is the same for every observation (homoskedasticity) and the covariance between errors of different observations is zero (no autocorrelation). Compactly: the variance-covariance matrix of the error vector is the identity matrix multiplied by a constant, the error variance. A diagonal filled with the same variance, zeros off the diagonal.

UR.4 (optional) — The errors are normal. Conditional on the X values, they follow a normal distribution with zero mean and that same covariance matrix. It is optional in the sense that it is not needed for OLS to work well, but it greatly simplifies the properties of the estimators and it is what allows exact intervals and tests in small samples.