2. Properties of OLS and the Gauss-Markov theorem

Level 7: Rigorous Statistical Diagnostics

Before discussing why OLS is good, it is worth separating two things that everyday language tends to blur. An estimator is a rule that can be applied to any sample of data to produce a number. An estimate is the specific number that comes out when you apply that rule to a specific sample. The beta in your results table is an estimate; the least squares formula is the estimator. And because the estimator is applied to a random sample, the estimator is itself a random variable: it has a mean, a variance and a distribution. All statistical analysis consists of studying those three things.

Three properties follow from the construction of OLS itself and hold always, sample after sample, without needing any assumption:

1. The sum of the OLS residuals is zero (and therefore so is their sample mean).

2. The sample covariance between each regressor and the residuals is zero: the sum of the products of each X by its residual is zero.

3. The point formed by the mean of X and the mean of Y always lies on the estimated line: evaluate the line at the mean of X and you get exactly the mean of Y.

We can check this on an already estimated factor model:

# model is an already fitted result, for example a factor model
model.resid.plot()          # the residual cloud over time
model.resid.mean()          # something of the order of 1e-18: numerically zero

# Sample covariance between the market regressor and the residuals
sum(model.resid.values * df_stock_factor['MKT'].values)   # ~ 0
import numpy as np
np.cov(model.resid.values, df_stock_factor['MKT'].values)  # off-diagonal ~ 0

Both the sum and the covariance come out essentially zero. It is important to understand that this is not a validation of the model: they are algebraic identities that would hold just as well for an absurd model. If someone shows you a residual mean of zero as proof that their model is sound, they have shown you nothing.

What does justify using OLS rather than any other estimator is the Gauss-Markov theorem: under assumptions UR.1 to UR.3, the OLS estimator is BLUE (best linear unbiased estimator) and consistent. Let us unpack the acronym.

Linear. Written in matrix notation, the model relates the vector Y to the matrix X linearly, and the resulting estimator is a linear combination of the observations of Y.

Unbiased. One can show that the estimated beta equals the true beta plus a term that depends on the error. Taking expectations and applying the law of iterated expectations, that term vanishes precisely because UR.2 states that the expectation of the error conditional on X is zero. Conclusion: the expectation of the estimator is the true parameter. Be careful about what this means: it does not guarantee that your particular estimate is close to the true value. It means that if you could draw random samples over and over and compute the estimate each time, the average of all those estimates would approach the true parameter.

Best (efficient). When several unbiased methods are available, the best one is the one with the smallest variance. Using UR.3, the variance-covariance matrix of the OLS estimator reduces to the error variance multiplied by the inverse of the product of the regressor matrix and its transpose. No other linear unbiased estimator achieves a smaller variance.