2. Econometrics versus machine learning: two different goals
Level 10: Machine Learning Applied to Trading
Many algorithms are identical across the two disciplines, but the goal is not, and confusing them is the source of half the misunderstandings between economists and machine learning practitioners.
The goal of econometrics is to find the estimated parameter, usually written as theta with a hat on top. And theta here means the causal impact of X on y. The main statistical concern of econometrics is sampling error: quantifying the uncertainty around that estimated parameter caused by the randomness with which the population was sampled. That is why econometrics papers are full of standard errors, confidence intervals and hypothesis tests.
The ultimate purpose is to make counterfactual predictions. For example: what would happen to Amazon's profits if it changed the design of its website? Translated to markets: what would happen to a stock's liquidity if the regulator changed the minimum tick size? We cannot observe the world under those alternative policies, so looking up the answer in the data is not enough. Knowing the counterfactual requires measuring a causal effect, and measuring a causal effect requires making assumptions. That is exactly what economics is about.
The goal of machine learning, by contrast, is to obtain the best possible prediction out of sample. We say its main concern is y with a hat, that is, the predicted value. The objective is to make sure the best prediction is obtained by fitting and validating many different types of model. That is what cross-validation is for, and it is the declared obsession of machine learning practitioners.
The practical consequence in trading is enormous. If your question is "does this factor cause returns, or is it an artefact?", you are doing econometrics and you need identification assumptions. If your question is "which model best predicts tomorrow's return on data it has never seen?", you are doing machine learning and you need an honest validation protocol.
Translation table between econometrics (statistics) and machine learning:
- Explanatory variables X ... are called input variables X
- The dependent variable y ... is called the output variable y
- The intercept or constant ... is called the bias, the standalone term
- The coefficients ... are called weights (w)
- The linear model f() ... is called the algorithm f()
- We estimate the coefficients ... they learn, train or fit the weights
- In-sample and out-of-sample ... is called train (plus validation) and test
- Shrinkage ... is called regularization
Beware of one classic terminological trap: in that table, bias means the standalone term of the equation, not the bias people talk about in the bias-variance trade-off. It is the same English word for two different things, and conversations mix them up constantly.