7. From linear regression to polynomial regression

Level 10: Machine Learning Applied to Trading

In machine learning problems we want to learn a function f that predicts the output y from the input X. That f is called the regression or classification function, and in mathematics this learning problem is known as function approximation.

The regression problem is basically a good way of predicting a continuous output, although there is nothing wrong with using a discrete y in a regression problem. A discrete y pushes the problem toward classification; for example, when the question is whether y equals 0 or 1.

We compute an estimate of f from the training data, and then bring in the test data and try it out: how good is that estimated f? Since doing that with no clues at all is extremely hard, we simplify the situation by giving f a special property. The most brazen assumption possible is that f is linear: y equals a constant plus each variable multiplied by its coefficient. Estimating f then becomes equivalent to finding the coefficients. Exaggerating slightly, linear regression consists of looking at the cloud of points and drawing your straight line.

The fit is computed by least squares, solving what are called the normal equations.

import numpy as np
import matplotlib.pyplot as plt

# Synthetic series shaped like a cycle plus drift: think of a cumulative
# return curve that rises over the long run but oscillates along the way
n = 500
x = np.arange(0, 100, 100 / n)
y_true = np.sin(x / 10) / 2 + x * 0.01
y = y_true + np.random.randn(n) / 4

# Design matrix: a column of ones (the intercept) + the variable
X1 = np.vstack((np.ones(n), x)).T
beta = np.linalg.solve(X1.T @ X1, X1.T @ y)   # normal equations
y_hat = X1 @ beta

plt.plot(x, y, 'o')
plt.plot(x, y_hat, linewidth=5)
plt.title('Linear regression in 1D')
print('Y = {:.5f} + {:.5f} X'.format(*beta))

Looking at the result, the fitted line seems both good and bad at once:

1. Does the line explain the original data? Badly. We chose the wrong model: we saw curved data and fitted a straight line to it. That is high bias.

2. Does it predict a new data point well? Well. That is low variance: the line would barely change with another sample.

The line is simple, and simple is good, because it makes computation easier, interpretation easier, and it works well when data are scarce.

Polynomial regression. One reason so many textbooks insist on linear regression is that it lets you fit far more complex shapes with exactly the same method. The idea is simple: if we want a quadratic curve instead of a straight line, we add the quadratic term to the model, and the training data change in the same way: where there was a column of ones and a column with x, there are now a column of ones, one with x and another with x squared. The final computation is done with an ordinary linear regression, because the model is still linear in the parameters even though it is curved in x.

X2 = np.vstack((np.ones(n), x, x ** 2)).T
beta2 = np.linalg.solve(X2.T @ X2, X2.T @ y)

X3 = np.hstack((X2, (x ** 3).reshape(-1, 1)))
beta3 = np.linalg.solve(X3.T @ X3, X3.T @ y)

print('Y = {:.5f} + {:.5f}X + {:.5f}X^2'.format(*beta2))
print('Y = {:.5f} + {:.5f}X + {:.5f}X^2 + {:.5f}X^3'.format(*beta3))

Here is the deep lesson of this level: adding columns derived from the original ones is the cheapest way to capture non-linearity without abandoning linear machinery, and it is the gateway to the models that extract those combinations automatically.