1. Beyond linearity: what a neural network is

Level 11: Neural Networks for Trading

You already know how to fit a linear and a polynomial regression. Both have an obvious ceiling: they force you to decide the shape of the relationship in advance. If the price responds to your signals in a way that is neither a straight line nor a parabola, no coefficient is going to save you.

Although many phenomena in the world are modeled well with linear regression or classification, the genuinely interesting phenomena tend to be non-linear. Many families of models have been developed to deal with them. Parametric models assume that the data follow some known class of non-linear function (polynomial, power, exponential). A different class, the one we care about here, are artificial neural networks: non-linear models inspired by the physiological architecture of the nervous system, which chain together very simple non-linear computations that, in aggregate, implement complex and robust non-linear functions.

The central idea is this: extract linear combinations of the inputs as derived features, and then model the target as a non-linear function of those features. The name comes from the fact that they were first developed as models of the human brain: each unit represents a neuron and each connection a synapse. It is worth saying early on, because there is an enormous amount of hot air around the subject: a neural network is neither magic nor mysterious. It is a non-linear statistical model.

A bit of history, which does matter here. The first were Walter Pitts and Warren McCulloch, with a technique called the threshold logic unit, designed to mimic how a neuron was then believed to work. But the true forerunner of modern networks is Frank Rosenblatt's perceptron: it came with a learning procedure that could be proved to converge and was able to recognize letters and numbers. Rosenblatt was so convinced that this would lead to real AI that he made grandiose statements in 1959.

Marvin Minsky, one of the fathers of AI, began to suspect that something was wrong. Together with Seymour Papert he wrote the book Perceptrons, which duly killed the idea: they proved that the perceptron was incapable of learning the exclusive OR (XOR) function, and worse still, that it was theoretically impossible for it to learn it no matter how long you trained it. Today this does not surprise us — the model implicit in the perceptron is linear and XOR is not — but at the time it was enough to freeze research and bring on the first AI winter.

The architecture. A neural network is a two-stage regression or classification model. M derived features Z are created from linear combinations of the inputs, and the output is modeled as a function of linear combinations of those Z.

# Stage 1: hidden layer with M derived units
# alpha has shape (p, M) and alpha_0 is the bias of each unit
Z = sigmoid(alpha_0 + X @ alpha)    # Z has shape (n, M)

# Stage 2: output layer with K units
T = beta_0 + Z @ beta               # T has shape (n, K)
f = g(T)                            # g = identity for regression, softmax for classification

For regression K is normally 1 and there is a single output unit, although the network handles several quantitative responses without any trouble. For classification into K classes there are K units on top, and unit k models the probability of class k, with K targets coded as 0-1 variables.

In trading this is exactly what you need when you suspect that the relationship between your signals and future returns is not a straight line: that an RSI of 20 is bullish but an RSI of 5 is capitulation, and that on top of that the effect depends on volatility. A linear regression cannot capture that; a hidden layer with several units can.