3. Backpropagation: gradient descent and chain rule

Level 11: Neural Networks for Trading

The model has unknown weights and we want values that make it fit the training data well. The generic approach to minimizing the error function is gradient descent, which in this context goes by the name of backpropagation. It is worth pinning this down from the start, because it is the most common confusion: backpropagation is not an optimizer that competes with gradient descent. It is the efficient way of computing the gradient, exploiting the fact that the model has a compositional form, by applying the chain rule of differentiation. The update step is still plain old gradient descent.

Thanks to that compositional form, the gradient can be computed with one forward sweep and one backward sweep over the network, storing only quantities local to each unit.

The derivatives, in words. Let z be the vector of hidden-layer outputs for one observation. The derivative of that observation's error with respect to an output-layer weight is: minus twice the residual (actual minus predicted), times the derivative of the output function evaluated at its input, times the value of the hidden unit feeding that weight. The derivative with respect to an input-layer weight is analogous but chains one more factor: sum over all outputs of the residual times the derivative of the output function times the weight connecting that hidden unit to that output, all of it multiplied by the derivative of the sigmoid evaluated at the hidden unit's input, and by the value of the corresponding input.

With those derivatives, the update at iteration r plus 1 is the usual one: the new weight is the old weight minus the learning rate times the sum of the derivatives over all observations.

The backpropagation equations. Here comes the pretty part. The two derivatives above can be written compactly as a local "error" multiplied by the signal entering the weight. Let delta be the error at an output unit and s the error at a hidden unit. Then the derivative with respect to an output weight is delta times the value of the hidden unit, and the derivative with respect to an input weight is s times the value of the input. And those two errors are tied together by the relationship that gives the algorithm its name: the error of a hidden unit is the derivative of the sigmoid at that unit, multiplied by the sum of the errors of the output units weighted by the weights that connect them. In other words, the error is distributed backwards along the same connections by which the signal traveled forwards, weighted by the same weights.

The algorithm comes down to two passes:

1. Forward pass: with the weights fixed, the hidden layer's Z are computed, then the output layer's T, and finally the predictions.

2. Backward pass: the delta errors at the output are computed and backpropagated with the relationship above to obtain the s errors of the hidden layer.

Both sets of errors are then used to compute the gradients for the update.

Advantages of backpropagation. Its virtue is its simple, local nature. Each hidden unit passes and receives information only from and to the units with which it shares a connection. It does not need to know the whole network, only its neighborhood. That is why it can be implemented very efficiently on parallel architectures, and why it scales to enormous networks.

Two honest warnings for trading. First: a network's error function is not convex, so gradient descent finds a local minimum, not the global one; reinitializing with different seeds gives different models. Second: summing the gradients over all observations before updating is batch descent; in practice mini-batches are used, and with financial time series the split must respect chronological order even if the mini-batch is shuffled during training.