2. The sigmoid and the error function you are going to minimize

Level 11: Neural Networks for Trading

The activation function applied by each hidden unit is usually the sigmoid. In words: it takes any number, passes it through a negative exponential and returns a value strictly between 0 and 1. In plain code:

import numpy as np
import matplotlib.pyplot as plt

def sigmoid(v, s=1.0):
    # 1 divided by 1 plus the exponential of minus s times v
    return 1 / (1 + np.exp(-s * v))

v = np.linspace(-10, 10, 200)
plt.plot(v, sigmoid(v, 1), color='C3', label='s = 1')
plt.plot(v, sigmoid(v, 0.5), '--', color='C0', label='s = 0.5')
plt.plot(v, sigmoid(v, 10), '--', color='C4', label='s = 10')
plt.xlabel('v')
plt.ylabel('sigmoid')
plt.legend()
plt.show()

The scale parameter s controls how fast the unit activates. A large s amounts to a hard activation at v equal to zero: the curve looks like a step, the unit is either off or on with almost nothing in between. A small s flattens it until it is almost linear in the middle. And if you write sigmoid of s times (v minus v0), what you are doing is shifting the activation threshold from zero to v0 — which is exactly the role of the bias term inside the unit.

This has an enormous practical consequence that will bite you as soon as you start training: saturation. With very large or very small input values, the sigmoid flattens out and its slope tends to zero. A saturated unit barely propagates any gradient and stops learning. That is why inputs are standardized before training and why weights are initialized small.

The parameters. We call the complete set of weights theta. It is made up of the weights and biases of the hidden layer — there are M times (p plus 1) of them, where p is the number of inputs and M the number of hidden units — and of the weights and biases of the output layer, which number K times (M plus 1). Count those numbers before training: they are the honest measure of how many degrees of freedom you are handing the model. With 8 technical signals, 32 hidden units and one output that is 32 times 9 plus 33, that is 321 parameters, to fit perhaps 1000 trading days. If that sounds like overfitting to you, you are hearing it right.

The error function. For regression we use the sum of squared errors: you sum, over all observations and all outputs, the square of the difference between the actual and the predicted value. For classification you can use squared error or, better, the cross-entropy (deviance): minus the sum, over observations and classes, of the actual target multiplied by the logarithm of the predicted probability. The resulting classifier assigns each observation to the class with the highest predicted probability.

There is an elegant detail worth keeping in mind: with softmax activation on the output and cross-entropy as the error, the neural network is exactly a linear logistic regression on the hidden units, and all the parameters are estimated by maximum likelihood. Put another way: the network does not invent a new paradigm, it learns a non-linear representation of the data and then applies the good old linear model on top of it.

def squared_error(y, f):
    return np.sum((y - f) ** 2)

def cross_entropy(y, f, eps=1e-12):
    # y are 0-1 labels, f are predicted probabilities
    return -np.sum(y * np.log(f + eps))