4. A network in NumPy and another with Scikit-Learn

Level 11: Neural Networks for Trading

Implementing a network from scratch at least once is an extremely valuable exercise. It forces you to understand how it works on the inside, and that is essential for designing effective models afterwards.

A typical training script always has the same steps: define the network, initialize the parameters, iterate over the inputs, propagate forward to obtain the output, compute the loss from the output and the label, backpropagate the gradient, and update the parameters with gradient descent.

Here is a complete three-layer network, with no learning library at all. The inputs are four binary signals per day (for example: price above the 50-day average, RSI above 50, volume above its average, and a bullish candle) and the label is whether tomorrow's close is higher.

import numpy as np

# Each row is a day; each column a binary signal
X = np.array([[1, 0, 1, 0],
              [1, 0, 1, 1],
              [0, 1, 0, 1]])

# Label: 1 if tomorrow goes up
y = np.array([[1], [1], [0]])

def sigmoid(x):
    return 1 / (1 + np.exp(-x))

# CAREFUL: it takes the ALREADY ACTIVATED output, not the pre-activation
def derivatives_sigmoid(x):
    return x * (1 - x)

epoch = 500        # training iterations
lr = 0.5           # learning rate
inputlayer_neurons = X.shape[1]
hiddenlayer_neurons = 5
output_neurons = 1

wh = np.random.uniform(size=(inputlayer_neurons, hiddenlayer_neurons))
bh = np.random.uniform(size=(1, hiddenlayer_neurons))
wout = np.random.uniform(size=(hiddenlayer_neurons, output_neurons))
bout = np.random.uniform(size=(1, output_neurons))

for i in range(epoch):
    # --- Forward pass ---
    hidden_layer_input = np.dot(X, wh) + bh
    hiddenlayer_activations = sigmoid(hidden_layer_input)
    output_layer_input = np.dot(hiddenlayer_activations, wout) + bout
    output = sigmoid(output_layer_input)

    # --- Backward pass ---
    E = y - output
    slope_output_layer = derivatives_sigmoid(output)
    slope_hidden_layer = derivatives_sigmoid(hiddenlayer_activations)
    d_output = E * slope_output_layer
    Error_at_hidden_layer = d_output.dot(wout.T)   # before touching wout
    d_hiddenlayer = Error_at_hidden_layer * slope_hidden_layer

    # --- Update ---
    wout += hiddenlayer_activations.T.dot(d_output) * lr
    bout += np.sum(d_output, axis=0, keepdims=True) * lr
    wh += X.T.dot(d_hiddenlayer) * lr
    bh += np.sum(d_hiddenlayer, axis=0, keepdims=True) * lr

print(output)

Note three details. First: the derivative of the sigmoid is computed as the output times (one minus the output), which means you have to pass it the already activated value, not the pre-activation; if you pass hidden_layer_input instead of hiddenlayer_activations the gradient is garbage and the network does not learn. Second: Error_at_hidden_layer is computed before updating wout, because it needs the weights that were used in the forward pass; reversing that order is a classic silent bug. Third: the weights are added rather than subtracted because E is defined as actual minus predicted, so the sign is already built in.

With Scikit-Learn. In production you do not write this by hand. Scikit-Learn ships ready-made multilayer perceptrons, with MLPRegressor for regression and MLPClassifier for classification.

from sklearn.neural_network import MLPRegressor
import numpy as np
import matplotlib.pyplot as plt

x = np.arange(-1.5, 1.5, 0.01).reshape(-1, 1)
y = np.sin(2 * np.pi * x).ravel()

nn = MLPRegressor(
    hidden_layer_sizes=(4,), activation='tanh', solver='lbfgs', alpha=0.001,
    learning_rate='constant', learning_rate_init=0.1, max_iter=5000,
    shuffle=True, random_state=9, tol=0.0001, momentum=0.9,
    nesterovs_momentum=True, early_stopping=False, validation_fraction=0.1)

nn.fit(x, y)
test_x = np.arange(-1.5, 1.5, 0.05).reshape(-1, 1)
test_y = nn.predict(test_x)

plt.scatter(x, y, s=1, c='b', marker='s', label='actual')
plt.scatter(test_x, test_y, s=10, c='r', marker='o', label='NN prediction')
plt.legend()
plt.show()

With just four hidden units and a hyperbolic tangent activation, the network reproduces a full sine wave: that is precisely what a linear regression could never do. The alpha parameter is the L2 regularization and it is your first line of defense against overfitting; the lbfgs solver converges well on small datasets, while adam is the sensible option when you have a lot of data.

4. A network in NumPy and another with Scikit-Learn

The rest of Quant500