5. Autoencoders and PCA: reducing dimensions
Level 11: Neural Networks for Trading
When you have forty technical indicators correlated with one another, feeding them all to a model is a recipe for overfitting. You need to compress that information into a few dimensions that keep what matters. There are two routes, and it is worth understanding exactly how they differ.
Autoencoders. These are one of the unsupervised deep learning models. Their purpose is dimensionality reduction and feature discovery. An autoencoder is trained to predict its own input but, to stop it simply learning the identity function, constraints are imposed on the hidden units — typically that the hidden layer has far fewer units than the input, or sparsity penalties.
Autoencoding is a data compression algorithm where the compression and decompression functions are: first, data-specific; second, lossy; and third, learned automatically from examples rather than designed by a human. On top of that, in practically every context where the term is used, those functions are implemented with neural networks.
To build an autoencoder you need three things: an encoding function, a decoding function and a distance function measuring how much information is lost between the compressed representation and the decompressed one — that is, a loss function. The simplest form is a feedforward network where the input feeds the hidden layer and the output of that hidden layer is used to reconstruct the original input.
Autoencoders map the data to a lower-dimensional space by combining their most important features, and on top of that they decide for themselves how those features are combined — hence the "auto" in the name. We call those encoded features latent variables. This is useful for two reasons: it reduces training time and the latent representations can improve the model's performance.
PCA. Like the autoencoder, Principal Component Analysis is widely used to reduce dimensions, but it maps the input in a different way.
PCA works by projecting the input data onto the eigenvectors of the data's covariance matrix. The covariance matrix quantifies the variance of the data and how much each variable varies with respect to the others. Eigenvectors are simply vectors that keep their direction when a linear transformation is applied to them: they point the same way before and after. The covariance matrix orients the original basis vectors in the direction of the covariance between variables. In simpler terms, the eigenvector lets us reframe the orientation of the original data so as to look at it from another angle without actually transforming it. We are extracting the component of each variable that produces the most variance when the data are projected onto those vectors. We can then select the dominant axes using the eigenvalues of the covariance matrix, because they reflect the magnitude of the variance in the direction of their corresponding eigenvector.
These projections give rise to a new space where each basis vector encapsulates the greatest possible variance: the projections onto the eigenvector with the largest eigenvalue have maximum variance, those onto the second have the second largest, and so on. These new basis vectors are the principal components. We want them to point in the direction of maximum variance because greater variance in the values of an attribute can lead to better predictive power. Think of predicting the price of a car from two attributes: color and make. If every car were the same color, guessing the price from the color — a feature with zero variance — would be impossible; with the make, which does vary, the estimates improve a great deal. The principal components are linear combinations of the input variables, and it is that linearity that makes the transformed data interpretable.
Watch out for one important distinction: both PCA and the autoencoder are lossy compression, some of the original information is lost. And both differ from feature selection: selection algorithms discard some variables and keep others as they are, chosen for statistical reasons such as their correlation with the label. The autoencoder, as a feature extraction algorithm, projects the data into a new space.
Advantages of PCA: it reduces dimensionality, it is interpretable and it is fast to run. Drawback of PCA: it is unable to learn non-linear representations.
Advantages of the autoencoder: it can learn non-linear representations and it reduces dimensionality. Drawbacks of the autoencoder: it is expensive to train, extremely hard to interpret, the underlying mathematics is more complicated, and it is prone to overfitting, although the last of these can be mitigated with regularization.