5. Unsupervised and semi-supervised learning

Level 10: Machine Learning Applied to Trading

Unsupervised learning algorithms are used when we only have input data and no corresponding output values. There are no right answers, and therefore there is no teacher. What these algorithms can do is find hidden structure or relationships among the different inputs. One of the best use cases for unsupervised learning is clustering problems.

Examples of unsupervised tasks:

• Clustering, such as K-means.
• Dimensionality reduction, such as principal component analysis (PCA).

There is a nuance that often gets confused: unsupervised does not mean that there is no data, nor that the algorithm works without an objective. It means that there are no labels. The algorithm does optimize something, for instance the distance from each point to the center of its cluster, but nobody told it in advance what the right answer was.

In markets, the natural example is grouping assets by behavior without imposing beforehand the sector they belong to. Nobody labels stocks as "defensive" or "cyclical" in the price data: if those groups emerge, they emerge from the structure of the returns themselves.

import numpy as np
import pandas as pd
from sklearn.preprocessing import StandardScaler
from sklearn.cluster import KMeans

# prices: DataFrame with one close column per ticker
returns = np.log(prices / prices.shift(1)).dropna()

# We describe each stock with a handful of behavioral features
features = pd.DataFrame({
    "annual_vol": returns.std() * np.sqrt(252),
    "mean_return": returns.mean() * 252,
    "skewness": returns.skew(),
    "market_beta": returns.apply(
        lambda c: np.cov(c, returns.mean(axis=1))[0, 1] / returns.mean(axis=1).var()
    ),
})

X = StandardScaler().fit_transform(features)
clusters = KMeans(n_clusters=4, n_init=50, random_state=42).fit_predict(X)
features["cluster"] = clusters
print(features.sort_values("cluster"))

Notice that at no point did we pass a y. There is no truth to get right: there is a partition of the space that the algorithm considers compact. Interpreting what each cluster means is human work, and that is where the analyst adds value.

Semi-supervised learning is used when there is a huge amount of input data but only a small amount of labeled output data. These problems sit right between supervised and unsupervised learning, which is why techniques from both worlds are combined. One of the best examples is identifying objects in photographs: some are labeled as apples or as lions, but the vast majority are not.

In finance the case is identical and very common: you have millions of news items, earnings reports and filings, of which an analyst has hand-labeled only a few hundred as positive or negative for the share price. With the few hundred labeled ones you train in a supervised way; with the millions of unlabeled ones you learn the structure of the language or of the document space, and you use that structure to propagate labels. Labeling is expensive; observing is cheap. That asymmetry is exactly what justifies the semi-supervised approach.