2. Feature engineering: lags and rolling windows

Level 8: Time Series - Stationarity

A time series has to be reframed as a supervised learning problem before machine learning algorithms can be applied to it. In a time series there is no notion of input and output variables: there is only a column of values and a time index. We have to choose the variable to predict and engineer all the inputs we will use to forecast future steps.

What needs transforming is this:

• time 1, value 1
• time 2, value 2
• time 3, value 3

into this:

• input 1, output 1
• input 2, output 2
• input 3, output 3

Lag features. These are the classic way of turning a temporal forecast into supervised learning. The simplest approach is to predict the value at the next step (t+1) given the current value (t). In pandas, shift(k) moves the series k positions down, so that today's row shows the value from k days ago:

values = pd.DataFrame(df.values)
df2 = pd.concat([values.shift(2), values.shift(1), values], axis=1)
df2.columns = ['t-2', 't-1', 't']
df2.head()

The first two rows will contain null values: for the first observation there is no t-1 or t-2. It is the unavoidable price of looking backwards and the reason why a model with many lags loses the start of the sample.

Rolling window statistics. A step beyond adding raw lagged values is to add a summary of the values at previous steps. We can compute statistics over the values that fall inside a sliding window and include them as features. The most useful is usually the mean of the previous values, the moving average:

values = pd.DataFrame(df.values)
shifted = values.shift(1)
window = shifted.rolling(window=2)
means = window.mean()

df3 = pd.concat([means, values], axis=1)
df3.columns = ['mean(t-2,t-1)', 't']
df3.head()

Note the critical detail: shift first, apply the window second. That shift(1) is what guarantees that the mean in row t is computed from t-2 and t-1, without including the value at t itself. If you applied values.rolling(window=2).mean() directly, the feature in row t would contain information from time t, that is, from the very value you are trying to predict. That is lookahead bias, the error responsible for more spectacular and fake backtests than any other in the history of quantitative trading.

Expanding window statistics. Another useful kind of window includes all the previous data in the series. It is called an expanding window and it helps keep track of the limits of what has been observed to date:

values = pd.DataFrame(df.values)
window = values.expanding()

df4 = pd.concat([window.mean(), values], axis=1)
df4.columns = ['mean', 't']
df4.head()

The practical difference between the two: the rolling window has finite memory and forgets the distant past, so it adapts to regime changes; the expanding one never forgets, so it becomes steadier and steadier and slower and slower to react. For all-time highs, drawdowns or cumulative normalizations, the expanding window is the right choice. For an average meant to reflect the current market, the rolling one.