3. Backtesting without code, and overfitting

Theory 5: What algorithmic trading really is

Test enough parameter combinations and it is mathematically almost certain that you will find a strategy that would have worked spectacularly well in the past. And that means absolutely nothing about the future.

We now have the pieces of a complete strategy: signal, entry, exit, position sizing. Before risking a single real euro there is a mandatory intermediate step, one we already mentioned briefly in the second level of this course.

It is backtesting: testing the strategy against historical data to see how it would have behaved in the past. This module goes deeper into how to do it well and, above all, into how it gets done badly almost without anyone noticing.

What a backtest actually does

Conceptually, a backtest walks through historical data day by day, or minute by minute depending on the strategy's frequency. It applies exactly the same signal, entry, exit and position sizing rules that would apply in real time, recording the hypothetical result of each simulated trade.

At the end of the walk you obtain familiar metrics: total return, the Sharpe ratio we saw in the previous level, or the maximum drawdown we will cover in the next module.

So far it looks like a fairly mechanical and objective exercise. The problem is that there are several ways, all of them easy to commit with no bad intent, for a backtest to produce results far better than the strategy would ever achieve in reality.

Look-ahead bias: knowing what could not yet be known

The most serious trap, and the one that breeds the most false confidence, is look-ahead bias: unknowingly using information that was not yet available at the real moment of decision.

A typical example. You compute a signal using the day's closing price, and you simulate that the trade is executed at that same day's close. In reality you cannot know what the close will be until the day is over. The real execution could have happened, at the earliest, at the next day's opening price.

There is another example, subtler. Using a company's financial data 'as it looks today', which often includes revisions and restatements made after the original publication date, instead of the exact data that was available at the historical moment being simulated.

A backtest with look-ahead bias, however small, can artificially inflate results in a way that is hard to detect without reviewing every step of the simulation very carefully.

Survivorship bias: the list of those left standing

A second common trap is survivorship bias.

It consists, for instance, of building the backtest using today's list of S&P 500 companies, applied retroactively to the past twenty years. The problem is what that list leaves out: dozens of companies that were in the index during that period and no longer exist. They went bankrupt, were acquired, or simply dropped out of the index for poor performance.

The companies that survived and prospered are, by definition, the ones still on today's list.

So a backtest built that way systematically overstates the real historical return of having invested in 'the S&P 500 as it stood at each moment'. It silently excludes everything that failed along the way.

Overfitting: memorizing the noise

The most treacherous trap of all, however, is overfitting. It consists of tuning a strategy's parameters so tightly to one particular set of historical data that it ends up memorizing the specific noise of that period, instead of capturing a genuine, repeatable pattern.

This is the claim we opened with. If you test enough parameter combinations on the same stretch of data, it is mathematically almost certain that you will find one that would have worked spectacularly well in that particular period. We mean things like:

• The exact threshold of a moving average.
• The exact number of days in an indicator.

That winning combination turns up by pure statistical chance, without that implying anything at all about its future behavior.

The defense: data you never touch

The main defense against overfitting is to split the data into two separate stretches:

• In-sample: the stretch used to design and tune the strategy.
• Out-of-sample: a stretch that is never touched during the design process, reserved exclusively for checking the result once the strategy is fixed.

The warning sign is unmistakable. If performance degrades sharply on moving from the in-sample stretch to the out-of-sample one, much of the original 'good' result was really overfitting dressed up as a real pattern.