1. Optimizing without fooling yourself: what a backtest really is

Level 12: Backtesting and Real Risk Metrics

A backtest is, put plainly, the process of testing a trading hypothesis over past periods of time. Instead of applying the strategy going forward and waiting years to judge its behavior, the trader simulates the rules on relevant historical data and gets an answer in seconds. It is the most powerful tool you have, and also the easiest one to misuse.

Alongside the backtest there is always its dangerous sibling: optimization. Optimizing lets you improve a strategy's results by tuning the parameters and formulas that govern it. You can tune a single parameter or a whole set of them to reach a specific objective. One optimization objective might be to maximize total profit. Another might be to minimize drawdowns, that is, the falls from peak equity.

Having stated the purpose of optimization, it is worth being aware that it is a double-edged sword. If you pile enough rules onto the training data until you get the equity curve you wanted, the result is overfitting: the model will most likely lose its predictive power on the test data and on the future. Put another way, if you torture the data long enough, it will confess whatever you want to hear. And the market is under no obligation to repeat that confession tomorrow.

There is an intuition that helps you stay grounded: every parameter you add and every combination you test consumes degrees of freedom. If you evaluate thirty thousand parameter combinations over the same stretch of history, the best of them will show a spectacular result even if your signal is worthless, simply because you went looking for the maximum of the noise. That maximum does not repeat out of sample.

Types of backtesting systems

Broadly speaking, there are two kinds of backtesting system for putting a hypothesis to the test.

• Research backtesters. These tools do not fully simulate every aspect of the interaction with the market; instead they make approximations in order to give a quick read on the strategy's potential performance. Although they are frequently used both for backtesting and for execution, they are not suitable for strategies that approach high-frequency intraday trading. They are widely used across the professional quantitative trading industry to produce a first draft of any idea before putting it through a rigorous backtest in a more realistic environment.

• Event-driven backtesters. Here the automated strategy connects to a live market feed and to a broker, so that the system receives new information which is passed to a component that fires an event to generate a new trading signal. These systems run in a continuous loop and can have subcomponents such as a historical data handler and a broker simulator, which allows backtesting that closely resembles live execution. The only drawback is that they are complicated to design and more prone to programming errors.

Backtrader, the library you will use in this level, is an event-driven engine: it processes bars one at a time and only lets you see what has already happened.