7. Analyzers, pyfolio and out-of-sample validation

Level 12: Backtesting and Real Risk Metrics

You now know which metrics to look at. What remains is computing them without writing them by hand and, above all, computing them on data the strategy has not seen during its design.

Backtrader analyzers

Backtrader ships with analyzers that you attach to Cerebro before running the simulation:

cerebro.addanalyzer(bt.analyzers.SharpeRatio, _name='sharpe',
                    riskfreerate=0.0, annualize=True)
cerebro.addanalyzer(bt.analyzers.DrawDown, _name='dd')
cerebro.addanalyzer(bt.analyzers.TradeAnalyzer, _name='trades')
cerebro.addanalyzer(bt.analyzers.TimeReturn, timeframe=bt.TimeFrame.Days,
                    _name='timereturn')

results = cerebro.run()
strat = results[0]

print('Sharpe:', strat.analyzers.sharpe.get_analysis())
print('Max drawdown (%):', strat.analyzers.dd.get_analysis()['max']['drawdown'])
print('Closed trades:', strat.analyzers.trades.get_analysis()['total']['closed'])

cerebro.run() returns a list of strategy instances; with a single strategy, element 0 is yours. TradeAnalyzer is especially useful because it breaks down winning and losing trades, the longest streak and the average duration: that is where you discover that 90% of the profit came from three trades.

Specialized libraries

Python offers several libraries for computing performance and risk metrics:

• pyfolio: a portfolio performance and risk analysis library developed by Quantopian. It lets you visualize many aspects of performance and risk, and includes functions to compute common metrics such as the Sharpe, the Sortino and many more.
• ffn (Financial Functions for Python): contains many functions that are useful to anyone working in quantitative finance, and computes metrics such as the Calmar ratio or maximum drawdown in a few lines.
• empyrical: also from Quantopian, it offers a variety of risk and performance metrics; it is simple to use and computes Sharpe, Sortino and the like.
• bt: a flexible backtesting library for quantitative strategies that also lets you compute risk and performance metrics.
• QuantLib: more oriented toward derivatives and quantitative computation, but adaptable to many complex financial calculations.

Pyfolio connects to Backtrader through a dedicated analyzer:

cerebro.addanalyzer(bt.analyzers.PyFolio, _name='pyfolio')
results = cerebro.run()

pyfoliozer = results[0].analyzers.pyfolio
returns, positions, transactions, gross_lev = pyfoliozer.get_pf_items()

import pyfolio as pf
pf.create_full_tear_sheet(returns, positions=positions,
                          transactions=transactions)

Pyfolio only needs a date-indexed time series of returns, which is what the analyzer produces. From it, it generates the tear sheet: annualized metrics, equity curve, monthly returns heatmap and the worst drawdowns with their dates and durations. A practical note: pyfolio and empyrical have gone years without active maintenance since Quantopian shut down, so expect conflicts with recent versions of pandas; quantstats covers the same ground today.

What no library can do for you

All these metrics are computed on the very sample in which you designed the strategy. If you tuned parameters while looking at those numbers, they are no longer an evaluation: they are part of the fitting, and they are contaminated.

The minimum serious validation is out of sample: reserve a final stretch of history that you do not touch until the end, and look at it once. Once. The moment you go back and change a parameter because you did not like the result, that stretch has stopped being out of sample.

Better still is walk-forward: you optimize on one window, evaluate on the stretch immediately after it, shift the window and repeat. You obtain a chain of out-of-sample evaluations that resemble how you would actually trade, reoptimizing periodically. If walk-forward performance is a small fraction of in-sample performance, you have a direct measure of how much of your backtest was overfitting.

Two further checks remain that are not metrics but data hygiene. Survivorship bias: if your universe is the current constituents of the S&P 500, you are trading in the past with the list of the ones that survived. And revision bias: GDP or corporate earnings are published and then corrected; using the corrected series means using information nobody had at the time. Both inflate results without leaving any trace on the chart.