4. How to spot that your backtest is contaminated

Level 2: The biases that break a backtest

The previous three modules describe diseases. This one is the diagnosis: a set of checks you can run on any backtest before believing it. No single test proves that a result is valid, but taken together they rule out the overwhelming majority of those that are not, and they are cheap.

The signs that should put you on alert

• Return and Sharpe out of range. In equities, with daily rebalancing and no leverage, a net Sharpe above two demands a very specific explanation of where the money comes from and who is putting it up.
• A curve that is too smooth. A straight line with a positive slope and no drawdowns worthy of the name is not a good strategy: it is a hint that the return is being computed with information you did not have.
• Too few losing trades. A hit rate above seventy or eighty percent at a daily horizon is more typical of a leak than of an edge.
• Extreme concentration. If eighty percent of the profit comes from five days or from two stocks, you do not have a strategy, you have an anecdote.
• Brutal sensitivity to one parameter. An isolated peak in the parameter grid is overfitting with near-total certainty.
• Out-of-sample degradation. A held-out stretch that returns a fraction of the fitting stretch is telling you what you are going to get in live trading.

An executable diagnosis

import numpy as np
import pandas as pd

def diagnostico(pnl_diario):
    """pnl_diario: Series of the strategy's net returns."""
    d = {}
    d["sharpe"] = pnl_diario.mean() / pnl_diario.std() * 252 ** 0.5
    d["pct_dias_ganadores"] = (pnl_diario > 0).mean()

    curva = (1 + pnl_diario).cumprod()
    d["max_drawdown"] = (curva / curva.cummax() - 1).min()

    # What fraction of total profit comes from the best 1% of days
    top = pnl_diario.nlargest(max(1, int(len(pnl_diario) * 0.01))).sum()
    d["concentracion_top1pct"] = top / pnl_diario.sum()

    # Linearity: R2 of the equity curve against time
    y = np.log(curva.values)
    x = np.arange(len(y))
    d["r2_curva"] = np.corrcoef(x, y)[0, 1] ** 2
    return pd.Series(d)

An r2_curva above 0.98 in an equity strategy is a red flag, not an achievement: it means the profit arrives with a regularity the market does not have. And a concentracion_top1pct at or above one tells you that without a handful of days the strategy does not exist.

These figures are read together and against a benchmark, never on their own. The cheapest benchmark is the index itself over the same period: compute the diagnosis for buy and hold and compare. If your strategy has half the drawdown, three times the Sharpe and a winning-day percentage ten points higher, you have not found an edge, you have found a bug. The second benchmark is a randomized version of your own signal: shuffle the positions while keeping the average exposure and repeat the calculation a few times. If the shuffled result looks too much like the original, what you are measuring is market exposure, not your signal.

The lag test, which is the most informative of them all

for retraso in [0, 1, 2, 3]:
    pnl = senal.shift(retraso) * retornos
    s = pnl.mean() / pnl.std() * 252 ** 0.5
    print(retraso, round(s, 2))

# 0 -> 3.10   1 -> 0.28   2 -> 0.21   3 -> 0.19

That pattern is unmistakable: the entire result lives at lag zero, which is to say in information from the very instant of the decision. A strategy with real content degrades smoothly, because its signal predicts several days ahead and not just the next one. If your result collapses when you add a day, the problem is not that the market is fast: it is that you were looking at the answer.

Plateau versus peak

rejilla = {}
for ventana in range(5, 61, 5):
    for umbral in [0.5, 1.0, 1.5, 2.0]:
        rejilla[(ventana, umbral)] = backtest(ventana, umbral).sharpe

tabla = pd.Series(rejilla).unstack()
print(tabla.round(2))
print("fraccion de celdas por encima de 1:", (tabla > 1).mean().mean())

What you are looking for is not the maximum cell, it is a contiguous region of decent cells. A real economic effect does not switch off because you change the window from twenty-one to twenty-three. If the grid is a desert with a geyser in it, the conclusion is not that you have found the optimal parameter but that you have found the coincidence that best fits this particular history.

What none of these tests fixes

It is worth being honest about the limits. These checks detect leaks and overfitting; they do not detect that you have tried four hundred ideas and are presenting the best one, because for that you need a piece of information that is not in the backtest: the count of attempts. Nor do they detect a leak that acts uniformly across the whole sample, such as a global normalization or a universe built from today's composition, because those sail through the lag test unruffled. That is why the diagnosis is combined with the written record from the previous module.

A real case: how many attempts it takes to manufacture a Sharpe

Bailey, Borwein, Lopez de Prado and Zhu proved something worth keeping in mind at all times: with a short history, on the order of a few years of daily data, and a modest number of variants tested, you can obtain a backtest with a Sharpe of around one whose expected out-of-sample performance is exactly zero. It takes no bad faith and no thousands of attempts; the normal exploration of a reasonable researcher who does not keep count is quite enough.

That is where the deflated Sharpe ratio comes from, adjusting the result for the number of tests carried out and for the length and shape of the series. The practical conclusion matches Campbell Harvey's when he calls for higher significance bars: a backtest without the count of attempts alongside it is not a result, it is a well-formatted anecdote. Before you risk money, run the diagnosis, run the lag test, look at the whole grid and write down how many times you tried.