3. Selection bias: you pick the universe unawares

Level 2: The biases that break a backtest

The previous two biases live in the code. This one lives in your head, which is why it is the hardest to eliminate: it does not show up in any line you can fix, it shows up in the set of decisions you took after looking at a result and before writing the report.

Choosing after looking

The pattern is always the same. You run the backtest, it does not come out well, and you change something reasonable:

• The period. You start in 2010 because the data before that is of poorer quality, and it just so happens that 2008 was the stretch that ruined the curve.
• The universe. You strip out the stocks below a certain average volume because they are not tradable, with the threshold set exactly where the result improves.
• The frequency. You move the rebalance from monthly to weekly because it fits the signal's horizon better, something you discovered by trying it.
• The market. You switch index and keep the one where the idea works, with the economic explanation written afterwards.

Every individual change is defensible. The problem is that the defense was built after seeing the number, and that turns each decision into one more parameter of the model.

None of those parameters shows up in your degrees-of-freedom count. If you have tried six start dates, four liquidity thresholds and three rebalancing frequencies, you have run seventy-two backtests even though you only kept one, and the one you kept is the maximum of seventy-two.

The maximum of N attempts is not an estimate, it is an order statistic

This is the part intuition handles badly. Even if the strategy has no real edge at all, the best of many attempts on noisy data comes out positive almost every time, and its expected value grows with the number of attempts roughly like the square root of the logarithm of that number. It is worth seeing with data generated with no signal in it.

import numpy as np
import pandas as pd

rng = np.random.default_rng(0)
n_dias, n_intentos = 2520, 200   # 10 years, 200 variants

# Pure noise: by construction there is no strategy worth anything
ruido = rng.normal(0, 0.01, size=(n_dias, n_intentos))
pnl = pd.DataFrame(ruido)

sharpes = pnl.mean() / pnl.std() * 252 ** 0.5
print(round(sharpes.median(), 2), round(sharpes.max(), 2))
# the median hovers around zero; the maximum comfortably approaches 1

That maximum is the number you would end up presenting if those two hundred columns were two hundred variants of your idea. And if on top of that you only count the variants you remember trying, you will be correcting for two hundred when in reality it was a thousand. Lopez de Prado hammers this point: the indispensable piece of information for judging a backtest is not its Sharpe, it is how many backtests were run before settling on that one.

The universe is a parameter in disguise

The slipperiest variant is the choice of market. You try the idea on the S&P 500 and it does not work; you try it on the Russell 2000 and it does; you construct the explanation that small companies are less efficient. The explanation may well be true, but you have not tested it: you manufactured it to fit the result you already had.

resultados = {}
for indice in ["sp500", "russell2000", "stoxx600", "topix"]:
    for umbral_volumen in [0, 1e5, 5e5, 1e6]:
        for inicio in ["2000-01-01", "2005-01-01", "2010-01-01"]:
            r = backtest(indice, umbral_volumen, inicio)
            resultados[(indice, umbral_volumen, inicio)] = r.sharpe

s = pd.Series(resultados)
print(s.describe())          # 48 combinations
print(s.idxmax(), s.max())   # the headline
print((s > 0.5).mean())      # the fraction that holds up: this is the informative bit

The right way to read that table is not to hunt for the maximum, it is to look at the whole distribution. An idea with economic content produces decent results in most of the cells and nothing spectacular anywhere; an empty idea produces one brilliant cell surrounded by nothing. The headline is the cell; the evidence is the distribution.

The bias you do not commit yourself

There is one more layer, and it is social. The strategies that get published are the ones that worked, and the ones that did not never make it into the literature at all. You inherit that filter without having taken part in it: when you read a paper about a factor, you are seeing the survivor of a selection process whose graveyard you cannot inspect.

The mechanism is the one from the first module, with the journals and the blogs playing the part of the index and the null results playing the part of delisted companies. A researcher who tests an idea and finds nothing does not write a paper called "this does not work": they file the notebook away and try something else. A course salesman who has tested thirty rules shows you the one that came out well. And an asset manager that launches eight quant funds closes the worst three after two years and markets the track record of the five that remain. In all three cases, what reaches your eyes has already passed through a funnel you do not control and whose rejection rate you do not know.

A real case: what survives of the published work

The figures here are uncomfortable and they have been measured. McLean and Pontiff followed a broad set of anomalies documented in academic journals and found that their returns fall by around twenty-six percent out of the original sample, before anyone knows about them, and by around fifty-eight percent after publication. That double decline separates the two causes: the first part is overfitting and selection, the second is the market incorporating what has now become public.

Harvey, Liu and Zhu inventoried more than three hundred published factors and argued that, with that volume of tests on the same data, the usual bar for statistical significance is inadequate and one should demand a t-statistic close to three rather than two. And the large-scale replication by Hou, Xue and Zhang, covering several hundred anomalies, left around two thirds of them without significance once stricter construction criteria were applied.

The operational conclusion: write down, before running anything, the universe, the period, the filtering rules and the number of variants you intend to try. Not because it is good bureaucratic practice, but because it is the only way of knowing afterwards how many tests you have to correct for.