6. Selection biases beyond survivorship bias

Level 3: Descriptive and inferential statistics applied to markets

The strategies that work, you see all of them. They appear in journals, at conferences, in product catalogs.

The ones that do not work, nobody sees. They stay in a drawer.

That mismatch, and three others like it, mean that the material reaching your hands is a systematically misleading sample of the real universe of trading ideas.

Beyond survivorship bias

Classic survivorship bias was already covered in the market theory course: keeping only the companies or funds that still exist today, ignoring those that went bankrupt or disappeared.

But there is a wider family of sample selection biases that operate with the same perverse logic: the sample you end up observing is not representative of the real universe of possibilities.

They are just as relevant as survivorship bias, or more so, for judging serious quantitative work.

The file drawer problem

The first is the survivorship bias of published strategies, sometimes called the file drawer problem.

When a researcher, a fund or a commentator tests thousands of trading ideas over the years, the vast majority do not work and never see the light of day. They stay in the drawer: unpublished, no paper, no commercial product.

Only the strategies that showed good results —by chance or by genuine merit— get published in academic journals, presented at conferences or sold as a financial product.

The result is that the set of strategies the public gets to see is systematically biased toward apparent success. Not because quantitative trading works better than it really does, but because the selection process for publication filters out the failures.

Look at it with numbers. If a thousand variants of a strategy with no real predictive power are tested, by pure statistical chance some will show spectacular results over the period tested.

It is precisely those that get published and circulated, while the remaining 990 disappear without a trace. A layman who only sees the catalog of strategies that work on the internet or in journals is observing a radically biased sample of the real universe of ideas.

The prehistory that adds itself

The second is backfill bias, or instant history bias, extensively documented in databases of investment funds and indices.

When a fund first joins a commercial database —a hedge fund one, for example— it often also contributes its track record prior to the date it joined. That prehistory is added retroactively to the database as if the fund had always been there.

The problem is when funds decide to register:

• They tend to make themselves known precisely after a period of good results. Nobody rushes to publicize their track record just after a bad year.
• Funds that had a bad start and closed before becoming known never get to contribute that poor track record, because they ceased to exist before registering.

The result is that the complete track record you see in the database, including the years prior to official registration, is inflated. It disproportionately contains the good years of funds that survived to tell the tale, and it does not contain the bad years of those that did not survive long enough to register.

This same phenomenon occurs with thematic or factor equity indices that are launched today but publish a simulated history running backwards. That retroactive history is usually built with the benefit of already knowing which companies or which factors worked.

It is a different and subtler way of looking into the future when building the past. It is related to the look-ahead bias seen in the markets course, but applied to the construction of the universe or the index, not to the trading signal.

A thousand variants, one winner

The third is perhaps the most insidious, because it requires no bad intentions at all: the multiple comparisons bias, or data snooping.

If you test a single strategy once on a set of data and obtain a statistically significant result, the standard interpretation is reasonably reliable.

But imagine you test a thousand different variants on the same set of data —different moving average periods, different thresholds, different filters— and keep only the best one. You have completely changed the statistical problem without changing the evaluation procedure.

By pure probability, among a thousand variants with no real predictive power, it is almost certain that several will show a result that looks statistically significant purely by chance.

It is the same arithmetic as with coins: tossing a coin a thousand times in groups of twenty will produce, almost certainly, some group of twenty with sixteen or more heads, even if the coin is perfectly fair.

Picking the best of a thousand variants and reporting only its result is therefore a form of selection bias as serious as classic survivorship bias. Especially if neither of these two things is done:

• Adjusting the statistical significance threshold for the number of tests carried out.
• Validating on genuinely new data.

And it is extremely common in practice, because the very process of optimizing a strategy on historical data is, structurally, a search across thousands of variants.

The universe rebuilt backwards

Finally, a bias related to but distinct from the survivorship bias of companies: the bias of retrospectively constructing the investment universe.

When building today the history of which companies made up the S&P 500 in 2005 in order to run a backtest, it is easy to use the current composition of the index instead of the real historical composition. It is usually done out of convenience with the data available.

The problem is that this brings in companies that in 2005 were not even listed, or did not meet the inclusion criteria of the time.

This is not exactly the same as ignoring companies that went bankrupt, the classic survivorship bias. It is an error in constructing the universe of analysis: the strategy is being tested on a set of companies that, in the reality of that date, was not the investable universe available.

It also artificially inflates the results, if the retroactively added companies are the ones that later succeeded. And they are by definition: they are the ones that made it into today's index.

None of them is cured with more data

What these four biases share is decisive: none of them is corrected by gathering more data of the same kind. They are structural defects in how the sample was built, not defects of sample size.

The only real defense is methodological:

• Use databases that explicitly include vanished entities, free of survivorship bias.
• Be wary of retroactively simulated histories.
• Adjust the statistical significance criteria when multiple variants have been tested.
• Rebuild the investment universe as it really was at each historical date, not as it is today.