5. The multiple comparisons problem: why testing 100 variants almost guarantees winners by chance

Level 6: Hypothesis testing and its traps in backtesting

One weekend, five ideas going round in your head. Four of them do not quite convince you. The fifth gets programmed, tested, and comes out with a beautiful p-value.

You never wrote a single line of code for the other four. And you have already contaminated the result of the fifth.

The seed the previous module left

The previous module left an uncomfortable seed. A significance threshold of 5% means that, even for a strategy genuinely without any edge, there is roughly a 5% probability that it clears the filter anyway, by pure chance.

The multiple comparisons problem is what happens when that seed multiplies. If you test many different strategies, all of them without a real edge, each one carries that same individual risk of a false positive, and those risks accumulate.

The arithmetic of the hundred strategies

The intuition can be anchored with a concrete number, with no need for complicated formulas.

Suppose you test 100 completely random strategies with no real edge, each at a 5% threshold.

The probability that none of them clears the filter by chance is 0.95 multiplied by itself 100 times, which gives approximately 0.006.

That is, the probability that at least one of those 100 pure-noise strategies looks "significant" is roughly 99.4%.

Finding a "winner" among 100 variants of pure noise is practically guaranteed. It says nothing about whether that winner has anything real behind it.

Where overfitting is generated

This connects directly with how overfitting is generated in practice, now with its precise mechanism laid out:

• Parameter sweeps, such as testing a moving average crossover strategy with windows of 5, 10, 15, 20 up to 200 days, that is, dozens of combinations.
• Trying dozens of different indicators.
• Tweaking entry and exit rules over and over.
• Testing the same concept on different asset universes, or on different historical subperiods, until "something works".

Each of those explorations is one more multiple comparison, without the trader necessarily perceiving it that way.

And the consequence is the one that matters: the final "winner" inherits the entire accumulated false positive risk of the whole search, not just the risk of its own isolated test.

Where exactly the problem lies

It is important to be precise about where the problem lies, because it is not where many people assume.

Exploring many ideas is not in itself illegitimate. It is a normal and necessary research process.

The problem arises when the raw p-value of the single winning result is reported and trusted as if it were the only test ever performed.

That p-value, taken in isolation, drastically understates the real probability of having arrived at that result by pure chance, given the true number of attempts made.

Three lines of defense

There are three lines of disciplined defense against this problem, and they are worth internalizing as a habit, not as an occasional formality.

The first is to keep an honest count, and to declare it, of how many variants or ideas were really tested, including all those discarded along the way, not just the winner.

The second is to apply a proportionally stricter threshold the more things you have tested. The simplest correction intuition consists of dividing the level of false positives you are willing to tolerate across the whole search by the number of tests performed.

Put numbers on that. Suppose you tested 50 variants and want to keep an overall false positive risk of 5% for the search as a whole. Each individual variant should really need something of the order of a p-value below 0.001 (5% divided by 50), not merely below 0.05, before you take it seriously.

The third, and the most solid in practice, is out-of-sample validation. And better still, outside the time period used for exploring.

Only a test on data collected and examined after the winning variant has already been fixed gives you an honest verdict, uncontaminated by the search itself.

No bad intent is needed

One uncomfortable subtlety is worth pointing out, and it is the scene at the start of this module.

The multiple comparisons problem requires no bad intent and no formal loop over 100 programmed combinations.

It is enough to informally explore five different approaches "in your head" over a weekend, mentally discard the four that did not convince you, and formally report only the fifth.

With that you have already incurred exactly the same problem, even though not a single line of code was ever written for the other four ideas.

The question you have to ask

The mental habit worth taking away from this module is simple to state and hard to practice honestly.

Every time you come across an impressively low p-value on a single strategy, the first honest question you must ask yourself is "how many things did I look at before settling on this one?".