6. Data snooping and p-hacking: overfitting with a proper statistical name

Level 6: Hypothesis testing and its traps in backtesting

You move the backtest start date a few months. Now it does come out significant.

Nobody has lied, nobody has falsified a data point and the backtest engine is impeccable, but the result is now worth absolutely nothing.

The market theory course already introduced overfitting intuitively: a strategy too closely fitted to its particular history, which collapses as soon as it faces new data.

The previous module gave you the exact statistical mechanism explaining why testing many variants generates false "winners" almost by guarantee. This module names, precisely, the process that produces that outcome, sometimes deliberate and sometimes not: data snooping and p-hacking.

Data snooping: the same data for both jobs

Data snooping consists of using the same dataset, repeatedly and exploratorily, for two jobs at once:

• Generating a hypothesis, that is, which rules or parameters might work.
• Testing that same hypothesis, on that same data.

And here is the underlying problem. The logic of a hypothesis test assumes the hypothesis was fixed before looking at the data that will be used to put it to the test.

When it is the data itself that suggested the hypothesis, that logic breaks down completely.

The resulting p-value is emptied of any real meaning, because in essence you searched, among many possibilities, for exactly the pattern that was going to score well on that particular sample, which is almost a self-fulfilling prophecy.

P-hacking: the active version

P-hacking is the more active, and sometimes more conscious, version of the same thing. It consists of adjusting the shape of the analysis until the p-value crosses the "significant" line, and then stopping and presenting that particular version as if it were the only analysis performed.

The available adjustments are far more numerous than they seem:

• The particular time window chosen for the backtest.
• Which assets, or which subset of assets, are included.
• Which "outlier" days end up being excluded from the final calculation.
• Which of several fairly similar metrics ends up being reported.
• Whether the test is framed as one-tailed or two-tailed.
• The exact moment at which you stop collecting more data.

Tactics that show up constantly

It is worth listing concrete p-hacking tactics that show up constantly in amateur backtests, because recognizing them is half the work of avoiding them:

• Choosing the backtest start and end dates so as to conveniently exclude an uncomfortable stretch of losses.
• Testing the strategy on ten different stocks and reporting only the three where it worked.
• Trying several stop-loss or take-profit levels until one crosses the threshold.
• Repeating the same test after removing a few "outlier" days which, as it happens, were the worst loss days.
• Changing the benchmark used for comparison after finding that the first one did not give a significant result.

Where legitimate research ends

This practice must be clearly separated from legitimate iterative research, because the boundary matters: exploring, trying things and refining ideas is completely normal and necessary.

What turns that exploration into data snooping or p-hacking is two very specific things:

• Treating the same data used to explore and fine-tune as if it were also independent confirming evidence.
• Reporting only the final polished p-value, without disclosing the exploratory path that led there.

The garden of forking paths

There is also a more subtle and unintentional version of this problem, known colloquially as "the garden of forking paths".

Even a single researcher, acting in complete good faith, on a single dataset, makes dozens of small and seemingly reasonable decisions throughout the analysis:

• How exactly to define what counts as a "signal".
• What time lag to use between the data and the trade.
• How to handle missing data within the series.

Since for each of those decisions there are several equally defensible alternatives, the researcher ends up exploring, unknowingly and with no bad intent whatsoever, a hidden space of multiple comparisons.

That is why data snooping can creep even into the work of careful and honest people.

The antidote, in four steps

The antidote has a concrete structure, which extends the previous module's out-of-sample validation with more procedural discipline:

• First, something akin to pre-registration. Writing down, before touching the test data, exactly which single strategy, which single metric and which exact time window are going to be tested.
• Second, maintaining a strict separation between the data used to explore and refine ideas, which are the in-sample ones, and the data used to confirm them, which are the out-of-sample ones. Never letting the results from the confirmation set feed back into new adjustments.
• Third, if several ideas really do need comparing, applying the stricter combined threshold from the previous module, instead of reporting the raw p-value of the best one.
• Fourth, being more suspicious, not less, of a result you have long been hoping is true after a lengthy search.

That last point deserves a sentence of its own, because your own enthusiasm is not evidence. It is usually precisely the warning sign that you should look more closely, not less.

Honest data is not the same as an honest process

One last important clarification for the quant500 context remains, and it is worth reading slowly.

Using an honest backtest engine, built on point-in-time data with no look-ahead bias, eliminates look-ahead bias. But it does not on its own protect you from data snooping.

You can perfectly well data-snoop on a perfectly honest dataset if you iterate and adjust repeatedly against the same historical sample over and over.

The honesty of the data and the honesty of the search process are two different problems, and both have to be solved.