3. Statistical significance vs. practical significance

Level 6: Hypothesis testing and its traps in backtesting

A strategy can be statistically impeccable and, at the same time, guarantee net losses if it is actually traded with real money.

This is neither a paradox nor a rare laboratory case. It is what happens when two questions that look like the same one, and are not remotely the same, get confused.

Two questions that are not the same

Statistical significance and practical significance answer different questions, and treating them as if they were the same thing is probably the most expensive mistake someone who already masters the mechanics of the p-value can make.

Look at what each one asks:

• Statistical significance answers this: is it improbable that this result is pure noise, given the sample size?
• Practical significance, also called economic significance, answers a completely different question: if the effect is real, is it large enough to matter once trading costs are deducted?

They are independent questions. You can have one without the other, both together, or neither.

It is worth walking through the four combinations with concrete examples, because each teaches something different about how to read a backtest.

The good combination

The good combination is the one that comes out significant in both senses, the statistical and the economic.

It would be, for example, a factor strategy tested over fifteen years of data on large S&P 500 companies, with a mean annualized return above the index of 3% and a small p-value.

There you have reliable evidence of an effect that is also worth pursuing.

The arithmetic of the treacherous combination

The treacherous combination is statistically significant but practically irrelevant, and it deserves a detailed numerical example because it is exactly the scenario this level asks you to master.

Imagine a very high frequency mean reversion strategy, tested over several million historical trades.

The p-value comes out minuscule, of the order of 0.0001. Statistically, the evidence against "this is pure chance" is overwhelming.

But look now at the size of the effect. The estimated mean edge is barely 0.008% per trade, before costs.

And the real costs of executing that trade — commission, bid-ask spread, slippage — easily run between 0.02% and 0.05% per trade in a retail account.

The "real" edge is several times smaller than the cost of capturing it. The strategy is statistically impeccable and, at the same time, guarantees net losses if actually traded.

With huge samples this is almost mechanical

This scenario is not an invented extreme case: it is almost mechanical when you work with huge samples.

With millions of observations, even an absolutely tiny effect becomes statistically "improbable under chance", because, as you saw in the previous module, the p-value gets smaller the larger the sample, regardless of the true size of the effect.

Think about the current context, with high frequency data and backtest engines capable of processing years of information in seconds. In that setting, statistical significance alone stops being an interesting filter: almost everything clears it if you accumulate enough observations.

The filter that really separates the wheat from the chaff, in that context, is the economic one.

The retail trader's favorite trap

The third combination is the opposite one: practically attractive, but with no statistical backing.

A strategy shows an 8% mean annualized return above the index over barely eighteen months of data. The number, in isolation, is exciting.

But with only eighteen months and the typical daily variability of markets, the p-value of that result may perfectly well hover around 0.35, far from any reasonable threshold.

The conclusion is worth stating carefully. The number looks attractive on paper, but the accumulated evidence still does not allow you to tell it apart from luck.

The fourth combination, with no significance of any kind, is discarded without further debate.

Two habits of discipline

From this follow two habits worth building into any strategy evaluation:

• Do not settle for a binary verdict of "passes or fails 0.05". Always report as well the size of the effect in the units that matter, and not just whether the p-value crossed an arbitrary line. That is, mean return in percentage terms, ideally with a plausible range around that figure, recalling the idea of a confidence interval from level 3.
• Decide, before looking at the results, what the minimum economic threshold is that would make the strategy worth trading in practice, given its real costs and the operational complexity of keeping it alive.

Only if that economic threshold is cleared does it make sense to ask afterwards whether there is also enough statistical backing.

What significance says nothing about

Finally, it is worth connecting this with two ideas from the market theory course.

Statistical significance says nothing about the strategy's Sharpe ratio or about its drawdown.

An effect can be perfectly "real" in the statistical sense and still come with a rollercoaster of results so steep that it becomes impossible to sustain with real money.

There are two questions, and they must always be asked separately: is this statistically real, and is it worth putting up with this in practice?