7. An honest test before you risk a single euro

Level 6: Hypothesis testing and its traps in backtesting

Clearing a rigorous protocol does not prove that your strategy works.

It earns you the right to take it seriously, which is a rather different and rather more modest thing than is usually believed.

This final module introduces no new tools. It gathers everything seen in this level into a concrete, step-by-step protocol for putting a strategy to the test before risking real money.

What it gathers is this:

• The H0/H1 framework.
• The correct reading of the p-value.
• The distinction between statistical and practical significance.
• Awareness of type I and type II errors.
• Discipline in the face of multiple comparisons and data snooping.

Before touching the data

The first step is to define the hypothesis and the success metric before looking at the data that will be used to judge it.

This means writing down, in precise terms, what "success" will mean numerically. For example, mean annualized return above the benchmark, computed after deducting realistic transaction costs.

And it means stating the hypotheses H0 and H1 explicitly, all before running anything on the data that will serve as the final judge.

The second step is to honestly split the available data into two clearly distinct parts:

• An exploration and development window.
• A confirmation window, which stays completely locked and untouched until the end.

The temptation to "take a quick look" at the confirmation window while the strategy is still being fine-tuned is precisely what ruins the whole separation. It has to be resisted with explicit discipline.

Counting your own search

The third step is to count and declare your own search.

If several combinations of parameters, indicators or asset universes were tested during development, keep an honest record of how many, and then apply the stricter threshold that follows from module 5.

There is a simpler and rather cleaner alternative still, which consists of treating that entire exploratory phase as hypothesis-generating rather than confirming anything, and placing the entire burden of proof on the untouched confirmation window.

The single shot

The fourth step is to run, once only, the single predefined test on the confirmation data.

The p-value that comes out is interpreted exactly as in module 2. It tells you how surprising this result would be if there were in fact no real edge. Nothing more, nothing less.

Is it really worth it?

The fifth step is to assess practical significance separately.

Even if the result is statistically significant, you have to ask whether the estimated edge is large enough to justify the operational complexity, the monitoring effort and the risk of trading it live.

And that estimated edge has to come already net of all of this:

• The real costs of trading.
• Reasonable estimates of slippage.
• The strategy's capacity limits, that is, how much capital it can absorb before its own market impact erodes the edge.

It is also worth reporting a range for the magnitude of the effect, recalling the idea of a confidence interval from level 3, and not just a binary "pass or fail" label.

How much rigor, according to what you are risking

The sixth step is to weigh the asymmetry of errors according to the specific case, because not every allocation deserves the same bar:

• If the strategy is going to trade with leverage or with a large capital allocation, it is wise to lean toward a stricter threshold. You accept in advance a higher probability of mistakenly discarding a real opportunity, that is, more type II errors. And you accept it in exchange for reducing the risk of a costly false positive, that is, a type I error.
• For a small, exploratory allocation, a more permissive threshold may be perfectly reasonable.

Holding the conclusion with humility

The seventh and last step is to hold the final conclusion with humility, even after having cleared all the previous steps correctly.

Hypothesis testing reduces the risk of self-deception. It does not eliminate it.

Markets change regime, and a relationship that was genuinely real in the past may simply stop holding in the future, which is the same idea of non-stationarity you saw, applied to time series, in level 5.

It is worth saying this as precisely as possible. Passing a rigorous test earns you the right to take a strategy seriously, but it gives you no guarantee about its future behavior.

The sensible practice, after clearing the protocol, is to start with a modest capital allocation and keep monitoring live or paper performance against the same predefined hypothesis.

That monitoring has to be treated as a continuation of the test, not as a verdict already closed in the initial backtest.

What all this means

To close the level it is worth restating the central idea the first module began with.

Doing rigorous quantitative trading is not about finding a magic formula that proves a strategy works.

It is about a discipline of asking your own ideas hard questions, in the right order, before letting them anywhere near real money.

All the statistical machinery in this level exists for a single purpose, and this is where the null hypothesis, the p-value, practical significance, type I and type II errors, multiple comparisons and data snooping all come in: to make your own optimism accountable to the evidence, rather than the other way round.