1. The null hypothesis: doubting your own strategy
Level 6: Hypothesis testing and its traps in backtesting
You look at your backtest, you see a curve going up, and your judgment tells you the strategy works.
The problem is that this judgment depends on your eye. And your eye wants the strategy to work.
In the market theory course you already met backtesting in an intuitive way: separating in-sample and out-of-sample data, watching out for look-ahead bias and survivorship bias, looking at drawdown and win rate.
All of that gave you sound judgment, but intuitive judgment has that weak spot. Hypothesis testing is the machinery that replaces that "it looks like it works" with a procedure which, applied properly, does not depend on what you want to see.
The two hypotheses
A hypothesis test starts from two mutually exclusive claims about the reality that generates your data.
The null hypothesis, abbreviated H0, is the default position: "the true mean return of this strategy, once costs are deducted, is zero". Or, in a somewhat more demanding version, "it is equal to that of the benchmark".
It is worth understanding what that position implies, because it is blunt. Under H0, any positive number you see in your backtest is pure chance: one particular sample, among thousands possible, drawn from a process that in fact has no edge at all.
The alternative hypothesis, H1, is the opposite claim: there is a real and persistent edge, a process whose true mean is not zero.
Where the burden of proof sits
What matters, and what separates a rigorous hypothesis test from a backtest judged by eye, is where the burden of proof sits.
The test starts by assuming H0 is true and demands that the data, carrying enough weight, be what disproves it.
It is no coincidence that this resembles the presumption of innocence in a trial. The parallel is exact:
- The defendant, that is, your strategy, does not have to prove it is innocent, that is, that it does not work.
- It is the prosecution, that is, your data, that has to bring enough evidence to convict, that is, to claim there really is an edge.
Now imagine you reverse that order, so that you start from the belief that the strategy works and abandon it only if someone proves the contrary overwhelmingly.
Then the bar for continuing to believe in it becomes absurdly low, and almost any noise clears the filter.
Starting out by doubting is not pessimism
This starting attitude is not gratuitous pessimism. It is a fairly direct response to a structural fact about markets.
They are competitive environments where thousands of participants are hunting for exactly the same thing you are. So most trading rules that "look" profitable on one particular slice of history are really noise shaped like signal.
Do the thought experiment. If you generated thousands of completely random rules and tested them all on the same stretch of history, a far from negligible fraction of them would show, by pure chance, a positive return in the backtest.
In module 5 of this level you will see why that fraction is larger than intuition suggests. Starting out by doubting your own strategy is the only stance compatible with that reality.
What rejecting the null hypothesis means
So what does "rejecting H0" mean?
It means that the observed evidence would be too improbable if there really were no genuine edge, so, provisionally, you lean toward H1.
Note the word provisionally, because it is not rhetorical decoration. It is the nature of the conclusion itself.
And what the opposite does not mean
And here comes the most common trap of interpretation, which is worth reading slowly.
"Failing to reject H0" does NOT amount to "H0 is true", nor to "the strategy definitely does not work".
It only means that, with the data you have, you have not gathered enough evidence to accept the edge as established.
Absence of evidence is not evidence of absence, and this matters above all when the backtest is short. A strategy with a real but modest edge, tested over only six months, may perfectly well fail to clear the filter without that meaning the idea is bad.
The arithmetic of the three years
Let us look at a concrete numerical example to anchor the idea.
A strategy is tested over 750 sessions of daily data, that is, about three complete years, and produces a mean daily return above the benchmark of 0.04%. At first glance that is a positive number.
But look now at the day-to-day variability of that excess return. Its standard deviation is around 1.2%, that is, thirty times larger than the mean itself.
With such a large dispersion relative to the size of the effect being measured, that 0.04% could perfectly well be the result of luck. In a great many samples generated by a process with a true mean of zero, observed means of that order of magnitude would show up without any effort.
Resolving that tension is exactly what a formal hypothesis test does. The question is whether this is signal or noise, given how large the background noise is, and the test answers it by comparing three things:
- The size of the observed effect.
- The natural variability of the data.
- The number of observations available.
How much data lies behind the number
That last factor, sample size, matters qualitatively even before you get into technical detail.
The more independent sessions you accumulate, the more precise your average tends to be as an estimate of the true mean. You will come back to this formally when you recall the standard error from level 3.
That is why a strategy backed by fifteen years of clean data carries far more weight as evidence than the same strategy backed by six months, even if the mean return it shows on paper is identical.
The practical conclusion of this module is a change of stance, not a formula. When you build and evaluate your next backtest, do it as if you had to convince a skeptical judge that there is something real there, not as if you had to confirm something you have already spent weeks believing in.
That difference in attitude — doubt first, convince yourself afterwards, and only with sufficient proof — is the thread running through this entire level.