2. The p-value without formulas: what it is and, above all, what it is NOT

Level 6: Hypothesis testing and its traps in backtesting

Four statements about the p-value circulate quite normally among traders, and all four are false.

They sound so like the correct definition that almost nobody notices the difference when skimming them. The p-value is probably the most quoted and most poorly understood number in all of statistics applied to trading.

The definition, with its conditional intact

It is worth defining it precisely before touching anything else, because every word of the definition is load-bearing.

The p-value is the probability, computed assuming the null hypothesis is true, of observing a result as extreme as or more extreme than the one you actually obtained.

Note the conditional, because absolutely everything that matters lives there. It is a probability computed inside a hypothetical world where H0 holds, a world where the strategy has no real edge.

It is not, in any sense, a probability about whether that hypothetical world is the one that actually exists.

The arithmetic of a particular p-value

A numerical example helps fix the idea, so suppose your strategy's backtest yields a p-value of 0.03.

The correct reading always starts from the assumption. Imagine this strategy really had no edge at all, so that all its observed return were pure luck, and you repeated this same experiment on comparable stretches of history over and over. You would see a result this good or better only about 3% of the time.

That is a statement about how rare the data is under a specific assumption, and not a statement about the probability that the assumption is false.

The two misunderstandings that sound right

With that definition in hand you can already identify, with surgical precision, the four most frequent misunderstandings circulating among traders, which you should banish from your vocabulary.

First: "the p-value is the probability that the strategy has a real edge" is false.

That would be a probability about the hypothesis given the data, of the kind "how likely is H1 in light of what I observed?", whereas the p-value is a probability about the data given the hypothesis, of the kind "how likely is what I observed if H0 were true?".

They are two genuinely different questions, and confusing them is the most expensive error in this entire level.

Second, and more subtle: "the p-value is the probability that the result is due to chance" is also a mistaken formulation, even though it sounds almost identical to the correct definition.

The correct definition requires the conditional, because it is the probability of a result this extreme assuming that chance (H0) is the complete explanation, not the probability that chance really is the explanation.

The two misunderstandings that are not even of the right type

Third: "1 minus the p-value is the probability that the strategy works" has no basis whatsoever. No such complementary relationship exists.

A p-value of 0.03 does not imply a 97% probability that H1 is true, for the simple reason that the p-value is not that kind of number.

Fourth: "a very small p-value means the economic effect is large" is false. This one is the bridge to the next module.

What happens is that the p-value combines three ingredients that are quite distinct from one another:

• How far your observed result is from what H0 predicts.
• How much natural variability the data you are working with has.
• How many independent observations you have accumulated.

With a large enough sample, even a tiny and economically irrelevant effect can produce a minuscule p-value.

The p-value does not measure effect size: it measures incompatibility with chance, given the sample size you used.

The threshold is not a natural border

Something should be said about the conventional 0.05 threshold, which is a historical convention popularized more than a century ago.

It is not a natural border between "truth" and "falsehood", and there is absolutely nothing magical about crossing from 0.051 to 0.049.

Many quantitative teams in fact prefer considerably stricter thresholds, such as 0.01 or even tighter, in two situations:

• When a lot of money or a lot of reputation is at stake.
• When many variants were explored before arriving at the final result being presented.

The underlying reason for that extra demand is precisely the multiple comparisons problem you will see in module 5.

Choosing the direction before looking at the data

There is another practical nuance worth keeping in mind, because a test can be framed in two ways:

• In a single direction, for example asking whether the strategy beats the index, ignoring the possibility that it falls clearly below.
• In two directions, simply asking whether there is a difference in either sense.

Choosing the direction of the test is entirely legitimate, but it has to be decided before looking at the data.

If you define the direction after seeing how the numbers came out, picking whichever makes your result look more favorable, you are contaminating the meaning of the p-value in a way that anticipates the data snooping problem you will see in module 6.

The question that vaccinates you against all four

There is a sanity habit worth internalizing, and it consists of asking yourself explicitly, every time you read a p-value:

"What exactly would this same calculation have said if my strategy had in fact been pure random noise?"

That question forces you to keep in mind the conditional that defines the p-value, and it is the best vaccine against the four misunderstandings above.

The p-value does not tell you whether your strategy is good. It tells you how surprised you should be by your own result if, deep down, there were nothing there.