7. From descriptive to inferential statistics: the jump that serious backtesting demands

Level 3: Descriptive and inferential statistics applied to markets

"The strategy earned 11% in the backtest." "The strategy has a real edge."

They sound like the same sentence said in two different ways. They are not. The first is a verifiable fact about the past; the second is a bet on the future that needs a great deal more backing.

Telling them apart is the jump that turns a naive backtest into a serious one, and it is probably the most important idea of the whole level.

Two ways of using the same numbers

Everything seen so far in this level —means, medians, standard error, confidence intervals, outliers, sample biases— converges on a conceptual distinction that separates two completely different ways of using numbers:

• Descriptive statistics.
• Inferential statistics.

What it describes: the easy part

Descriptive statistics confines itself to summarizing what actually happened in the set of data in front of you, saying nothing about what will happen outside those data.

Take this sentence: "in this backtest, the strategy earned an average of 11% a year, with a volatility of 14% and a maximum drawdown of 22%".

It is, in its entirety, a descriptive claim. It accurately describes one particular historical episode, and it is as true or false as any other verifiable fact about the past.

It requires no assumption about the future, no theory about why it worked, no claim about what will happen next year.

It is, in that sense, the easy and safe part of the analysis: computing these numbers correctly is an arithmetic exercise, not a probabilistic one.

What it infers: the risky part

Inferential statistics takes a much riskier step. It uses those same sample data to make claims about something that has not been directly observed.

That something may be the future behavior of the asset. Or the true value of a parameter in the underlying population from which the sample was drawn, the concept of population seen in the second module of this level.

Let us go back to the two sentences we opened with. "The strategy earned 11% in the backtest" is descriptive. "The strategy has a real edge and will go on generating a positive return" is inferential.

And that second claim requires assuming three things:

• That the process that generated those data is reasonably stable, that is, that it does not change regime unpredictably.
• That the sample is representative and not distorted by the selection biases seen in the previous module.
• That the observed result is not simply statistical noise, indistinguishable from what chance would produce.

The hypothesis test

The most common formal framework for taking that step rigorously is the hypothesis test. Two hypotheses are set out:

• The null hypothesis represents the boring default scenario: typically, that the strategy has no real edge and that its true expected return is zero. Under it, any positive result observed would be pure chance in that particular sample.
• The alternative hypothesis says that the strategy does have a genuine, non-zero edge.

Here there is an essential nuance. Statistical analysis cannot demonstrate the alternative hypothesis directly.

What it does is something else: compute how surprising the observed result would be —an 11% mean annual return with that volatility and that number of observations, for example— if the null hypothesis were true and the strategy had no real edge at all.

That measure of surprise is the p-value: the probability of observing a result as extreme as, or more extreme than, the one obtained, assuming the null hypothesis is true.

A small p-value, conventionally below 0.05, means that the observed result would be unlikely if there really were no real edge. That is interpreted as evidence against the null hypothesis, not as definitive proof that the strategy works.

What the p-value is NOT

It is worth dismantling the commonest misreading of the p-value. It is as widespread as the misreading of confidence intervals you saw earlier.

A p-value of 0.03 does not mean that there is a 3% probability that the strategy does not work. Nor a 97% probability that it does.

The p-value is a probability computed under the assumption that the null hypothesis is true.

It is not the probability that the null hypothesis is true given what has been observed. That would be a different question, and generally a more useful one, which requires additional tools to be answered properly.

Confusing the two is one of the commonest and most persistent statistical errors, inside and outside finance.

The bridge to multiple comparisons

The connection with the previous module is direct and critical. The hypothesis testing framework, as described, implicitly assumes that a single test has been carried out.

Suppose that in reality two thousand variants of a strategy were tested and only the one with the lowest p-value is reported. That p-value no longer means what the standard framework assumes it means.

The significance threshold has to be corrected for the number of comparisons carried out, and there are specific statistical methods for doing so. Otherwise, the result will go on looking far more solid than it really is.

This is exactly why out-of-sample validation is indispensable in serious quantitative practice: testing the final strategy, once only, on data that were not used at all during the search and optimization process.

It is the only way to obtain a reasonably clean statistical test, uncontaminated by having peeked at the data thousands of times before deciding which strategy to report.

Statistics does not promise certainty

Statistics, correctly understood, does not promise certainty. Its job is not to prove that a strategy works with an unappealable stamp.

Its job is to quantify two things honestly. First, how much surprise a result deserves if there were no real edge behind it. Second, how much uncertainty surrounds any estimate made from a finite sample, potentially biased and drawn from a process that might not be stable over time.

That is, ultimately, the jump that separates two ways of looking at the same backtest. One sees a promise of future returns. The other sees a piece of partial evidence, to be weighed with the rigor of everything learned in this level:

• The possible skewness of the mean.
• The real size of the sample against the population it claims to describe.
• The width of the confidence interval behind the point figure.
• The origin of every outlier.
• The structural biases that may have inflated the result before you even started analyzing it.