3. Confidence intervals: quantifying how much we do not know

Level 3: Descriptive and inferential statistics applied to markets

Two strategies estimate exactly the same thing: a 9% mean annual return.

One of them deserves your money. The other does not. And that 9% does not tell you which.

From the point estimate to the range

Giving a single number as the result of an analysis conveys a false sense of precision. "The expected annual return is 8%." "The strategy's Sharpe is 1.1."

That number is a point estimate: the single best value the data suggest. But it is computed from a limited sample and is therefore surrounded by uncertainty.

A confidence interval is the tool that turns that point estimate into a range of plausible values. It comes with a confidence level, which indicates how much you can rely on the procedure used to build it.

What a 95% means, and what it does not

It is essential to understand what a 95% confidence interval means exactly. It is one of the worst explained, and worst understood, ideas in all of applied statistics.

It does not mean that there is a 95% probability that the true value lies inside that particular interval you have just computed.

Once computed, the interval is fixed: either it contains the true value or it does not. There are no intermediate degrees of probability about that already settled fact.

What the 95% really means is a property of the procedure. If you repeated the process of taking a sample and computing the interval a very large number of times, roughly 95% of those intervals would contain the true value of the population.

Notice the subject of that sentence: the 95% refers to the method in general, not to that one particular observation. It is a statement about the long-run reliability of the method, not a direct probability about the specific figure that came out.

Two strategies with the same 9%

Let us apply this to a backtest.

Suppose a strategy yields an estimated mean annual return of 9%, with a 95% confidence interval of [2%, 16%].

That width —14 percentage points of range— is crucial information that the point figure hides completely. The data are compatible with a mediocre 2% performance and equally with an excellent 16% one.

The true value could be anywhere in that wide range. Or outside it, because 95% is not 100%.

Now compare with another strategy that also estimates a 9% annual mean, but with an interval of [7%, 11%]. Even though the central point is identical, the second estimate is vastly more informative and reliable.

Two strategies with the same headline expected return may deserve radically different levels of confidence.

What makes an interval wide

What determines the width of a confidence interval? Two main factors, taken up in more detail in the next module:

• The variability of the underlying data. A highly volatile return series produces wider intervals than a stable one, all else being equal.
• The sample size. More observations, other things being equal, produce narrower intervals.

The confidence level chosen also plays a part. Asking for 99% confidence instead of 95% forces the interval to widen, because demanding greater reliability from the procedure can only be achieved by covering a wider range of plausible values.

There is, therefore, an unavoidable trade-off between precision (a narrow interval) and confidence (a high probability that the method gets it right).

The rankings that mean nothing

In the context of evaluating quantitative strategies, keeping intervals in mind completely changes how performance is compared across strategies.

It is also worth knowing that in most financial backtests intervals are wide by default, given the relative scarcity of independent data.

It is common to see rankings of strategies ordered solely by their point return or their point Sharpe, as if those figures were exact, error-free measurements.

If the intervals were computed, by contrast, many of those differences between first place and fifth would turn out to be statistically indistinguishable. The intervals would overlap extensively, and the order of the ranking could be entirely due to sampling noise, not to a real difference in quality between the strategies.

Apparent precision and real precision

None of this invalidates the use of point estimates. You have to give some concrete number in order to make decisions.

What it does demand is always accompanying them, mentally or explicitly, with a question: how wide would the interval around this figure be if I computed it?

Look at it in two cases:

• A 20% annual return with an interval of [−5%, 45%] is not a strong signal that the strategy works.
• An 8% annual return with an interval of [6%, 10%] is one, despite being the lower of the two point figures.

Confusing apparent precision —a single decimal place in the reported figure— with real statistical precision —a narrow interval— is one of the commonest errors when reading backtesting results.

It shows up in the marketing of financial products. And it shows up too in the self-deception of someone developing their own strategy who wants to believe it works.