2. Sample vs population: why a backtest is always a sample, never the complete truth

Level 3: Descriptive and inferential statistics applied to markets

You can toss a coin another hundred times. With the market you cannot: the market has only happened once.

That asymmetry, which sounds philosophical, is in fact the hardest limitation of any backtest. And it explains why no figure that comes out of one means what it appears to mean.

Population and sample

In statistics, the population is the complete set —often infinite or unobservable— of all the possible outcomes a process could generate. The sample is the finite subset of data we actually have in front of us.

With a coin, the distinction is clean. The conceptual population is the set of all possible tosses under those physical conditions, and each real run of 100 tosses is just one sample from that population.

Carried over to markets, the question becomes uncomfortable: what is the population of which a backtest of the S&P 500 between 2015 and 2024 is a sample?

Which population is a backtest a sample of?

The correct and disturbing answer is that there is no fixed, well defined population from which you can draw more data at will, as you can by tossing more coins.

The prices of 2015-2024 are not a random sample drawn from a set of possible histories of the S&P 500 from which the market picked one at random.

It is the only trajectory that actually happened. And it was conditioned by monetary policy decisions, particular geopolitical events, the sector composition of the index at that moment and an endless list of factors that will not repeat in exactly the same way.

Now think of a backtest reporting a Sharpe of 1.3 over that period. That figure is not an estimate of the true Sharpe of the S&P 500 in the sense in which the mean of a sample of coin tosses estimates the true probability of heads.

It is, at best, the estimate of a parameter of a hypothetical data generating process that we assume to be stable. And at worst, simply the description of an unrepeatable historical episode.

Two practical consequences

The first is that every statistic computed on a backtest is a sample estimate, subject to sampling error, never the truth about the asset. That includes the mean return, the volatility, the Sharpe and the maximum drawdown.

The second, subtler one, is the problem of stationarity. For it to make sense to speak of a sample from a population, that population should be reasonably stable over the time the sample covers.

But markets are not stationary in the strict statistical sense:

• Volatility changes regime. Compare 2017, an extraordinarily calm year, with 2020, with falls of 30% in a matter of weeks.
• Interest rates alter risk premia.
• The composition of the index changes: companies come in and go out, and the weight of technology has grown enormously in two decades.
• Market structure itself is transformed by algorithmic trading and the fragmentation of liquidity.

If the underlying population changes while you are taking the sample, it is not even clear what you are estimating. It may be that the true Sharpe of 2015-2019 and that of 2020-2024 are simply parameters of different processes. Averaging them in a single ten-year backtest would then produce a number that describes neither of the two periods well.

The past as history and the past as sample

A useful way to think about it is to distinguish two uses of the same historical period:

• The past as history: what actually happened. A fixed fact, with no statistical uncertainty in itself.
• The past as sample: what that episode allows us to infer, with uncertainty, about the future behavior of the asset.

Descriptive statistics deals with the first task: summarizing accurately what happened.

Statistical inference deals with the second, and is taken up in more detail in the last module of this level. It requires assuming, explicitly or implicitly, that there is something stable behind the data that can go on being used to talk about the future.

Where the jump to inference begins

The boundary is easier to see with an example.

A backtest reporting only descriptive statistics —"over this period, the strategy earned 12% a year"— is being honest about the past.

A backtest that also claims or hints that, therefore, the strategy will earn 12% a year in the future has made the jump to inference. And that jump needs additional justification.

The question to put to it is always the same: why would the mechanism that generated that return go on operating in the same way in a different market regime?

So, are backtests any use?

None of this means that backtests are useless. They are the only instrument available for evaluating strategies before risking real capital.

What it does demand is treating them with the right humility: as a sample, possibly unrepresentative, of a process we do not even know to be stable.

And there is a reasonable rule that follows from that. The shorter the backtest period, the less confidence it should inspire as an estimate of what will happen under different conditions. The same goes if it is concentrated in a single market regime: for example, only the bull market of 2013-2019, with no recession at all.

The next two modules, confidence intervals and standard error, give the concrete tools for quantifying how much uncertainty surrounds a sample estimate like this one, instead of treating it as a fact.