4. Standard error: more data is not more certainty

Level 3: Descriptive and inferential statistics applied to markets

To have half as much uncertainty about an estimate you do not need twice the data. You need four times as much.

And to cut it to a tenth, a hundred times as much. That arithmetic, which almost nobody does, is what decides when a ten-year backtest is not really any better than a seven-year one.

Standard error is not standard deviation

The standard error measures the variability you would expect in an estimate —the mean return of a sample, for example— if the sampling process were repeated over and over again.

It must not be confused with the standard deviation of the data themselves:

• The standard deviation describes how much the individual observations vary among themselves. For instance, how much a share's daily returns rise or fall.
• The standard error describes how much the computed mean would vary from one sample to another, if you repeated the experiment of gathering new data.

They are two different questions. One is: how much does the asset move from day to day? The other is: how reliable is my estimate of the mean return, given the sample size I used?

The square root law

The relationship between the two concepts follows a formula that is simple to put into words. The standard error of the mean is equal to the standard deviation of the individual data points, divided by the square root of the number of observations in the sample.

That formula contains the central idea of this module. The uncertainty of an estimate does not fall in direct proportion to the amount of data you add, but in proportion to the square root of that amount.

This is usually called the square root law, and its practical consequences are counterintuitive.

Four times the data for half the doubt

Concretely:

• To halve the standard error, you do not need twice the data. You need four times as much.
• To cut it to a third, you need nine times as much data.
• To cut it to a tenth, you need a hundred times as many observations.

This means that the first stretch of data you gather cuts the uncertainty drastically, but each additional stretch contributes less and less.

In short, there are clearly diminishing returns in the reduction of uncertainty as you accumulate more history.

The sums, with a daily 1.5%

Let us put in concrete numbers. Suppose a series of daily returns with a standard deviation of 1.5% a day.

With 100 observations, some 4-5 months of trading, the standard error of the daily mean is 1.5% divided by the square root of 100, which is 10. That is, 0.15%.

Now you want to halve that standard error, down to 0.075%. 200 observations will not do: you need 400. The square root of 400 is 20, and 1.5% divided by 20 is indeed 0.075%.

Translated: going from 100 to 400 observations —multiplying the sample size by 4, almost another year and a half of data— is what it takes merely to halve the uncertainty about the daily mean.

And to cut it to a quarter of the original, down to 0.0375%? We would need 1,600 observations. Sixteen times the initial 100.

The marginal cost of each further reduction in uncertainty grows very fast, whether it is measured in additional years of history you have to obtain or in waiting time.

Two complications that make the picture worse

This already explains why simply using more data is not the magic solution it appears to be at first sight. But in financial markets there are two further complications, which connect with the two previous modules.

The first is independence. The standard error formula, as presented here, assumes observations that are independent of one another.

Financial returns tend to have some autocorrelation —persistent runs of volatility, for example— and that reduces the amount of genuinely new information each additional observation brings. A thousand days of correlated data are not equivalent to a thousand truly independent observations, but to a smaller effective number.

The second is regime change, already seen in the module on sample and population. If the underlying process changes over time, adding very old data does not reduce the uncertainty about the current regime.

It may even introduce bias, by mixing in information from a market context that no longer applies. And that happens however much, mathematically, more observations reduce the naively computed standard error.

The practical conclusion

The conclusion is twofold.

On the one hand, when someone presents a strategy backed by ten years of backtest as though that settled every doubt, it is worth remembering something: the marginal gain in reliability from going from, say, seven years to ten is usually far smaller than intuition suggests. And smaller still if those extra years are not independent of one another, or belong to a different market regime.

On the other hand, when the standard error of an estimate is still large despite having many years of data, the problem is almost never solved by waiting patiently to accumulate even more. It is time to ask other questions:

• Is the volatility of the signal itself too high relative to its size?
• Is the phenomenon being measured simply not stable enough to be estimated precisely, however much patience you have?