4. The factor zoo and the replication crisis

Level 10: Why it runs out

The three previous modules explained how a real edge gets used up. This one raises a more uncomfortable question: how many of the published edges ever existed at all. The answer forces you to change, permanently, the bar you judge any strategy by.

Hundreds of candidates, and the count keeps rising

When Harvey, Liu and Zhu reviewed the finance literature, they catalogued more than three hundred different variables proposed in serious journals to explain differences in return between shares, and documented that the rate of appearance had accelerated to dozens of new proposals a year. That collection is known, mockingly, as the factor zoo.

A market cannot have three hundred independent sources of return. Many of those variables are first cousins measured in slightly different ways. And a proportion, which is what this module is about, is simply noise with good presentation.

Why searching a lot guarantees finding something

The usual statistical threshold accepts a result when the probability of obtaining it by chance is below 5%. That threshold is reasonable if you run one test. It stops being reasonable as soon as you run many.

If you test 300 investment rules that in reality are worth absolutely nothing, 5% of them will pass the exam through pure luck: about fifteen. And of those fifteen, none will look like chance up close, because each will come with its rising chart and its economic explanation. The problem is not that fifteen lie: it is that the 285 that failed are never written up, so the reader sees only the survivors.

It is worth doing the full sum, because the result is surprising. Imagine a thousand candidate ideas, of which a hundred are genuinely valid. Suppose a method that detects 80% of the valid ones and that lets through as good 5% of the invalid ones. You will find 80 true discoveries and 45 false ones: of the 125 results announced, more than a third are false. And if instead of a hundred valid ones there were only ten, you would get 8 true against some 50 false: more than 85% rubbish, without anyone having done anything wrong except search a lot.

The degrees of freedom nobody declares

The calculation above is optimistic, because it assumes each idea is tested once. In practice, a single idea is tested dozens of times without the author calling it testing:

• Including or excluding the smallest companies from the universe.
• Weighting the portfolio by capitalization or equally.
• Rebalancing monthly, quarterly or annually.
• Starting the analysis in one year or another.
• Excluding or not periods considered exceptional.

Five binary decisions are thirty-two combinations. Presenting the best of the thirty-two as if it were the only one looked at turns a 5% threshold into something far more permissive than it appears. None of this requires bad faith: it is what happens when a procedure is adjusted until it works.

When others try to repeat it

Hou, Xue and Zhang did the dirty work: they gathered more than four hundred and fifty published anomalies and re-evaluated them all with the same procedure, weighting by capitalization and preventing tiny stocks from dominating the result. Close to two thirds ceased to be statistically significant at the usual bar, and with a more demanding bar the failure rate was above eighty percent.

The main reason is revealing. The smallest companies are the majority by number and a tiny portion of the total value of the market. When all shares are weighted equally, a good part of the result rests on stocks almost nobody can buy in quantity. Correct that and the anomaly evaporates.

It is worth being honest: other researchers, with different statistical methods, have obtained considerably higher survival rates, and the debate remains open. But nobody now maintains that the number of published factors resembles the number of usable ones.

The practical criterion: six questions before parting with a euro

These questions work for an academic paper and for the salesman knocking at your door:

• Did the economic explanation exist before the data was looked at, or was it built after seeing the result? The latter is a story, not a hypothesis.
• How many variants were tested to arrive at this one? If the answer is 'only this one', it is almost never true.
• Does it work in another period, in other countries and in another universe of securities?
• Does it survive removing the smallest companies and weighting by capitalization?
• Does it survive net of costs and at the size intended to be managed?
• Who else is holding it, and what would have to happen for it to stop working? If nobody can answer the last one, then it has not been understood where the money comes from.

Real case: the funnel, with numbers

Put everything in this level together and a devastating funnel comes out. You start with several hundred published patterns. On re-evaluating them with homogeneous criteria, around a third retain significance at the usual bar, and considerably fewer if you demand the high bar Harvey, Liu and Zhu recommend to compensate for so many tests. Of the survivors, Novy-Marx and Velikov show that the high-turnover ones lose most of their edge once the cost of trading is deducted. From those still standing you must further subtract the post-publication erosion, around half the original return according to McLean and Pontiff, and the limited capacity from the previous module.

Four hundred published ideas do not leave four hundred strategies. They leave a handful, with less return than advertised, with less capacity than you would like and with more competition than there was. That handful still exists, and it is the subject of the rest of the course. But now you know how to demand proof from it.