3. Why a backtest lies if it does not subtract costs

Costs 6: The full bill

There is a line that turns up in almost every systematic-investing forum: 'my backtest returns 30% a year'. It is nearly always true and nearly always beside the point, because what that calculation describes is not a portfolio anybody could have held, but an imaginary one that buys and sells at prices nobody would ever have offered it, without paying anyone for the service and without declaring a thing for tax. This module sets out exactly what is being left out, and how much it weighs.

A backtest without costs describes an impossible portfolio

A backtest reproduces what a rule would have done on past data, and its worth depends entirely on that reproduction resembling reality. Costs are not a margin of error around the result: they are a systematic subtraction, always in the same direction, which grows with turnover. A backtest without costs does not give you a result with uncertainty around it, it gives you a result biased upwards by construction.

Marcos López de Prado insists that most published backtests are useless, and that overfitting, trying hundreds of variants and keeping the best one, is only half the problem. The other half is that the winning variant usually wins because it trades a great deal, and trading a great deal is precisely what the simulator is not charging for.

The costs almost nobody models

• Half the spread. Almost every backtest executes at the closing price. That price does not exist for trading purposes: there is a bid and there is an offer, and you are always on the wrong side. Executing at the midpoint is handing yourself half a spread at each end of every trade.
• Market impact. The simulator buys whatever quantity it likes at the same price. In the real market, your own order moves the price against you the moment your size starts to show (level 2). It is the cost that makes a strategy which is excellent with 20,000 euros unworkable with two million.
• The per-trade minimum commission. It is the most devastating in small accounts and the one that almost never appears, because simulators model commission as a percentage. With a lot of small positions, the minimum can multiply the effective commission tenfold.
• Taxes. A strategy that realizes capital gains every month pays tax every year; a position simply held defers that payment for decades (level 5). The simulator does not tell the two apart.
• The financing cost of short positions. Selling short requires you to borrow the security and pay for it. In hard-to-borrow names that cost can exceed 10% a year, and those are exactly the securities that generate the most sell signals in a great many strategies.
• Data and infrastructure. Subscriptions, clean historical data, platforms. On a small account, 1,200 euros a year of data is 2.4 percentage points of return if the account holds 50,000 euros.

To all of this you must add two biases which are not costs but produce the same effect of inflating the result: look-ahead bias, using information that had not yet been published on the date in question, and survivorship bias, testing on a list of companies from which the ones that went bust have already been removed.

The same system, with the bill attached

Take a plausible strategy, stated as an example. It picks 30 mid-cap shares on a quantitative criterion, rebalances every month and replaces around half the portfolio at each rebalancing. That comes to some 360 trades a year and turnover close to 600%. The costless backtest throws up 11% a year against the index's 7%: four points of advantage, a result anybody would call excellent.

Now the bill, with the assumptions stated. A round-trip cost of 0.35%, adding the full spread, impact and proportional commission. Turnover of 600%: 2.1 points of friction. A minimum commission of 5 euros per trade: 360 trades come to 1,800 euros a year, a figure that does not depend on the size of the account.

• A 30,000-euro account. Those 1,800 euros are 6.0 points. Net: 11 minus 2.1 minus 6.0 = 2.9%. The index ETF, with a TER of 0.20%, delivers 6.8%. The brilliant strategy loses by almost four points.
• A 300,000-euro account. Those 1,800 euros are 0.6 points. Net: 8.3%. Now it does win, but the four-point advantage has come down to 1.5, and at that size market impact in mid-caps stops being the 0.35% assumed.

The same strategy, picking just as well, is a disaster or a reasonable idea depending on the size of the account. That dependence shows up in no backtest that ignores the minimum commission.

Why this kills most published anomalies

Robert Novy-Marx and Mihail Velikov did the exercise systematically: they took a broad set of anomalies documented in the academic literature and re-evaluated them net of realistic transaction costs. Their conclusion, put without ornament, is that a considerable share of them stop being profitable, and that the survivors tend to be the low-turnover ones. Anomalies that demand frequent rebalancing, or that live in small securities, are precisely the ones that survive the bill worst.

Combine that with the fact that the typical gross edge of a published anomaly is modest, of the order of a few points a year, and the conclusion is direct: what separates a real anomaly from a statistical curiosity is not usually the discovery, but the cost.

The number you should always work out

There is one figure that turns all of this into a one-minute test: the break-even cost. It consists of not fixing the cost at all, but treating it as a variable and asking at what level per round trip the strategy stops beating the passive alternative. If your system dies at 0.10%, it does not exist outside the spreadsheet. If it holds up at 0.80%, you have room against what you do not control. It is the most informative check that can be run on a backtest, and one of the least common.