2. Outliers: telling a typo from a real move

Level 4: Cleaning and validation

A share prints minus 40% in a single day. It may be a decimal point the vendor put in the wrong place, in which case it is rubbish that contaminates everything you compute on top of it. Or it may be a failed takeover bid, a clinical trial that goes wrong, a suspension of payments, in which case it is exactly the kind of day your strategy has to know how to live through. Treating both cases the same ruins the series in both directions.

The cost of getting it wrong runs both ways

If you swallow the typo, estimated volatility goes through the roof, any reversion model sees an enormous opportunity where there is nothing at all, and the backtest buys a collapse that never happened, collecting an equally fictitious bounce.

If you delete the real move, it is worse, because the damage does not show. The series comes out clean and smooth, the Sharpe ratio improves, and you have eliminated precisely the sessions on which a strategy's survival is at stake. On October 19, 1987 the S&P 500 fell around 20% in a single session; on May 6, 2010, the so-called flash crash, US indices plunged about 9% intraday and recovered almost all of it within minutes. A filter that removes without asking anything above 15% a day erases the first, and one that cleans on intraday range swallows the second. Neither of them was a data error.

Four cheap filters that catch nearly everything

None of them decides anything on its own: each one raises a flag that a human or an explicit rule will resolve afterwards.

A threshold on the return. Not an absolute number for the whole universe, but one relative to the stock's own historical volatility. An 8% move is routine in a biotech and an anomaly in a regulated utility.
Internal consistency of the bar. High greater than or equal to low, open and close inside that range. These are accounting identities: if they fail, there is no possible interpretation, there is a typo.
Volume and price disagreeing. Zero volume with a close different from the previous day's means somebody invented a price, or the volume is wrong. Both possibilities deserve a flag.
Comparison between sources. The most expensive filter and the most conclusive. Two independent vendors rarely make the same typo; a takeover bid gets picked up by both.

import numpy as np

df = df.sort_index()
df["ret"] = df["cierre"].pct_change()

# Relative threshold: how many standard deviations of the stock itself, not of the market.
vol = df["ret"].rolling(60, min_periods=30).std().shift(1)
df["z"] = df["ret"] / vol

df["f_retorno"] = df["z"].abs() > 8
df["f_imposible"] = (df["cierre"] <= 0) | (df["ret"] <= -1)
df["f_ohlc"] = (
    (df["maximo"] < df["minimo"])
    | (df["cierre"] > df["maximo"]) | (df["cierre"] < df["minimo"])
    | (df["apertura"] > df["maximo"]) | (df["apertura"] < df["minimo"])
)
df["f_volumen"] = (df["volumen"] == 0) & (df["cierre"] != df["cierre"].shift(1))

banderas = [c for c in df.columns if c.startswith("f_")]
df["sospechosa"] = df[banderas].any(axis=1)
print(df[banderas].sum())

Note two details in the code. The first is that the reference volatility carries a one-position shift: it is computed on data prior to the session being judged, so that the suspicious move itself does not inflate the yardstick used to measure it. The second is that the bar check includes the open. It is the most frequent omission of all: the three obvious comparisons are written on the close, the filter looks complete, and the file passes the check with the typo intact in the column nobody looked at.

Confirming against a second source

# Two sources aligned by date. If both see the same jump, it is real.
comp = principal[["cierre"]].join(secundaria[["cierre"]],
                                  how="inner", lsuffix="_a", rsuffix="_b")
comp["dif_rel"] = (comp["cierre_a"] - comp["cierre_b"]).abs() / comp["cierre_b"]

# They disagree by more than 2%: one of the two is wrong, and it needs looking at.
comp["discrepan"] = comp["dif_rel"] > 0.02

# A big jump that BOTH sources record stops being suspicious.
df.loc[df.index.isin(comp.index[~comp["discrepan"]]), "f_retorno"] = False

This cross-confirmation turns a suspicion into a diagnosis, but it has a limit: when the two sources disagree you know one of them is wrong, not which. The tie-break is usually cheap: if the new price persists the following day in both, the jump was real and the discrepancy was an update lag.

Flag and log, never delete in silence

The rule admits no exceptions: a suspicious row is flagged, logged and decided on with an explicit criterion, it never disappears without a trace. The reason is reproducibility, not squeamishness. If you delete in silence, six months from now nobody, yourself included, will be able to explain why the series has 2,487 rows instead of 2,503.

incidencias = df.loc[df["sospechosa"], ["cierre", "ret", "z", "volumen"] + banderas]
incidencias = incidencias.assign(ticker=ticker, revisado_el=pd.Timestamp.utcnow())
incidencias.to_csv("incidencias.csv", mode="a", header=False)

# The clean series is DERIVED, it does not replace. The original is always kept.
df["cierre_limpio"] = df["cierre"].where(~df["f_imposible"] & ~df["f_ohlc"])

A worked example: the threshold that improves the Sharpe without improving anything

Picture a panel of 500 stocks and 2,500 sessions: 1.25 million observations. A filter that removes every session with a return above 15% in absolute value might touch some 1,200 rows, less than 0.1% of the panel. It sounds harmless, and that is exactly the argument it sneaks in on: nobody argues with a change that affects a thousandth of the data.

But the percentage of rows is the wrong metric. If a thousand of those twelve hundred were real moves, you have erased a thousand genuine collapses and a thousand genuine jumps, and those are the sessions that concentrate almost all of the strategy's potential loss. The Sharpe ratio will go up because the denominator has shrunk, not because the system has got better. The cheap way to check is to run the backtest twice, on the original series and on the clean one, and look at the difference in the worst day and in the maximum cumulative drawdown, not in the average return. If the maximum drawdown improves far more than the return does, you have not cleaned data: you have taken the risk off the paper without taking it off the market. Rule of thumb: when a clean-up improves the result a great deal, suspect the clean-up before you suspect the market.