4. Reproducibility: version the data, not just the code

Level 4: Cleaning and validation

You have the code in a repository, with its change history and its commit identifier. You pin the library versions. You fix the random generator's seed. You run the backtest in January and it gives 11.2% a year. You run it again in March, without having touched a single line, and it gives 8.7%. There is no fault in your code. What changed is outside it.

The code is deterministic; the source is not

A download script can be perfectly reproducible and still return different things at two different moments, because what is on the other side changes. It happens for legitimate and frequent reasons:

• Vendor revisions. A typo in a close from three years ago gets corrected, a dividend adjustment is recomputed, a stretch that arrived incomplete gets filled in. Nobody tells you, and from your side it is indistinguishable from the data having always been that way.
• Accounting restatements. A company files an amendment to its accounts with the regulator and the fundamentals of already-closed financial years change.
• Changes in the universe. An identifier gets reassigned, a stock is dropped from the index, a merger rewrites the history of the resulting company.
• Changes in the interface itself. Different pagination, different rounding, a different criterion for the reference price.

Versioning the code answers the question of which instructions were run. Versioning the data answers the other half: what they were run on. Without both halves there is no result, there is an anecdote.

Dated snapshot and checksum

The recipe is simple and boring, which is how these things should be: every download is frozen into an immutable file with the date in its name, a checksum is computed for it, and that checksum is recorded.

import hashlib, json, pathlib

def guardar_instantanea(df, nombre, carpeta="datos/instantaneas"):
    dia = pd.Timestamp.utcnow().strftime("%Y%m%d")
    ruta = pathlib.Path(carpeta) / (nombre + "_" + dia + ".parquet")
    ruta.parent.mkdir(parents=True, exist_ok=True)

    # Stable ordering: without this, the same table can produce different checksums.
    df = df.sort_index(axis=0).sort_index(axis=1)
    df.to_parquet(ruta, index=True)

    resumen = hashlib.sha256(ruta.read_bytes()).hexdigest()
    manifiesto = {
        "fichero": str(ruta),
        "sha256": resumen,
        "filas": int(len(df)),
        "desde": str(df.index.min()),
        "hasta": str(df.index.max()),
        "descargado_utc": pd.Timestamp.utcnow().isoformat(),
        "proveedor": "tu_proveedor",
        "clave_api": "tu_clave_aqui",   # never the real key in a manifest
    }
    with open(str(ruta) + ".json", "w", encoding="utf-8") as f:
        json.dump(manifiesto, f, ensure_ascii=False, indent=2)
    return resumen

Two practical warnings. The first: the checksum is computed over the file's bytes, not over its logical content. Changing the date format on export, the column order or the encoding alters the checksum without touching a single value, and produces a false positive. That is why rows and columns are sorted before writing and the output format is pinned: a check that fires over harmless things stops being taken seriously right before it fires over something serious. The second: snapshots are read-only. If something needs correcting you create a new one, you never rewrite the old one, because there are archived results pointing at it.

On the question of size, which is the usual objection: a daily snapshot of an equity panel takes up far less than people imagine in a compressed columnar format, and in any case you do not need to keep a full copy every day. It is enough to freeze the version used by each published result plus the month-end snapshots, and let the rest expire. What never gets deleted is whatever underpins a number you have shown to somebody.

Recording which data produced which result

def registrar_ejecucion(nombre_backtest, sha_datos, metricas, ruta="registro.jsonl"):
    fila = {
        "cuando_utc": pd.Timestamp.utcnow().isoformat(),
        "backtest": nombre_backtest,
        "commit": subprocess.check_output(
            ["git", "rev-parse", "HEAD"], text=True).strip(),
        "sucio": bool(subprocess.check_output(
            ["git", "status", "--porcelain"], text=True).strip()),
        "sha_datos": sha_datos,
        "ventana": ["2015-01-01", "2024-12-31"],   # pinned, never "today"
        "metricas": metricas,
    }
    with open(ruta, "a", encoding="utf-8") as f:
        f.write(json.dumps(fila, ensure_ascii=False) + "\n")

The dirty-working-tree flag is the most useful line in the whole log. A commit identifier means nothing if there were uncommitted changes at the moment of running: you will think you have the exact code and you will have a different one. And the date window is written out explicitly: a backtest whose end date is today never reproduces, because every run looks at a different sample.

The six-month rule

A result you cannot reproduce six months from now is not a result. This is not a bureaucratic demand or the whim of a tidy engineer: it is what decides whether you can attribute a change in performance to the strategy or to the ground shifting under your feet. With snapshots and a log, when the number changes you know exactly what changed: the checksum matches, in which case the problem is in the code or the environment, or it does not, in which case the source rewrote the past. Without that you have only a suspicion, and suspicions get settled with opinions.

There is a less obvious benefit, and it is the one you appreciate most over time. Dated snapshots give you for free something that is otherwise expensive to build: the picture of how the data looked at each moment, revisions included. With them you can ask whether your strategy would have worked on the information as it was actually available that day, rather than on the version corrected years later. It is an uncomfortable question and it almost always brings the result down, which is precisely the sign that you were measuring it wrong.

A worked example: reconstructing the difference

That jump from 11.2% to 8.7% stops being a mystery in five minutes if you saved what you should have. You retrieve the January snapshot, run it with the logged commit and 11.2% comes out again: the code is clean. You cross the two snapshots and up come, say, 6,400 modified closes in the history, almost all concentrated in a handful of stocks that restated their accounts. That is where the question becomes interesting: if your strategy only works on the old version of the data, perhaps what you discovered was a vendor error rather than a market inefficiency.