4. Cost, speed and the trade-off that really matters

Level 5: Storing, updating and scaling

The previous three modules were decisions about shape. This one is about the numbers, because nearly every argument about storage is won or lost on a multiplication nobody did. The underlying question is not how much space it takes: it is how much each thing costs compared with the alternative, and what latency your strategy genuinely needs.

Storing is cheap; re-downloading is not always

The first trade-off is between disk and download. Disk is cheap and predictable. Downloading has three costs that get forgotten: the vendor's price, the wall-clock time it takes you to repeat it, and the fact that it may no longer be possible, because deep history gets withdrawn, changes terms or the vendor disappears.

Always do the time calculation, because that is the one that surprises people. Suppose 500 stocks, twenty-five years of minute data, a vendor that returns one symbol and one month per request and limits you to five requests per minute. That is 500 times 300 months, which is 150,000 requests; at five a minute, 30,000 minutes: some 500 hours, more than twenty days of downloading solid. The disk that saves you from repeating that is not up for debate.

Daily, minute, tick: the factors that multiply

This is where people get it wrong by a large factor. The US regular session lasts six and a half hours: 390 minutes. So going from a daily bar to one-minute bars multiplies the number of rows by 390, not by 1,440, which would be the full calendar day.

On the data from module one, tens of megabytes in daily, that factor takes you to a store of tens of gigabytes in minute data. From minute to tick there is no fixed factor: it depends on the instrument's liquidity, and for heavily traded stocks we are talking about additional orders of magnitude. If you have not measured it on your instrument and with your vendor, do not quote a number; say it is orders of magnitude.

import time
import pandas as pd
from pathlib import Path

def tamano_mb(ruta):
    return sum(f.stat().st_size for f in Path(ruta).rglob("*.parquet")) / 1e6

t0 = time.perf_counter()
df = pd.read_parquet("almacen/precios", columns=["fecha", "simbolo", "cierre"])
segundos = time.perf_counter() - t0

print(len(df), "filas", round(tamano_mb("almacen/precios"), 1), "MB",
      round(segundos, 2), "s")
# Measure before deciding: intuition about these numbers is usually off by 10x

Precompute or compute on the fly

The second trade-off is between CPU and disk. Precomputing indicators and storing them turns a repeated calculation into a read; computing them on the fly keeps the store small and stops them going stale when you correct data.

The rule is narrower than people think: precomputing wins when the same calculation, with the same parameters, is repeated many times over data that does not change. If your optimization sweep moves the indicator's window, there is nothing to precompute, because every combination is a different calculation.

With numbers: 3.1 million rows, twenty fixed indicators that take eight seconds to compute in full, and a sweep of 2,000 combinations of entry and exit thresholds. On the fly that is 16,000 seconds, some four and a half hours spent purely recomputing the same thing. Precomputed, it is eight seconds once and 2,000 reads of a file that fits in memory.

import hashlib
import json
from pathlib import Path

def firma(parametros, version_datos):
    """The signature includes the store version: if the data changes, the cache expires."""
    texto = json.dumps({"p": parametros, "v": version_datos}, sort_keys=True)
    return hashlib.sha256(texto.encode()).hexdigest()[:12]

def indicadores(precios, parametros, version_datos):
    ruta = Path("cache") / ("ind_" + firma(parametros, version_datos) + ".parquet")
    if ruta.exists():
        return pd.read_parquet(ruta)
    salida = calcular_indicadores(precios, **parametros)
    ruta.parent.mkdir(parents=True, exist_ok=True)
    salida.to_parquet(ruta)
    return salida

The detail that makes that cache trustworthy is the signature including the store version, not just the parameters. Without it, the day a correction comes in you will carry on reading indicators computed on the old data, and the failure will be silent: plausible numbers, wrong results.

The question that settles everything above

All these decisions get simpler once you answer a single question: what latency your strategy genuinely needs. Not the one you would like to have; the one that changes your decisions.

• If you rebalance once a month on closing prices, your latency requirement is measured in hours. A real-time feed does not improve a single one of your decisions.
• If you trade once a day at the open, it is measured in minutes.
• If your decision depends on what happens within the session, you are into minute and tick territory, and then you really do need that volume and that infrastructure.

Almost always the real latency requirement is much lower than people think, and the highest-frequency data gets bought out of anxiety, not out of need. It is money, disk and complexity that do not buy a single better decision.

A case: the monthly strategy and the twenty-millisecond feed

Let us close with a recognizable case. A strategy rebalances on the first business day of each month using the previous day's closes. Its manager decomposes the deviation between the real result and the backtest's and finds that around 96% comes from execution slippage and from the difference between the opening price and the close used in the signal.

With that breakdown, replacing the free daily feed with fifteen minutes' delay by a professional twenty-millisecond one acts on the remaining 4%. The cost, by contrast, does not go up by 4%: the vendor's bill, the volume stored by a factor of 390 and the complexity of the ingestion process all multiply.

Measure before you buy. The correct sequence is: decompose where your error comes from, compute which part the new data would touch, and only then look at the price.