1. The five places a trading system fails

Level 11: When it fails

A trading system never breaks where you were looking. You spend weeks tuning the signal and the day the alarm goes off it turns out the disk was full or the provider sent a price of zero. This module is the map of the five places a production system really fails, ordered by real frequency: for each one, how it is detected and what the safe default response is.

The inventory, ordered by real frequency

In a system with a year of daily trading, the distribution of scares looks something like this:

• The data. Several times a month. By some distance the layer that fails most.
• The network and the broker. Every few weeks: a timeout, a rejection, an expired session.
• Your own code. Concentrated around each deployment, almost never between them.
• The infrastructure. A couple of times a year, and almost always in the small hours.
• The human. The least frequent of the five and, by some distance, the most expensive.

Cutting across all five there is a single rule: when in doubt, do not trade. A stopped system misses an opportunity; one that trades on false information loses money and creates positions that then have to be unwound. The asymmetry rules.

Layer 1: the data

They fail in four ways, and they are not ordered by severity the way you would expect:

• They do not arrive. The benign case: there is a gap where a number should be, and you can see it.
• They arrive late. The dangerous one. Nothing fails: you receive a well-formed price that was true forty minutes ago and your code treats it as the current one.
• They arrive incomplete. Six tickers out of five hundred are missing. The universe mean is computed anyway, on 494, and nobody notices.
• They arrive absurd. A zero, a negative, a price not adjusted for a split already applied, a decimal point shifted.

The first two are detected with a clock; the last two, with a count and a sanity band.

FRESCURA_MAXIMA_S = 90
BANDA_MAXIMA = 0.20

def barra_utilizable(barra, ultimo_cierre_bueno, ahora):
    if barra is None or barra.get('close') is None:
        return False, 'sin dato'
    edad = ahora - barra['ts']
    if edad > FRESCURA_MAXIMA_S:
        return False, 'dato viejo: %d s' % edad
    precio = barra['close']
    if precio <= 0:
        return False, 'precio no positivo'
    if ultimo_cierre_bueno:
        salto = abs(precio / ultimo_cierre_bueno - 1)
        if salto > BANDA_MAXIMA:
            return False, 'salto del %.1f por ciento' % (salto * 100)
    return True, ''

The function corrects nothing: it does not interpolate, does not carry the last good price forward, does not fill the gap. It only says yes or no, and why. The threshold is not universal either: in a large-cap stock 20% intraday is suspicious, in a crypto it is not. The number is a parameter; the check is not.

Layer 2: the network and the broker

Four ways here too, from best to worst. Down: it refuses the connection and you know within a second. Rejecting orders: it says no, with an explicit reason. With an expired session: it returns a 401 and a renewal is enough. And slow: the request does not fail, it takes fourteen seconds, and by the time the reply arrives your decision is no longer valid.

Every call carries an explicit maximum wait: a request with no limit leaves the process hanging, the quietest way of being down. And you watch latency, not just the error rate: a broker going from 200 milliseconds to six seconds is broken even if its replies are correct. When either of the two goes out of range, order sending is cut off until a full cycle comes back healthy.

Layer 3: your code, and layer 4: the machine it lives on

The code failure that matters is not the one that blows up on startup; that one is a gift, because you never get to trade. The one that matters is the uncaught exception halfway through a run: you have sent three of the rotation's eight orders and the process dies. That is not an error, that is half a portfolio.

Infrastructure fails in three flavors: the disk fills up, often because of the logs you write; memory runs out and the operating system kills the process; or the machine reboots on its own. None of them gives warning with a Python exception.

import shutil

MARGEN_DISCO_MB = 500

def preparado_para_operar(ruta_datos):
    libre_mb = shutil.disk_usage(ruta_datos).free / 1e6
    if libre_mb < MARGEN_DISCO_MB:
        return False, 'disco: solo %.0f MB libres' % libre_mb
    return True, ''

def ciclo_protegido(motor, freno):
    try:
        motor.ejecutar_ciclo()
    except Exception as e:
        freno.activar('excepcion no prevista: %r' % e)
        raise

The disk check runs before every cycle, not once at startup. And the wrapper does the only reasonable thing in the face of an unforeseen exception: it activates the brake, leaves a record and re-raises. What it does not do is swallow it and carry on, because a system that continues after an error it does not understand is trading on a state it can no longer justify.

Layer 5: the human who touches something while the system is trading

Somebody closes a position from their phone because they could see it losing. Somebody edits the configuration at half past eleven. Somebody restarts the process so it picks up the change. None of the three is bad in itself; what is bad is that the system does not find out and carries on believing a state that no longer exists. Detection is reconciliation against the broker, and the response is an explicit maintenance flag: before touching anything by hand the system is stopped and it is written down that it is stopped. That way, faced with a divergence, the system can tell apart it does not square because of a failure from it does not square because somebody said so.

Real case: Knight Capital, August 1, 2012

Knight Capital Group, one of the largest market makers in the United States, deployed new code to its routing servers. An old component that had gone unused for years was left active and, at the open, it started firing orders into the market in a loop. In less than an hour, about 45 minutes, the loss came to around 440 million dollars, more than the company's profit for the previous year. Knight was absorbed a few months later.

It is the canonical example because it failed in several layers at once: old code nobody cleaned up, a deployment that did not reach every machine and automated warnings nobody was obliged to escalate. And the system had no limit of its own capable of saying this is now too much. That is what the next module is about.