4. The incident review: turning a scare into a rule

Level 11: When it fails

Something has happened. It is fixed now, the system is trading again and everybody wants to forget about it. That is exactly the moment a team decides whether it is going to improve or repeat the same incident seven months from now in a different costume. The incident review is the mechanism that turns a scare into a rule, and it is the piece that separates a system that ages well from one that merely accumulates scars.

No culprits, and not out of politeness

The rule of not looking for culprits is not a courtesy: it is what makes the information come out. The moment somebody senses that the review ends with a name, the uncomfortable details stop being told, and it is precisely the uncomfortable details that explain the failure. Besides, the person who pressed the button is almost always the last link in a chain that had been assembled months earlier.

No culprits does not mean no owners of the fix. Every corrective action comes out of the review with a name and a date, or it does not exist.

The timeline with timestamps

The review always starts with the same mental table: what happened and when, with the exact time and in a single time zone. And here you see what all that logging you built earlier was for:

• Time of the first anomalous event, which almost never coincides with the time somebody noticed.
• Time of detection, and by what means: an automated alarm, a customer, sheer chance.
• Time of the first action, and what was done.
• Time of containment and time of resolution, which are not the same thing either.

The distance between the first event and detection is the most revealing metric of all. If your incident started at 09:31 and you detected it at 11:40 because somebody rang you, your main problem is not the failure: it is that you were blind for two hours. Without reliable timestamps in your own logs, this timeline cannot be built; the broker's statement tells you what was executed, but not what your system believed at each instant or why it decided what it decided.

Apparent cause and root cause

The apparent cause is the last thing that broke. The root cause is the reason that breakage was able to reach the money. A simple method is to keep asking why until the answer stops being technical:

• The system sold twice what it should have. Why? Because the process restarted and resent the order.
• Why did it resend it? Because on startup it believed the state on disk.
• Why did it believe it? Because startup did not query the broker.
• Why did it not query it? Because nobody had ever written down what should happen on starting up with an open position.

The apparent cause was the restart. The root cause is that no startup policy existed. Note that the chain runs through the three previous modules: an infrastructure failure, a startup decision and the absence of a limit to catch it.

Every incident ends in a new check or a new limit

This is the module's practical rule, and it is non-negotiable. A review that ends in we will be more careful, deployments need better review or the operator now knows not to do that has produced nothing. Attentiveness is not a mechanism: it cannot be deployed, it cannot be tested and it runs out by three in the afternoon. There are three valid outputs: a new automated check, a new hard limit, or removing the possibility of making the mistake.

def test_no_reenvia_orden_tras_reinicio():
    """Reproduces incident 2026-03-11 with its exact data."""
    broker = BrokerFalso(posiciones={'ACME': 0})
    estado = EstadoFalso(intencion={'ACME': -300, 'confirmada': False})

    resultado = arranque_seguro(broker, estado, politica_quieta)

    assert estado.freno_activo()
    assert broker.ordenes_enviadas == []
    assert 'intervencion' in resultado

That test is the incident's real product. As long as it exists, the failure cannot come back in silence: anybody who breaks that property will know before deploying. And the incident record is kept with the same discipline, versioned alongside the code:

incidente = {
    'id': '2026-03-11-01',
    'primer_evento': '2026-03-11T09:31:04Z',
    'deteccion': '2026-03-11T11:40:12Z',
    'medio_deteccion': 'aviso de un tercero',
    'causa_aparente': 'reinicio del proceso',
    'causa_raiz': 'no existia politica de arranque con posicion abierta',
    'impacto_eur': -4120,
    'control_nuevo': 'arranque_seguro obligatorio + freno por divergencia',
    'prueba': 'test_no_reenvia_orden_tras_reinicio',
    'responsable': 'pol',
    'fecha_limite': '2026-03-18',
}

The field that hurts most is the means of detection. If most of your incidents are discovered by somebody outside, your problem is not in the code: it is that your alarms are not watching what matters.

Real case: Knight's timeline and what would have changed

Let us go back to August 1, 2012. The apparent cause of that incident was an incomplete deployment: new code that did not reach every routing server, leaving an old component active. The root cause is more uncomfortable and far more useful: there was no automated verification that the deployment had ended up identical on every machine, nor a procedure obliging anybody to act on the automated warnings that arrived before the open, nor a limit of their own capable of saying that that morning's order volume was impossible.

That is the summary of the entire level. Detection took too long, the response was not written down in advance and there was no cap acting without asking permission. Some 440 million dollars in less than an hour, and the three corrective actions that would have prevented it are deadly dull: check the deployment, escalate the warnings and count the orders.