3. Deploying without breaking things: versions, rollback and hot changes

Level 12: Testing, deploying and taking the step

A deployment is changing a piece of a system with money on the table. The difference from any other program is that here the state does not live only on your disk: it also lives at the broker, in the form of live orders and open positions that have no idea you have changed version.

Deployment does not coincide with the market being open

The window is defined in the same calendar Level 9's scheduler already knows, and it is not enough for the market to be closed. You have to stop the process deliberately, check that nothing is left live and only then change anything. That order matters: checking with the process still running is checking a snapshot that can expire while you copy files.

from datetime import datetime, timezone

def guardia_de_despliegue(calendario, broker, estado):
    ahora = datetime.now(timezone.utc)
    if calendario.mercado_abierto(ahora):
        raise SystemExit("mercado abierto: despliegue abortado")
    if broker.ordenes_abiertas():
        raise SystemExit("hay ordenes vivas: cancelalas o espera a que venzan")
    if estado.fase != "reposo":
        raise SystemExit("la maquina de estados no esta en reposo")
    return True

Going back in a minute

The criterion is not "having a backup": it is that the previous version is still installed and startable with a single command. The simplest form that works: each version in its own directory, with its own environment, and a link called current pointing at the one in use. Deploying is moving the link and starting; rolling back is moving it to the previous directory and starting. Seconds, not a rebuild.

Three conditions for that to be genuinely true:

• The dependencies live inside the version's directory, with pinned versions. If the old one had to reinstall packages, there is no rollback.
• The configuration is versioned alongside the code, never edited by hand on the server.
• There is a smoke test after startup and before enabling order sending: connect, ask for positions, ask for the broker's clock, check the state's version.

And the detail almost nobody honors: the rollback has to be rehearsed. A rollback that has never been executed is an intention, not a procedure.

The state has to survive the version change

This is where apparently innocent deployments break. Level 8's persistent state was written by the old version and is going to be read by the new one. If the new one adds a mandatory field, changes the meaning of an existing one or renames a phase, startup fails; or worse, it does not fail and does something else. The discipline is that of any database: the state carries a schema version number and startup checks it.

ESQUEMA_ESPERADO = 7

def migrar_6_a_7(e):
    e = dict(e)
    e["precio_referencia"] = e.get("precio_decision")
    e["version"] = 7
    return e

MIGRACIONES = {6: migrar_6_a_7}

def cargar_estado(bruto):
    v = bruto.get("version", 0)
    while v < ESQUEMA_ESPERADO:
        if v not in MIGRACIONES:
            raise RuntimeError("no se como migrar el estado desde esa version")
        bruto = MIGRACIONES[v](bruto)
        v = bruto["version"]
    if v > ESQUEMA_ESPERADO:
        raise RuntimeError("estado escrito por una version mas nueva: no arranco")
    return bruto

The dangerous case is the last one: an old version reading state written by a new one. It happens precisely when you roll back, which is the worst possible moment to discover it. That is why the rollback rehearsal is done with the new version's state in front of you, not with a clean one.

Live orders while you deploy

Three policies, and you choose one in writing before you need it:

• Cancel everything before stopping. The simplest and the right one for almost everybody. It requires the strategy to tolerate being left with no pending orders.
• Leave them live and adopt them on startup. The new version queries the open orders and takes them into its state by intent identifier. This only works if those identifiers survive the version change.
• Let them expire. Valid until the close and deploy afterwards. Tidy, but useless if you trade across several time zones.

What is not a policy is stopping the process and hoping there was nothing there.

Hot configuration is the fast route to disaster

Lowering an exposure limit by editing a file on the server at eleven in the morning works, and that is the problem: it works without leaving a trace. Nobody will know a month from now which value was active, the next deployment will silently revert it and Level 11's incident review will be done on a configuration that was not the one in place. Configuration is treated as code: repository, review, the same deployment procedure, and the system logs at startup the hash of the file it has loaded. If some parameter must be changeable hot, and Level 11's kill switches need it, that change goes through a route that writes down who, when, previous value and new value.

Knight Capital, August 1, 2012

Knight Capital was one of the largest market makers in the United States. It deployed new code to its execution servers and on one of the eight the update never arrived. The new code reused a configuration flag that in the old code switched on a test function retired years earlier. When the market opened, seven servers did what was expected and the eighth started running the old behavior, buying and selling without a brake at market pace.

In less than an hour the pre-tax loss came to around 440 million dollars and left the firm on the brink of collapse; it survived with an emergency capital injection and ended up absorbed by another company, and the SEC later sanctioned it for deficiencies in its controls. Four things from this module failed at once: a deployment that did not verify that the eight machines ended up identical, a flag reused rather than retired, dead code still installed and no automatic cut-off on cumulative loss.