3. What to do with an open position when the system goes down
Level 11: When it fails
It is 15:42. You have four open positions, one live limit order in the market and another sent eight seconds ago that you have no confirmation for yet. The process dies. At 16:05 it is back up. This module is about those twenty-three minutes and, above all, about the first decision the system takes on coming back, which is where the money is really lost.
What not to do: start up and believe the disk
The naive startup is to read the state file, see that it says four positions and one live order, load it into memory and carry on trading from there. It is what nearly everybody does the first time and it is wrong for one specific reason: what you saved to disk is your last intention, not a fact about the world. Between the last save and the process's death there is room for a complete fill, a partial one, a cancellation by the broker or the filling of that order you had no confirmation for.
The broker is the source of truth; your file is what you believed. And there is one especially treacherous case: the order sent without confirmation. If you assume it did not go out, you can end up with a double position. If you assume it did, you can be left waiting for a fill that never happened.
The startup protocol with an open position
You always start in read-only mode and compare before touching anything:
def arranque_seguro(broker, estado, politica):
real_pos = broker.posiciones()
real_ord = broker.ordenes_abiertas()
intencion = estado.ultima_intencion()
diffs = comparar(intencion, real_pos, real_ord)
if not diffs:
return 'coincide: reanudar normal'
if politica.puede_quedarse_quieta and not politica.autocierre:
estado.freno('divergencia al arrancar: %r' % diffs)
return 'parado: requiere intervencion humana'
for o in real_ord:
if not estado.reconoce(o.id):
estado.freno('orden desconocida %s' % o.id)
return 'parado: orden no reconocida'
return politica.resolver(diffs)Three details. The system cancels or closes nothing on its own before having compared. An open order your records do not recognize is never canceled blind, because it may belong to another strategy, to another account or to a human hand. And when the divergence cannot be explained, the way out is not a trade: it is stopping and calling somebody.
Strategies that can stay still and strategies that cannot
Now the question that organizes everything else: if the process dies now and nobody touches it for twenty-four hours, what is the worst possible loss?
There are strategies where the answer is a nuisance. A monthly rotation of cash equities, unleveraged and diversified, can go a day without supervision: what it risks by standing still is ordinary market risk, the same one taken by anybody who goes on holiday. That one can stay still, and its safe mode is to do nothing.
And there are strategies where the answer keeps you awake. Anything leveraged, naked option selling, concentrated intraday, everything that depends on a programmed exit to bound the loss. There the risk profile you measured presupposes that somebody is watching, and standing still is not neutral: it is a new position you never decided to take.
This classification is made beforehand, in the cold, and is written into the configuration alongside the startup policy. In the heat of the moment, at 16:05, with an open position, nobody decides well between closing and holding. That day you do not decide what to do: you execute what you had already decided.
Protective orders at the broker: the net that outlives your server
If your strategy cannot stay still, the only net that works is the one that does not live on your machine. A protective order placed at the broker is still there even if your process dies and the whole data center burns down. It is not free, and it is worth knowing the price before paying it:
- It executes at the worst moment. A market stop is triggered exactly when there is a liquidity vacuum, and you get out far from the level you wrote down.
- It exposes you to wicks. A one-second spike, even an erroneous one, touches it and takes you out of a position the strategy would have kept.
- It is a second piece of state that goes out of sync. Every time you change the position you have to change the protection.
- It depends on the broker when you disconnect. Some markets automatically cancel orders when the connection is lost, exactly the opposite of what you want here. You have to know which of the two yours does, and know it beforehand, not on the day of the scare.
That is why the protection is placed in the same step as the position is opened, not in a later cycle, and reconciliation checks both directions:
def abrir_con_red(broker, estado, simbolo, cantidad, stop):
orden = broker.mercado(simbolo, cantidad)
broker.esperar_ejecucion(orden)
proteccion = broker.stop(simbolo, -cantidad, stop)
estado.vincular(orden.id, proteccion.id)
return proteccion
def protecciones_huerfanas(broker):
pos = {p.simbolo: p.cantidad for p in broker.posiciones()}
return [o for o in broker.ordenes_abiertas()
if o.tipo == 'stop' and pos.get(o.simbolo, 0) == 0]An orphan protection is not a net: it is an order that will open an opposite position the moment the price touches it.
Worked scenario
What follows is an example built for the course, not a real incident. Intraday system, equity of 100,000 euros, three long positions worth 60,000 in total and one live limit sell order. The process dies at 15:42 for lack of memory. The supervisor relaunches it at 15:44.
With no startup protocol, the system reads the state on disk, does not see that the limit order was filled at 15:43 and sends the same sell again: it ends up short in a name it never wanted to sell twice, and finds out at the close. With a protocol, the startup asks the broker for positions, sees that one line is gone, finds no explanation in the recorded intention, activates the brake and sends an alert. Cost of the second path: twenty minutes without trading and a phone call. Cost of the first: an unwanted short position, plus whatever the market does with it until somebody looks.