1. State is the problem: what to store, where and why

Level 8: State and reliable execution

A backtest has no state. It starts, walks a vector of prices and finishes. If you kill it halfway you relaunch it and nothing has happened. A production system is precisely the opposite: it exists to accumulate state over time, and that state is worth money. If you lose it you do not lose a run, you lose the ability to know what you hold in the market.

What state is made of

When we say "the state" of a trading system we are talking about five different things, and it is best not to mix them up because each is stored and recovered differently:

• Open positions. How many units of each symbol you hold and at what average price you got in.
• Live orders. What you have sent that has not finished yet: accepted, resting in the book, partially filled.
• Cash and buying power. How much is genuinely available once you deduct the holds from orders that are still live.
• Last bar processed. The timestamp of the last candle you ran the model on. Without this, on restart you do not know whether you have to reprocess or not, and you end up emitting the same signal twice or skipping one.
• Pending signals. Decisions already taken that have not yet turned into a sent order. It is the piece of state that gets forgotten most often and the one that costs most.

Why the process's memory is no good

A process dies. This is not a remote hypothesis: it dies because you deploy a new version, because the operating system kills it when it runs out of memory, because an uncaught exception bubbles all the way up, because the server reboots for a kernel patch, or because somebody stops it by hand. And here is what matters: the market does not find out that your Python has died. The position stays open, the orders stay in the book and the fills keep happening while your process does not exist.

An in-memory dictionary also has a defect worse than disappearing: it lies quietly. While the process lives, that variable looks like absolute truth, and nobody ever checks whether it still matches reality.

The broker is the source of truth; your database is a cache

This is the rule that organizes the whole level: the broker is the source of truth for positions and orders; your database is a cache and a journal of intentions. In any discrepancy the broker wins, always, no argument. Your job is not to be right, it is to detect the difference and stop trading until you understand it. The formal process of comparing and adjusting has its own name, reconciliation, and it is covered in depth in Level 10; here it is enough that you accept who is in charge.

That said, there is information the broker does not have and that only exists on your side:

• Why you sent that order: which strategy, which signal, which bar generated it.
• Which signals you have decided on and have not yet sent.
• How far you have processed the data stream.
• Your client identifiers and the link between each order and its strategy.

From that comes the practical rule: always ask the broker for positions, open orders and cash; store on disk everything the broker can never give back to you.

Writing to disk without leaving the file half-done

Opening a file in write mode empties it immediately. If the process dies between that truncation and the end of the dump, you are left with a truncated or empty file, and on startup the system does not find corrupt state: it finds none. The solution is never to write over the good one.

import json
import os
import tempfile

def guardar_atomico(ruta, datos):
    directorio = os.path.dirname(os.path.abspath(ruta))
    fd, temporal = tempfile.mkstemp(dir=directorio, suffix='.tmp')
    try:
        with os.fdopen(fd, 'w', encoding='utf-8') as f:
            json.dump(datos, f, ensure_ascii=False, indent=2)
            f.flush()
            os.fsync(f.fileno())
        os.replace(temporal, ruta)
    except BaseException:
        if os.path.exists(temporal):
            os.remove(temporal)
        raise

Three details that are not decoration. The temporary file is created in the same directory because the rename is only atomic within the same file system. The fsync forces the bytes onto the disk for real, not into the operating system's buffer, before the rename declares them official. And os.replace is atomic on both Linux and Windows: any reader will see either the old file whole or the new file whole, never a mixture.

Better still: let a database do it

If you are going to write several things that must change together, a JSON file falls short very quickly. SQLite ships in the standard library, needs no server and gives you transactions and durability ready made.

import sqlite3

con = sqlite3.connect('estado.db', timeout=30)
con.execute('PRAGMA journal_mode=WAL')
con.execute('PRAGMA synchronous=FULL')
con.execute("""
CREATE TABLE IF NOT EXISTS ordenes (
    client_order_id TEXT PRIMARY KEY,
    simbolo         TEXT NOT NULL,
    lado            TEXT NOT NULL,
    cantidad        REAL NOT NULL,
    estado          TEXT NOT NULL,
    barra           TEXT NOT NULL,
    broker_id       TEXT,
    creada_en       TEXT NOT NULL
)""")
con.commit()

Look at the first column: it is the primary key. That constraint, which looks like a footnote, is the pillar of the whole next module.

Worked scenario: forty seconds of blindness

A scenario built for the course, not a documented case. You deploy at 15:44:10 and the process takes forty seconds to come back. In that gap, a buy order for 300 units that was already in the book is partially filled: 120 units. The process starts up, reads its local file, which was saved before the deployment and says "0 filled, order live", and decides it is still flat. The next signal computes the size from scratch and sends another buy for 300. Result: 420 units where the model believed it had 300, 40% extra exposure that no risk rule authorized, and all because of trusting a forty-second-old cache instead of asking. The failure was not the market's nor the broker's: it was believing that a file knows what is going on outside.