1. Scheduling execution: periodic tasks, queues and workflows

Level 9: Orchestration and the calendar

You already have a strategy that generates correct orders when you launch it yourself, by hand, with the laptop open. Production begins with a boring question: who launches it when you are not there. The answer is not an infinite loop with a sleep inside; that is the fastest way to discover three weeks later that the process has been dead for nine days and nobody noticed. Triggering work is a problem with known solutions and choosing badly costs money.

The three ways of triggering work

• Scheduled periodic task. The trigger is the clock: cron, a systemd timer or a scheduler inside the process itself. You say "this, at this time" and the system takes care of it. It is the option with the fewest moving parts and the only one still standing when everything else has fallen over.
• Message queue. The trigger is an event. Somebody publishes a message (there is a new candle, the broker has confirmed a fill) and one or several consumers pick it up. The producer does not wait for the consumer: if the consumer is down, the work piles up in the queue instead of being lost. It is the natural way to absorb bursts and to spread load across several machines.
• Workflow orchestrator. It also triggers on the clock, but it additionally knows the graph: which task depends on which, what state each one is in today, and it lets you rerun only the piece that failed without repeating what already went well.

When the simplest one is enough

The practical rule: choose the smallest mechanism that survives your worst day, not your normal day. If your strategy decides once a day, has four steps and you can redo the whole session by hand in five minutes, a scheduled task is more than enough. Adding an orchestrator there does not give you robustness: it gives you one more service to maintain, update and watch, and one that also falls over.

You need a queue when work arrives pushed and in bursts: broker messages, webhooks, market events you cannot afford to lose while you restart. You need an orchestrator when the number of tasks and dependencies no longer fits in your head, when you want to recompute three late days without touching the rest, or when several people depend on knowing whether today's data step finished properly.

And a warning that saves grief: every mechanism you add is one more component that can fall over and that has a log of its own nobody reads. A badly watched orchestrator is no more robust than a cron job: it is the same thing with more failure surface and the false feeling that somebody is looking.

Overlap: two copies on the same state

Here is the classic failure. You schedule a task every minute. It takes ninety seconds. From the second minute onwards you permanently have two live copies: the one that has just started and the previous one, which has not finished yet. Both read the same position, both compute the same signal and both send the same order. You end up with double the size you wanted, and with two processes writing the same state file in an order nobody has decided.

The dangerous part is that the number of copies need not stop at two. If those two copies compete for the same connection, the same rate limit or the same disk, each takes longer, and the overlap grows. A system that was already tight becomes a saturated system without any component having failed.

A lock that does not lie

The solution is a mutual exclusion lock: before working, the task tries to acquire it; if it cannot, it withdraws without doing anything and records the fact. A usable lock has three properties: it is acquired atomically, you know who holds it, and it is released on its own if the owner dies. A marker file created and deleted by hand fails on exactly the third: if the process dies abruptly, the file stays and blocks the system forever.

import fcntl
import os
from contextlib import contextmanager

class YaEstaCorriendo(Exception):
    pass

@contextmanager
def cerrojo(ruta):
    """File lock. The operating system releases it if the process dies."""
    f = open(ruta, "a+")
    try:
        fcntl.flock(f, fcntl.LOCK_EX | fcntl.LOCK_NB)
    except BlockingIOError:
        f.close()
        raise YaEstaCorriendo(ruta)
    try:
        f.write(str(os.getpid()))
        f.flush()
        yield
    finally:
        fcntl.flock(f, fcntl.LOCK_UN)
        f.close()

The usage is deliberately boring: if the lock is taken, this minute's run is not queued or postponed, it simply does not happen.

import logging

def main():
    try:
        with cerrojo("/var/run/estrategia.lock"):
            ciclo_de_trading()
    except YaEstaCorriendo:
        logging.warning("ciclo anterior todavia vivo: esta ejecucion se descarta")

If instead of cron you use a scheduler inside the process, ask for the same thing through configuration: a single simultaneous instance per task, coalescing late runs into one and discarding those that arrive too late.

from apscheduler.schedulers.blocking import BlockingScheduler

sched = BlockingScheduler(timezone="UTC")
sched.add_job(
    ciclo_de_trading,
    "cron", minute="*",
    max_instances=1,       # never two copies of this task
    coalesce=True,         # several late ones run only once
    misfire_grace_time=20  # if it arrives more than 20 s late, it is discarded
)

Worked scenario

A worked scenario, with example figures, not a public case. A task running every minute walks a universe of sixty names and queries each one's price with a call that takes 1.4 seconds on average: 84 seconds of network alone, plus the computation. With the machine idle it fitted inside the minute; with the market busy, it did not. For twenty-six minutes there were two overlapping copies and duplicate orders were sent on fourteen of the sixty names.

The two fixes are independent and both are needed: the lock, which prevents the damage, and batching the sixty queries into a single bulk request, which brings the cycle down to about four seconds and removes the cause. The lock without the second one leaves you a system that does not break but that skips half its runs. This is education, not financial advice.