3. Dependencies between tasks and partial failures

Level 9: Orchestration and the calendar

A strategy in production is not one task, it is five chained together: download data, compute signals, size positions, execute and reconcile. When all five work, the chain is invisible. The day the first returns half the data and the second computes anyway, the chain decides your money. This module is about that: about what the system does when a link fails halfway. We are not talking here about retrying a particular call with a growing wait, but about what happens to the whole task and to everything that depends on it.

The chain and its real dependencies

• Downloading data depends on the provider responding and on the universe being complete.
• Computing signals depends on the download having finished properly, not on it having finished.
• Sizing depends on the signals and on the real capital and positions, not on the ones you thought you had.
• Executing depends on the sizing and on the previous module's calendar.
• Reconciling depends on execution, but it is the only one that must also run when execution failed, because it is the one that finds out what actually happened.

That penultimate sentence is the key to the module: finishing is not the same as finishing well, and the system has to tell them apart unambiguously.

Declare the dependency, do not trust the ordering

Putting the calls one after another in a file declares nothing: if the first throws an exception somebody caught for convenience, the second runs anyway. Declaring a dependency means a task only starts if the recorded state of its predecessors is correct, and that this state outlives the process.

from dataclasses import dataclass

OK, FALLO, OMITIDA = "OK", "FALLO", "OMITIDA"

@dataclass(frozen=True)
class Tarea:
    nombre: str
    depende_de: tuple = ()

def ejecutar_flujo(tareas, funciones):
    estado = {}
    for t in tareas:
        if any(estado.get(d) != OK for d in t.depende_de):
            estado[t.nombre] = OMITIDA          # not even attempted
            continue
        try:
            funciones[t.nombre]()
            estado[t.nombre] = OK
        except Exception:
            estado[t.nombre] = FALLO            # and it drags its dependants down
    return estado

What matters about this loop is not what it does, but what it does not do: there is no path by which execution starts if sizing ended in FALLO. The omission propagates through the graph on its own and is recorded, which is different from not having happened.

It is also worth fixing the number of attempts and the point at which the task gives up. Something retried indefinitely is not resilient: it is a task that never finishes and that stops the workflow reaching reconciliation, which is exactly what you need to read when something goes wrong. Three attempts and a declared failure are worth more than an endless loop.

Incomplete data: the signal that looks fine

The most expensive failure is not the noisy one, it is the silent one. The provider responds with code 200 and returns 380 of the universe's 500 names. The signal is computed, a number comes out, and that number is computed over a different universe from the one you defined. If your strategy ranks by percentile, 120 missing names move every cut-off; if it compares against the universe mean, the mean is a different one.

That is why the completeness check is not a warning in the log: it is a hard failure of the data task.

class DatosIncompletos(Exception):
    pass

def validar_descarga(df, universo_esperado, sesion):
    faltan = set(universo_esperado) - set(df["simbolo"].unique())
    if faltan:
        raise DatosIncompletos("faltan " + str(len(faltan)) + " valores del universo")
    viejos = df[df["fecha"] < sesion]
    if not viejos.empty:
        raise DatosIncompletos("hay filas con fecha anterior a la sesion de hoy")
    return df

Filling the gaps with yesterday's price looks prudent and is the opposite: it turns a visible failure into a false signal nobody is going to review.

Retry or skip: they are not interchangeable

Retrying a task means: I believe the failure is transient and that trying again may give a correct result. Skipping a task means: I accept carrying on without it. They are different decisions and only the first is almost always safe.

• You retry what is transient and idempotent: a download that timed out, a position query that returned a server error.
• You skip what is optional and only what is optional: emailing the daily report, updating a dashboard, exporting an analysis file.
• You never skip a link that decisions depend on: if sizing failed, execution does not skip the step, it is canceled.

There is an asymmetry worth bearing in mind: nearly every error at this level leads to trading too much, not too little. Incomplete data generates orders, and so does a signal computed over a truncated universe. A system that shuts down when in doubt errs in the cheap direction.

The rule that sums up the whole module: when in doubt, do not trade. A day without trading costs a day of expected return; a day trading on incomplete data can cost the whole portfolio, and it also pollutes the log you will use tomorrow to try to understand what happened.

def clasificar(exc):
    if isinstance(exc, (TimeoutError, ConnectionError)):
        return "reintentable"
    if isinstance(exc, DatosIncompletos):
        return "fatal"          # never blindly retried, never ignored
    return "fatal"              # the unknown is treated as serious, not as minor

Worked scenario

A worked scenario, with example figures. A universe of 500 names; the provider delivers 380 and responds with success. With no validation, the strategy computes its deciles over 380 names: the cut-offs shift and 41 names change decile, of which 12 go from being out of the portfolio to being in the buying decile. The system executes twelve entries the real strategy would never have asked for, and the next day's reconciliation finds them in the portfolio with no explanation.

With the validation, the data task ends in FALLO, signals and execution end in OMITIDA, reconciliation runs anyway and confirms the portfolio is unchanged from yesterday. The cost is one day without rebalancing. Education, not financial advice.