3. Alerts: when to wake you up and when not to

Level 10: Knowing what is going on

An alert is a formal request that a human being drop what they are doing and act. That is the complete definition, and everything else follows from it. If there is nothing a person can do right now, it is not an alert: it is a figure that goes to a dashboard or, at most, a ticket for Monday.

The criterion: immediate human action, and nothing else

The temptation is to alert on everything that goes wrong, because it looks prudent. It is not. Always ask yourself two things before creating an alert: what exactly would the person receiving it do, and what happens if they do not receive it until tomorrow. If the answer to the first is "look at it" and to the second "nothing serious", it is not an alert.

It is worth having three levels and respecting them:

• Critical: rings a phone, whatever the hour. There is money at risk and there is a specific action to take.
• Warning: opens a ticket. It is looked at during working hours, it breaks nothing meanwhile.
• Informational: goes to the dashboard and to the metrics. It never interrupts anybody.

A log error and an alert are not the same thing, and confusing them is the origin of almost every unbearable notification system. The log stores everything that happened so it can be reconstructed later; the alert picks from it the handful of things that cannot wait. That an event is written at error level means somebody will have to look at it some day, not that they have to look at it now.

Alert fatigue: the one that fires every day no longer exists

An alert that goes off daily and always resolves itself is worse than having no alert, and the reason is not psychological but operational: it trains the team to ignore that channel, and it drags with it every other alert sharing the channel. The day the warning is real, it will arrive at the same place nobody is looking.

The cure is not turning the volume down or sending it to another folder. It is one of three things: raising the threshold to the point where it genuinely demands action, demoting it to the dashboard if it never demands action, or fixing the cause. And you also have to throttle the sending itself, so that a failure repeating a hundred times in ten minutes generates one notification and not a hundred.

import time

ULTIMO_ENVIO = {}

def alertar(urgencia, clave, mensaje, silencio_s=1800):
    ahora = time.time()
    if ahora - ULTIMO_ENVIO.get(clave, 0) < silencio_s:
        M.contar("alertas_silenciadas", clave=clave)
        return False
    ULTIMO_ENVIO[clave] = ahora
    M.contar("alertas_enviadas", urgencia=urgencia, clave=clave)
    registrar(logging.WARNING, "alerta", urgencia=urgencia,
              clave=clave, mensaje=mensaje)
    if urgencia == "critico":
        llamar_al_telefono(mensaje)
    elif urgencia == "aviso":
        abrir_ticket(clave, mensaje)
    return True

Measure your own alerts. The number of critical alerts sent per week and the percentage of them that ended in a real action are the best X-ray of whether your notification system is healthy or has already turned into noise.

Alerting on absence: the failure that generates no errors

Here is the part almost nobody implements and that prevents the most grief. Alerting systems are usually built by looking at what happens: exceptions, rejections, error codes. But the most dangerous failure in a trading system produces no error, it produces silence. If the scheduler dies at eight in the morning, the 9:30 task does not run and there is no exception to catch, because there is no code running. A counter of errors per hour will never fire.

That is why you have to declare explicitly what is expected to happen and check that it has happened.

TAREAS_ESPERADAS = {
    "descarga_precios": "09:00",
    "generar_senales":  "09:30",
    "enviar_ordenes":   "09:35",
    "reconciliacion":   "22:15",
}

def comprobar_tareas(ejecutadas_hoy, ahora_hhmm, margen_min=10):
    for tarea, hora in sorted(TAREAS_ESPERADAS.items()):
        limite = sumar_minutos(hora, margen_min)
        if ahora_hhmm > limite and tarea not in ejecutadas_hoy:
            alertar("critico", "tarea_no_ejecutada:" + tarea,
                    "la tarea %s deberia haber corrido a las %s" % (tarea, hora))

A logged error is a nuisance; a task that has not run is a strategy that is not trading or, worse, a position that should have been closed and is still open.

The heartbeat: confirming the system is still alive

The natural complement is the heartbeat. The process writes a mark every minute with the time and the task in progress, and a watchdog checks that this mark does not go stale. The condition that makes the whole arrangement useful is that the watchdog runs on another machine and in another process: if it lives inside the system it watches, it dies with it precisely when it is needed.

import json, os

RUTA_LATIDO = "/var/run/estrategia/latido.json"

def latir(tarea, estado="ok"):
    with open(RUTA_LATIDO, "w") as f:
        json.dump({"tarea": tarea, "estado": estado, "ts": time.time()}, f)

def vigilar(edad_maxima_s=300):
    if not os.path.exists(RUTA_LATIDO):
        alertar("critico", "sin_latido", "no existe el fichero de latido")
        return
    with open(RUTA_LATIDO) as f:
        latido = json.load(f)
    edad = time.time() - latido["ts"]
    if edad > edad_maxima_s:
        alertar("critico", "latido_viejo",
                "ultimo latido hace %d s en la tarea %s" % (int(edad), latido["tarea"]))

Worked scenario: the alert that stopped being read

A scenario invented for teaching purposes. A system sends on average about twelve "high data delay" alerts a day for three months; all of them resolve themselves within a couple of minutes. The team stops looking at them. In the fourth month the provider changes a format and the system trades half a session on the previous day's prices: the alert fires, indistinguishable from the thousand before it, and nobody opens it until the afternoon. The fix was to raise the threshold to the delay that makes trading unsafe, send the rest to the dashboard, and add a check of expected tasks. The volume dropped to fewer than one critical alert a week, and those do get read.

The lesson, before moving on to the last module: a channel with zero alerts for a month can be a magnificent sign or a terrible one, and the only thing that tells them apart is the heartbeat. If the watchdog keeps confirming everything is running, silence is good news; if nobody is confirming anything, silence means absolutely nothing.