3. Retries, timeouts and rate limits

Level 8: State and reliable execution

You already know how to retry without duplicating. What is left is knowing when to retry, how long to wait between attempts and when to give up. A badly written retry loop is worse than having none at all: it turns a one-second transient failure into a fifteen-minute block for excessive requests.

Not every error deserves a retry

The distinction is simple to state: if the error describes a decision by the broker, do not retry it; if it describes a transport or capacity problem, retry it.

• Retryable. Connection failure, timeout, a 429 for rate limiting, and the server's 500, 502, 503 and 504. These are problems with the channel or the machine at the other end, and in a little while they may be gone.
• Not retryable. Order rejected, insufficient balance or buying power, invalid symbol, market closed, expired credentials, malformed parameters. Retrying a balance rejection five times does not give you balance: it brings you closer to the rate limit and fills the log with noise that hides the real failure.

import random
import time
import requests

REINTENTABLES = {408, 425, 429, 500, 502, 503, 504}

class ErrorPermanente(Exception):
    pass

def clasificar(resp):
    if resp.status_code < 400:
        return "ok"
    if resp.status_code in REINTENTABLES:
        return "reintentable"
    return "permanente"

Watch out for two subtleties. First: some brokers return 200 with a body that says "rejected". The HTTP code does not always match the business status, so classify by looking at the body too when necessary. Second: a 401 may look permanent and not be, because sometimes it is a clock skew rather than bad credentials. The list above is a sensible starting point, not dogma; tune it to what your actual broker returns, checking it against their test environment before you trust it.

Growing waits, with a bit of noise

Retrying immediately is hitting somebody who is already down. The wait doubles on each attempt (0.5 s, 1 s, 2 s, 4 s...) up to a ceiling, and on top of that randomness is added. Why the noise? Because if you have five processes, or five symbols in parallel, and they all fail at once because of the same outage, they would all retry at exactly the same instant and knock the service over again the moment it recovers. Randomness desynchronizes them.

def con_reintentos(hacer_peticion, intentos=5, base=0.5, tope=30.0):
    for intento in range(intentos):
        resp = None
        try:
            resp = hacer_peticion()
        except (requests.ConnectionError, requests.Timeout):
            pass
        else:
            tipo = clasificar(resp)
            if tipo == "ok":
                return resp
            if tipo == "permanente":
                raise ErrorPermanente(str(resp.status_code) + " " + resp.text[:200])

        espera = random.uniform(0, min(tope, base * (2 ** intento)))
        if resp is not None and resp.status_code == 429:
            espera = max(espera, float(resp.headers.get("Retry-After", 1)))
        time.sleep(espera)
    raise TimeoutError("agotados los reintentos")

And one stopping condition that does not appear in the code but does appear in a serious system: if the signal has expired, do not retry. An order meant for the open is no use twenty minutes later. Better to abandon it and record that than to execute late something the model never decided.

Two timeouts, not one

No network call without an explicit timeout. The default in nearly every library is "wait indefinitely", and a thread hung forever on a request is the quietest way for a bot to stop trading without any alarm going off.

In requests the parameter accepts two numbers: the first limits establishing the connection and the second the maximum gap between bytes received. That second subtlety matters: it is not a limit on the total time of the response. A server sending data very slowly can keep you hooked far longer than the number suggests. If you need an absolute ceiling, you also need a global deadline covering all the attempts.

resp = requests.post(url, headers=cabeceras, data=cuerpo, timeout=(3.05, 15))

You control the rate limit, not the broker

Waiting for the 429 to slow down is like driving by watching only for the crash. The right thing is never to go over, with a counter of your own on the client side. Remember that the limit is normally applied per API key or per IP address, so opening more connections or more processes does not multiply it: it divides it.

class Cubo:
    def __init__(self, por_segundo, capacidad):
        self.ritmo = por_segundo
        self.capacidad = capacidad
        self.fichas = capacidad
        self.ultimo = time.monotonic()

    def pedir(self):
        ahora = time.monotonic()
        self.fichas = min(self.capacidad,
                          self.fichas + (ahora - self.ultimo) * self.ritmo)
        self.ultimo = ahora
        if self.fichas < 1:
            time.sleep((1 - self.fichas) / self.ritmo)
            self.fichas = 1
        self.fichas -= 1

Also split the budget: data queries are expendable, sending and querying orders are not. If something has to wait, let the chart wait, never the order. In practice this means two buckets with different rates, or a single bucket with a priority queue in front, so that a price sweep of a hundred symbols does not starve the one request that moves money.

And there is an interaction with the previous module worth underlining: retries can only be applied to sending orders if those orders travel with their deterministic identifier. Without it, this module's retry loop becomes a very efficient position-duplicating machine. The two mechanisms are inseparable: the growing wait decides when to try again and idempotency decides whether trying again is safe.

Worked scenario: the 9:30 storm

A scenario built for the course. A bot follows 40 symbols and at the open asks for each one the last price, the day's candles and the positions: some 120 requests in the same second against a limit of 200 per minute. The broker starts returning 429. The retry loop, with no noise and without looking at the Retry-After header, relaunches all 120 at once after one second, and again, and again. In three minutes it has spent more than 1,000 requests, the broker applies a temporary block and, when the good signal finally arrives, the order cannot be sent. Nobody lost money on a bad prediction: it was lost on a badly written loop.