6. Errors, retries and reconnection in production

Level 13: Connecting to Real Brokers

A live trading system spends most of its life handling things that do not go as expected. The quality of a system is not measured by what it does when everything works, but by how it behaves when the network fails.

Response codes and what to do with each one

Not all errors are handled the same way, and this is the classification you need:

400 (malformed request) and 401/403 (authentication or permissions): the problem is in your code or in your credentials. Retrying is useless: it will fail in exactly the same way. You have to log the error, raise an alert and stop that operation.
429 (too many requests): you have exceeded the rate limit. Here you do retry, but waiting, with exponential backoff and jitter: each attempt waits twice as long as the previous one plus a random component. The random component prevents several processes that failed at the same time from hitting the server again in sync.
500, 502, 503 and network timeouts: the problem is on the other side or along the way. They are retryable, carefully.

import random

def request_with_retries(func, attempts=5):
    for i in range(attempts):
        try:
            r = func()
            if r.status_code == 429 or r.status_code >= 500:
                wait = (2 ** i) + random.random()
                time.sleep(wait)
                continue
            return r
        except requests.exceptions.RequestException:
            wait = (2 ** i) + random.random()
            time.sleep(wait)
    raise RuntimeError('Request failed after exhausting the retries')

The dangerous case: a timeout when sending an order

Here is the most serious trap of the whole level. If you send an order and the request times out, you do not know what happened. The order may never have reached the exchange. But it may equally have reached it, been accepted perfectly, and what was lost was only the response on the way back. The two scenarios are indistinguishable from your side.

Retrying blindly in that situation can double your position. The correct response has two parts:

1. Query before retrying. Faced with a timeout on an order submission, do not resend: ask. Query GET /orders and the recent fills to find out whether the order exists. Only if you confirm it did not arrive should you resend.

2. Use a client identifier. Most APIs let you attach a client_oid: a unique identifier you generate for that specific order. If you resend with the same identifier, the exchange recognizes it as the same order and does not duplicate it. That makes the submission an idempotent operation: repeating it does not change the result.

The general rule: read operations are safe to retry; write operations are only safe if they are idempotent.

Reconnecting the data channel

A WebSocket connection drops. Always. A serious client implements:

Automatic reconnection with exponential backoff, not in a tight loop.
Resubscription to all channels after reconnecting: the new connection does not remember your previous subscriptions.
State resynchronization: things happened while you were disconnected. You have to request a full snapshot of the book and of your positions over REST, and discard the state you held in memory.
Heartbeat: if no message arrives for N seconds, the connection is considered dead even if the socket is still technically open. This case, the zombie connection, is more frequent and more treacherous than a clean disconnection.

And the question that closes the module: if you lose the connection while holding an open position, what should the system do? The reasonable answer is stop generating new signals, try to reconnect, and on recovering the connection resynchronize the real state before making any decision. Trading blind on stale data is worse than not trading.

6. Errors, retries and reconnection in production

The rest of Quant500