3. Configuration, credentials and separate environments

Level 7: From the notebook to the process

There are three different things people throw into the same bag: the code, which says what gets done; the configuration, which says with which parameters; and the credentials, which say who you are. Mixing them is what turns a small mistake into a phone call to the bank.

Configuration is not edited, it is declared

If trading a different portfolio means opening a file and changing a number, the project has a problem: every different operation is a different version of the code, and none of them is recorded. Configuration must arrive from outside, through command-line arguments, environment variables or a configuration file the process reads at startup.

A split that works well in practice:

Arguments for what changes on each run: symbol, date, simulation mode.
Versioned configuration file for what is stable and not sensitive: universe, windows, exposure limits, tolerances. It goes in the repository and its history tells the story of how the strategy evolved.
Environment variables for everything that depends on the machine and for anything secret: keys, passwords, addresses of internal services.

There is a quick test for telling whether something is configuration or code: ask yourself whether two legitimate runs of the same system might want different values. The moving-average window, no: it is part of the strategy and changing it is changing the strategy. The capital allocated or the name of the environment, yes. And a warning: configuration is not a junk drawer. If you end up putting conditions inside the parameter file, what you are doing is writing code somewhere nobody will review or test it.

Failing at startup is better than failing halfway

Reading the configuration is the first thing the process does, and it must do it in such a way that a missing value kills it right there, not forty minutes later with half a portfolio already built.

# config.py
import os

def requerido(nombre):
    valor = os.environ.get(nombre)
    if not valor:
        raise RuntimeError('Falta la variable de entorno ' + nombre)
    return valor

# On the server, outside the repository:
#   export BROKER_KEY='tu_clave_aqui'
#   export BROKER_SECRET='tu_secreto_aqui'
#   export ENTORNO='pruebas'
BROKER_KEY = requerido('BROKER_KEY')
BROKER_SECRET = requerido('BROKER_SECRET')
ENTORNO = requerido('ENTORNO')          # no default value: on purpose

That last detail matters more than it looks. A default value for the environment is a trap waiting its turn: the day somebody forgets to export it, the process will not fail, it will do something. And what it does will depend on what whoever wrote the default decided months ago.

Credentials: never in the repository, and if it happens, revoke

A key written into the code is a published key. Credential files live outside the project or, if they live inside, the file is listed in the exclusions file from day one, alongside an example with no real values that is versioned.

And this has to be said plainly: if a key gets pushed by mistake, deleting the commit fixes nothing. The history has already been cloned, there are mirrors and caches, and public repositories are crawled by automated bots that find secrets within minutes. The only action that helps is going into the provider, revoking that key and issuing a new one. Cleaning up the history comes afterwards, and it is hygiene, not a remedy. In 2016 Uber suffered a breach that exposed the data of some 57 million users and drivers: the attackers got in using storage credentials that were sitting in a private company code repository. Private does not mean safe.

The most common accident: pointing at the wrong environment

Ask anybody who has been trading for a while and they will not tell you about a strategy that failed: they will tell you about the day they ran against the live account believing it was the test one. That is why the environment cannot be a comment or a habit, but an active barrier:

# guardas.py
CUENTAS = {
    'pruebas': 'https://sandbox.broker.example/v1',
    'real':    'https://api.broker.example/v1',
}

def resolver_destino(entorno, confirmar_real=False):
    if entorno not in CUENTAS:
        raise RuntimeError('Entorno desconocido: ' + str(entorno))
    if entorno == 'real' and not confirmar_real:
        raise RuntimeError('Operativa real sin confirmacion explicita: abortado')
    return CUENTAS[entorno]

And at the entry point, the confirmation is asked for in a way that makes typing it a conscious act:

p.add_argument('--confirmar-real', action='store_true',
               help='Obligatorio para operar contra la cuenta real')
destino = resolver_destino(ENTORNO, args.confirmar_real)

The key to the design is that the flag defaults to false and there is no way of switching it on without writing it. If tomorrow somebody turns it into a parameter defaulting to true to save typing, the safeguard disappears entirely even though the code is still there.

Real case: GitLab, January 31, 2017

A GitLab engineer was working through a database replication incident. To rebuild the replica a data directory had to be emptied. He ran the deletion, but in the wrong console: the production one. By the time he interrupted it, seconds later, a few gigabytes were left of the roughly three hundred that had been there.

What turned the scare into a serious incident was what came next: of the five backup and recovery mechanisms the company believed it had, none worked as expected. Recovery came from a snapshot taken hours earlier, and around six hours of project, issue and merge request data were lost. GitLab published the full postmortem and live-streamed the recovery, which is the admirable part of the story.

Translated into running a strategy: the person did not get the command wrong, they got the environment wrong. That is a system design failure, not an attention failure. A process that demands the environment be declared and an explicit flag to touch the live one turns that slip into an error message.