6. Reinforcement learning: agent, environment and reward

Level 10: Machine Learning Applied to Trading

Reinforcement learning is influenced by behavioral psychology and its reward system. We can think of it as playing with a dog. We have a dog (which in the terminology is called the agent) and we have a park (the environment) where it can do whatever it likes. Every time the dog brings you a stick, you reward it with food. After a while, the dog starts developing better strategies for finding other sticks nearby. That is, it learns from the reward system. That process inside the dog's head is what we call reinforcement learning. We have an agent, an environment and a reward.

The difference from supervised learning lies in the output data. In reinforcement learning, while the dog is wandering around, it does not know whether it will get a reward or not. It may never get one. But when it finds one, it has learned how to win. So the output data can arrive much later. In supervised learning, by contrast, we know the output in advance and make predictions while looking at it.

That delayed credit assignment is what makes reinforcement different and hard. If the dog ran twelve laps, sniffed three bushes and then found the stick, which of those actions deserves the credit? The algorithm has to spread a scarce, delayed reward across a long chain of decisions.

In trading the analogy is direct, and that is why reinforcement generates so much interest:

• Agent: the trading system that decides.
• Environment: the market, which answers with new prices and, if you trade size, with the impact of your own orders too.
• Actions: buy, sell, hold, or more finely, what position size to take.
• Reward: the result of the trade, ideally risk-adjusted and net of commissions and slippage.
• State: the information available at the moment of deciding (prices, indicators, current position, exposure).

There are three practical consequences worth internalizing before you touch a single line of code.

First: the reward is delayed and noisy. An excellent purchase can be down for three days and up six weeks later. The agent has to learn that a good action does not always bring an immediate reward; that is exactly the dog's problem.

Second: the exploration versus exploitation dilemma. If the agent only repeats what has already worked, it will never discover a better strategy. If it explores too much, it burns money experimenting. In a park, exploring is free. In a market, exploring costs commissions and real losses.

Third, and most important: the market is not a park. The dog's environment is stable; the market changes regime, and other participants react to what you do. An agent trained on a historical record can memorize the specific path of those years instead of learning a robust policy. Overfitting, in reinforcement learning, is especially insidious, because the agent also chooses which data it sees when it chooses its actions.

In short: supervised is learning from a teacher who already has the answers; unsupervised is looking for structure without answers; reinforcement is learning by acting, on the back of rewards that arrive late, badly and sometimes never.