Trading bot backtesting

What a backtest is actually for

A backtest replays historical prices through a strategy's rules and reports what would have happened. Its purpose is not to show you a profit — it is to rule out strategies that cannot work, which is a much lower bar and a much more useful one.

The trap is that a backtest will produce a good-looking number for almost any strategy if you try enough variations. Search hard enough over any sample and something fits it. So the discipline is not in running the test; it is in the things you do to stop yourself believing the result.

Charge the real costs, or the result is fiction

Every trade pays the spread. On short timeframes the spread is frequently larger than the average move the strategy is trying to capture, which means a strategy that looks profitable without costs can be reliably unprofitable with them.

Charge the spread as a round-trip on entry and exit. Charge commission where the contract has one. Charge overnight financing on leveraged positions held past the rollover. And use a spread appropriate to the instrument — synthetic indices carry a genuinely wide spread, major currency pairs a very narrow one, and using the wrong figure will either flatter or destroy the result for no real reason.

The four mistakes that produce fake profits

• Lookahead bias — using information the strategy could not have had at the time. The classic version is reading a swing high that is only confirmed several bars later, which makes the backtest a fortune teller.
• Overfitting — tuning parameters until the result looks good on the sample you tuned on. That number has no predictive value whatsoever, and it is the single most common reason a bot fails live.
• Too few trades — a profit factor over twenty trades tells you nothing. A few hundred trades starts to mean something; a few thousand means considerably more.
• Ignoring the losing runs — a strategy with a lovely total return and a drawdown you could not have sat through is not a strategy you can trade.

Hold data back, and never look at it twice

Split the history: fit the strategy on the first portion, then score it once on a portion it has never touched. The out-of-sample number is the only one worth anything, and it is usually much worse than the fitted one. The gap between them is a direct measure of how much noise the search memorised.

The rule that makes this work is that the held-back data is used once. If you look at the out-of-sample result, adjust the strategy and test again, that data is now part of the fitting process and its number means as little as the in-sample one. This is easy to do accidentally and it invalidates everything downstream.

The check almost nobody runs

A strategy is a claim that when it enters carries information. The way to test that claim directly is to strip the timing out and keep everything else — same instrument, same bars, same number of trades, same stop and target, same costs — but enter at random.

Run a few hundred of those random versions and you have a distribution of what luck alone produces. If the real strategy's result sits comfortably inside that distribution, its entry logic contributed nothing a coin flip would not have. Every indicator in it is decoration and the backtest profit was luck landing on the tuned parameters.

This is the most useful single test we know of, and it is the one that most often returns an unwelcome answer. That is exactly why it is worth running.

Check the neighbourhood, not just the winner

When a parameter search names a winner, look at the settings either side of it. A real edge sits on a plateau — if a 21-period average works because it genuinely tracks this market's swings, then 13 and 34 should also be reasonable.

If the winner makes money and its immediate neighbours lose it, you have found a spike rather than an edge: the one parameter value whose entries happened to line up with this particular sequence of prices. The next sequence will put the spike somewhere else.

We check this automatically before any configuration is adopted, because we have a concrete example of why it matters — a configuration that scored a 1.474 profit factor on data it had never seen, and ran at 0.518 live. The out-of-sample test passed it. The neighbourhood would not have.

After the backtest

• Walk-forward: repeat the fit-then-judge cycle across several rolling windows rather than one split, so consistency is tested rather than assumed.
• Paper trade on a demo account for weeks, on live prices, to catch execution problems a backtest cannot model.
• Then start small with real money, and compare live results against what the backtest predicted. A large divergence is information, not bad luck.

Common questions