What a backtest is, and why yours looks better than it is

A backtest takes your rule, replays the last few years of prices through it, and reports what would have happened. Buy here, sell there, up 34% over three years, worst dip along the way 12%.

It’s genuinely useful. If a strategy loses money over ten years of history, you’ve learned something cheaply. That’s worth having.

The trouble is what a good backtest result means, which is much less than it appears.

The uncomfortable bit

You already know what happened in that history. You lived through some of it. And so, without ever intending to cheat, you’ll adjust the rule until the result looks good — a 50-day average instead of a 40, skip the trades on Mondays, don’t trade below $5. Each change seems reasonable. Each one is also, quietly, a fact about the past being copied into the rule.

The result is a strategy that describes history beautifully and knows nothing about tomorrow. There’s a name for this — overfitting — and it is the single most common way people lose money with automated trading.

A rough sense of scale

If you try 100 variations of a rule against the same history and pick the best one, you should expect that best one to look good even if none of them has any real edge at all. Pure chance produces a winner out of 100 coin-flippers too. The backtest can’t tell you which kind you’ve got, because it only ever saw the winner.

So what is it good for

Three honest uses:

• Elimination. A rule that fails badly across history is unlikely to be worth pursuing.
• Expectation-setting. It shows you roughly what the ride feels like — how long the flat stretches are, how deep the dips go.
• Bug-finding. Weird results usually mean the rule doesn’t do what you thought it did.

What it can’t do is tell you a strategy will work. Only time can do that, and paper trading is how you buy that time without paying for it.