Overfitting, and how to test around it

Imagine fitting a line through a scatter of dots. A straight line misses most of them but captures the trend. A wiggly enough curve can pass through every single dot perfectly — and will be useless for predicting the next one, because it has memorised the dots rather than learned the shape.

That’s overfitting, and a trading rule can do exactly the same thing.

What it looks like in practice

You start with “buy when the 50-day average crosses the 200-day.” It makes 8% a year in the backtest. Not bad. So you tune it:

• 50/200 → 47/210, which tested better
• skip trades in December, which was a bad month
• only trade when volume is above average
• add a filter that would have avoided the 2020 crash

Now it makes 31% a year. It also has four extra rules, every one of which was chosen because of something that already happened. You haven’t improved the strategy; you’ve written a compressed description of the last ten years.

The rule of thumb

Every parameter you add needs to earn its place, and most don’t. A rule with two settings that makes 8% is very often better in real trading than a rule with nine settings that makes 31% on paper. Fewer moving parts means fewer opportunities to have accidentally fitted noise.

Walk-forward testing

The standard defence is to not let the strategy see all the data.

Tune it on 2015–2019 only. Then, without touching it again, run it on 2020–2023. That second stretch is data the rule has never met, so the result is a fair one. If performance collapses — and it often does — you’ve learned something genuinely valuable for the price of nothing.

Then slide the window forward and repeat. That’s walk-forward: tune, test on unseen data, move on.

The final honest test

Even walk-forward can be gamed, because if you keep adjusting after seeing the “unseen” results, it stops being unseen. The only test nobody can cheat is forward time — which is why every strategy should run on paper for a while before it touches real money.