Overfitting and curve fitting

You can explain how testing many rule variations creates backtests that look excellent but fail later, and name practical defenses against overfitting.

Proof, or a lucky fit?

A backtest with a smooth, steadily rising equity curve looks like proof that the rules work. Strategy sellers show such curves for exactly that reason.

Often the curve proves something else: that someone tried many variations of the rules until one happened to fit the past. It is one of the most common and most expensive mistakes in strategy development, and it is easy to make without noticing.

Best of 500 by chance

Overfitting, also called curve fitting, means tuning rules so closely to past data that they capture random noise instead of a repeatable pattern. The rules then fit history beautifully and fail on new data.

It happens through the search itself. Suppose you test a moving-average rule with 500 combinations of lengths, stops and targets. Even if none of them has any edge, some will look excellent on the test period by luck alone. Picking the best of 500 means picking the luckiest.

In this fictional example, the best of 500 combinations reaches a profit factor of 2.10 on the test data and 0.92 afterwards. A simple rule with two parameters reaches 1.30 and holds up at 1.20.

Research describes the problem precisely. Bailey, Borwein, López de Prado and Zhu (2014) showed that high backtest performance is easy to achieve after trying a relatively small number of configurations, and that a backtest that does not report how many configurations were tried cannot be judged. Harvey and Liu (2015) proposed ways to discount reported results for the number of tests behind them.

Test once on unseen data

Practical defenses:

Keep out-of-sample data. Split your history before you start:

Develop and tune only on the first part. Then lock the rules and test them once on the out-of-sample part. If you tune again after seeing that result, it is no longer out of sample.

Use few parameters. Every extra setting gives the rules more freedom to fit noise.
Prefer explainable logic. A rule that follows a clear idea, such as trend following, is less likely to be an accident than an unexplained combination of filters.
Check stability. If changing a moving average from 20 to 22 periods destroys the result, the result was fragile.
Count your tries. Write down how many variations you tested.

Common mistakes

Optimizing until the curve is smooth. Smoothness on old data is a warning, not a goal.
Peeking at the test data and tuning again. It quietly turns out-of-sample data into in-sample data.
Adding filters to remove past losers. Each filter fits the past a little more.
Believing a backtest without knowing the number of tries. Ask, every time.

The takeaway

Try enough variations and some will look brilliant by chance. Fewer parameters, untouched test data and simple rules protect against overfitting.

Check yourself

Quick quiz

4 questions. Instant feedback, nothing is saved.

1Tap the result that reveals the overfitting.

Tap the chart.

The best of 500 variations drops from 2.10 to 0.92 on new data. Its test result was largely luck from searching so many combinations.
2Why does testing many variations raise the risk of overfitting?

Bailey and coauthors showed that the more configurations are tried, the more likely the best one is overfit.
3What is out-of-sample data?

Only data the rules have never 'seen' can show whether they generalize.
4Which rule set is usually less prone to overfitting?

Every extra parameter gives the rules more freedom to fit noise in the past.

of 4 correct.

Sources

Back to Trading Styles & Strategy Development