Scams, Myths & Reality Check · Intermediate · 5 min

Checking backtest screenshots critically

You can explain how backtests are overfitted, why a single impressive screenshot proves little and which questions reveal the real quality.

Proof, or best of many?

A post shows a backtest: a smooth equity curve rising for ten years, a high win rate, a small drawdown. Underneath: "Proven strategy, get it now."

A backtest can be useful. But a single screenshot of one hides the most important information: what had to happen to produce it. This lesson explains how impressive backtests are made, and which questions show whether one means anything.

Fitted to the past

A backtest applies a set of trading rules to historical prices to see how they would have performed. It is a normal tool for developing strategies, and the strategy chapter covers it in detail.

The problem is overfitting: adjusting rules until they match the past very closely, including its random noise. Every change, a different indicator setting, another filter, a new stop distance, can make the past results look better. After enough changes, the rules describe the history of one data set rather than anything that will repeat.

Mathematicians David Bailey, Jonathan Borwein, Marcos López de Prado and Qiji Jim Zhu examined this in 2014. As the American Mathematical Society's summary of their paper explains, for a model based on five years of data, looking at as few as 45 configurations is enough for at least one to stand out with good backtest results but do badly on data it was not built on. The authors also note that when a backtest does not report how many configurations were tried, it is impossible to judge how likely it is to be overfitted.

The best defense is an out-of-sample test: running the finished rules on data that was not used to build them, such as a later period. A strategy that only works on the data it was fitted to has not shown anything.

Other things screenshots often leave out:

Costs. Spreads, commissions and slippage, especially for frequent trading.
The full period. A start and end date chosen to look best.
Position size. A high return may come from very large risk per trade.

One of 45

A fictional seller builds a strategy based on two moving averages and tests 45 combinations of settings on five years of data.

Most combinations do poorly or roughly break even after costs.
One combination shows a smooth curve with a gain of 180% and a small drawdown. That one becomes the screenshot.
Run on the following year, which was not part of the test, the same settings end the year roughly flat.

Nothing in the screenshot was false. It was the best result of 45 tries on data the rules had been shaped to fit. A buyer who sees only the screenshot cannot know that the other 44 exist.

Common mistakes

Taking a smooth curve as proof. Smoothness is easy to produce by fitting.
Not asking how many versions were tried. Without that number, overfitting cannot be judged.
Ignoring costs. Many backtests look far worse once costs are included.
Confusing backtest and live results. Ask for results on data the rules have never seen.

The takeaway

A backtest shows how rules would have done in the past. Try enough variations and one will look great by chance. Ask how many were tried.

Check yourself

Quick quiz

4 questions. Instant feedback, nothing is saved.

1Tap the step that turns luck into a 'proven' strategy.

Tap the chart.

Picking the best of many tries selects the one that fit the past data best, often by chance. That is the core of overfitting.
2According to Bailey and colleagues, how few configurations can mislead you on five years of data?

With five years of data, among 45 tried configurations at least one can look good in the backtest and still do badly on new data.
3What is an out-of-sample test?

Data the strategy has never seen shows whether it captured something real or only fitted the past.
4Which question reveals most about a backtest screenshot?

The authors note that without the number of tried configurations, the risk of overfitting cannot be judged.

of 4 correct.

Sources

Back to Scams, Myths & Reality Check