Evidence

Methodology: how the BacktestAtlas evidence is made

By Justin JaegerPublished Updated

BacktestAtlas develops ideas on data up to 2014, tests them on 2015–2026 with costs, counts every attempt and compares signals with random placebos.

Costs turned half our winning backtests into losers
Rules fell half as deep as holding, but stayed underwater 5× longer
99.7% of trading rules dodged the 2020 crash. Then they missed the rebound
Same forex rule, same years, second data file: 57% of winners flipped

Every verdict in BacktestAtlas is made with five safeguards: out-of-sample test, every try counted, luck baseline, costs included, frozen verdict.

1. Out-of-sample test

Every idea is developed on data up to 2014 and judged on 2015–2026. Classic technical rules are additionally reported as in-sample upper bounds.

2. Every try counted

Every variant is counted, including failures. A result is judged against how many attempts it took.

3. Luck baseline

Random placebo signals run on the same markets. A real signal family has to beat that range to count. For a single backtest, the expected best result of pure chance after the same number of tries is the bar.

4. Costs included

Spreads, slippage and financing are included in every out-of-sample test.

5. Frozen verdict

See the public chain. From November 2026 every verdict is timestamped and hashed when it is made, so the track record cannot be edited later.

What we do not claim

The test window has been viewed many times, so it is labelled and not called a clean holdout. Many tests use US-listed ETFs. The strategy library of 542,100 backtests is in-sample: it shows how rules did over the full history, not on data they were not built on, and is labelled that way in every answer. We publish aggregated statistics only, no prices or trades.

Model knowledge check

On 2026-11 we asked three Claude models 22 questions about register results and 4 about well-known papers. Mean error on the papers: haiku 4.5 pp, opus 0.0 pp, sonnet 6.8 pp; on the register: haiku 18.2 pp, opus 16.7 pp, sonnet 19.7 pp.

Before your next backtest

Let your AI check how ideas like yours did before it runs a single test. In our pilot, agents that checked first spent 1% of their backtests on known dead ends instead of 15%, and every backtest not wasted is time and tokens you keep. Cite this entry as ER-METHOD, as of 2026-11-01.

Questions

How is the evidence tested?

Development up to 2014, test 2015–2026, costs included, every attempt counted, signals compared with placebos.