Glossary

In-sample upper bound: definition and a real example

By Justin JaegerPublished Updated

A backtest over the full history including the tuning period. The most favourable view; if a rule loses here, it will not do better on new data.

The typical winning strategy made 1.4% a year and fell 31% on the way
Your worst drawdown isn't the worst: shuffle the trades, 1.5× deeper

Definition

A backtest over the full history including the tuning period. The most favourable view; if a rule loses here, it will not do better on new data.

Example from the register

Only 4.3% of 336,445 classic technical-rule backtests beat cash after retail costs over the full history. Because this includes the tuning period, it is the best case. See the backtest upper bounds.

Why it matters

Most strategy research reports the best result and leaves out how it was found and what failed. Base rates, placebo tests, trial counts and out-of-sample checks are the four tools that separate an idea with evidence behind it from a lucky backtest.

See it applied in the base rates, the dead ends and the methodology.

Questions

What is a in-sample upper bound?

A backtest over the full history including the tuning period. The most favourable view; if a rule loses here, it will not do better on new data.