Definition
The share of comparable ideas that held up out of sample after costs. Start from it before trusting your own backtest.
Example from the register
Across the register, 32% of 2,825 out-of-sample tests beat their benchmark after costs. That is the base rate to start from before trusting a single backtest. For calendar effects on US equities it is much lower: see the calendar entry.
Why it matters
Most strategy research reports the best result and leaves out how it was found and what failed. Base rates, placebo tests, trial counts and out-of-sample checks are the four tools that separate an idea with evidence behind it from a lucky backtest.
See it applied in the base rates, the dead ends and the methodology.
Questions
What is a base rate?
The share of comparable ideas that held up out of sample after costs. Start from it before trusting your own backtest.