Glossary

Trial count: definition and a real example

By Justin JaegerPublished Updated

The number of variants tried before a result was found. The more attempts, the higher the bar a result has to clear.

Definition

The number of variants tried before a result was found. The more attempts, the higher the bar a result has to clear.

Example from the register

A result that appears after 5 attempts is stronger evidence than the same result after 500. The register counts every variant, including failures, so each verdict can be read against the number of tries behind it.

Why it matters

Most strategy research reports the best result and leaves out how it was found and what failed. Base rates, placebo tests, trial counts and out-of-sample checks are the four tools that separate an idea with evidence behind it from a lucky backtest.

See it applied in the base rates, the dead ends and the methodology.

Questions

What is a trial count?

The number of variants tried before a result was found. The more attempts, the higher the bar a result has to clear.