Assumptions this note depends on
- The note discusses methodology only. It contains no strategy, no parameters and no performance figures, invented or otherwise.
- Examples are generic. Any resemblance to a specific published result is unintended.
- A backtest that survives all four checks is still not evidence that a method will work in future.
1. Survivorship
A dataset of assets that exist today has quietly excluded every asset that stopped existing. In crypto the exclusion rate is high and non-random: tokens that failed are exactly the ones a naive strategy would have been holding. A universe defined at the end of the period cannot be used to test a rule applied at the start of it.
The check is procedural. Ask how the universe was constructed, and specifically whether an asset that was delisted or abandoned mid-period is still in the file. If the answer is that the dataset was downloaded from a current listing, the result is measuring the survivors.
2. Look-ahead
Look-ahead is the use of information that was not available at the moment the decision is dated. It is rarely deliberate and usually enters in one of four ways: a restated or revised figure, a corporate or protocol event applied to earlier bars, a timestamp in the wrong timezone, or a computation that centres a window rather than trailing it.
The check is to reconstruct one signal date by hand from data as it stood on that date. If you cannot, the pipeline does not know when it learned things.
3. Overfitting
Try enough parameter combinations and one of them will look excellent on any dataset, including a random one. The problem is not that a sweep was run; it is that only the winner is reported, so the reader cannot see how many comparisons produced it.
The check is to ask for the distribution, not the maximum: how many configurations were tested, and what the median one did. A method whose best configuration is dramatically better than its typical one is describing the dataset rather than the market.
4. Frictionless execution
A backtest that fills at the mid price, in unlimited size, with no fees and no failed transactions, is testing a market that does not exist. The gap widens exactly where a strategy looks best — high turnover, small edges, volatile periods — because those are the conditions in which cost is largest.
The check is to re-run with pessimistic assumptions: cross the spread, add venue and network fees, cap participation as a fraction of volume. A result that only survives optimistic execution is a result about execution, not about the signal.
The honest version of the conclusion
These four checks do not make a backtest predictive. They remove specific reasons a result would be wrong, which is a different and smaller claim. A method that passes them has earned the right to be considered, not the right to be trusted.
This is the last unit of the market structure course for a reason. The assessment asks you to critique a supplied backtest, and the supplied backtest fails at least two of these.
Sources
- 01The Probability of Backtest OverfittingBailey, Borwein, López de Prado & Zhu, Journal of Computational Finance
- 02Advances in Financial Machine LearningM. López de Prado, Wiley
- 03… and the Cross-Section of Expected ReturnsHarvey, Liu & Zhu, Review of Financial Studies
Cited by title and publisher rather than by link, so a moved page cannot turn a citation into a dead end.
Educational disclaimer
Ledgerworks Institute provides educational content and learning tools only. Nothing on this platform constitutes financial advice, an investment recommendation, or a guarantee of any outcome. Cryptocurrency and digital-asset markets carry substantial risk, including the total loss of capital. Conduct your own research and consult a qualified, licensed professional before making any financial decision.