Why one backtest is not a conclusion

State the hypothesis before looking

A testable hypothesis states, in advance, which mechanism the return could come from and which observable evidence would reject it. A hypothesis added after the results were seen can no longer be tested by them.

This is why the research record exists before the result: the discussion follows the written question, not a curve.

What out-of-sample testing protects

Repeatedly adjusting rules on the same history will always find parameters that look good. Out-of-sample testing moves the evaluation onto data that did not participate in development, turning a question into an answerable one: does the rule still hold on data it has never seen?

Resampling helps assess estimation uncertainty, but it reshuffles the same history; it does not replace a time-ordered holdout.

Costs and invalidation rules

Returns in a frictionless backtest may not survive trading costs, slippage, and capacity constraints. Costs belong in the research phase, not the post-hoc correction.

Invalidation rules are part of the research too: which evidence lowers the weight, pauses validation, or retires the signal is written down before the conclusion. An invalidation rule written afterwards cannot trigger.

From one test to a record

One backtest is a single data point. Putting the hypothesis, data versions, the trial ledger, out-of-sample results, and invalidation rules into one reviewable record is what lets a conclusion be checked independently — and withdrawn.