Multiple testing and the false discovery trap in factor research

Data mining and significance inflation

When testing 100 independent random hypotheses that possess zero predictive power, standard 5% significance thresholds mathematically guarantee that roughly five will appear statistically significant by pure chance.

Quantitative research routinely scans vast permutations of indicators and parameters. Without multiple testing corrections, the vast majority of discovered high-Sharpe 'alphas' are merely memorized noise slices from history.

Controlling false discovery and reality checks

Ziju Quant incorporates False Discovery Rate (FDR) controls and White's Reality Check into factor screening, penalizing discovery thresholds as the search universe expands.

A hypothesis earns consideration for out-of-sample validation only after clearing critical thresholds adjusted for the full history of unchosen trials.

Economic rationale before statistical scores

Rigorous statistical adjustments cannot substitute for sound economic rationale. Excess returns fundamentally stem from bearing unhedged structural risks or capturing identifiable behavioral biases and liquidity frictions.

Empirical patterns lacking structural causality remain fragile and decay rapidly as market participant behavior evolves or macro regimes shift, even if they clear statistical hurdles.

Maintaining the full ledger of null trials

The most critical engineering safeguard against multiple testing bias is maintaining an immutable ledger of every failed trial, parameter adjustment, and discarded exploration. The trial ledger is the only valid anchor for assessing true statistical significance.