Hypothesis graveyard
What we already tried, and what didn’t work
Nineteen ideas discarded across eight families, each with the data that buried it. It is published for two reasons: if you propose something on this list, the answer is already written and has numbers attached; and it is the proof that when we say something does not work, we have measured it.
Almost all of it came out of auditing our own bot, across more than fifty formally recorded experiments. Before holding anyone else to this bar, we held ourselves to it.
Market-regime detection
The idea that has been attempted most often and failed most often: four independent attempts, with techniques of increasing complexity, and the same result every time.
- ADX, volatility ratio and volatility level as a classifier: it does not separate winning years from losing ones. 2024 had 11 % strong-trend days and still lost money.
- Hamilton MSAR (Markov switching, three configurations): entropy ≈ 0.99 on silver. The model finds no structure at all; it is indistinguishable from noise.
- Regime filters on the entry: rejected as a family, because the simulation had already established that the edge lives in the trend and in exit management, not in the entry signal.
If you bring us this: If your bot uses a simple regime detector as an entry filter, ask for the evidence that it separates winning periods from losing ones with data. The odds that it does not are high: measured four times independently on the same asset.
External confirmation filters (macro, cross-asset)
Confirming a signal with another asset sounds prudent, and in practice it blocks profitable trades.
- Lead-lag between the dollar and gold on daily data, 2022–2026: the relationship turned out to be 100 % contemporaneous, so there is nothing to anticipate.
- A 200-period moving-average trend filter on the 4-hour chart: it blocks trades that were making money.
If you bring us this: A filter that is only tested against the final result looks harmless. You have to measure which trades it would have prevented, not just whether the total goes up.
Machine learning on top of an existing signal
A meta-model that decides whether to act on the signal. It closed the whole family of discriminant filters.
- The classifier’s area under the curve stayed inside the noise: the features of winning and losing trades are indistinguishable.
- If there is no signal to separate, no model is going to find one, however sophisticated. The problem was not the model.
If you bring us this: Stacking layers on top of a signal that does not discriminate improves nothing, and it does multiply the number of times you have searched — which is exactly what the overfitting analysis penalises.
“Fixing the wiring” of a quantity that was already saturated
A parameter can look refuted without actually being connected to anything.
- 144 combinations were swept and produced only 9 distinct result series: the swept parameters were saturated upstream and moved nothing.
- Correcting a calculation that was already pinned to its ceiling changes nothing. Before refuting a parameter, check that it does anything at all.
If you bring us this: If your optimisation says 144 passes but there are only 9 distinct results, you did not try 144 things. Counting them as 144 understates your statistical luck.
Parametric fat-tail models
Two standard ways of estimating extreme losses, rejected by measurement.
- The Cornish-Fisher expansion understated risk by 52 % on real data, and outside its domain of validity it stops being monotonic.
- Extreme value theory needs on the order of 50 exceedances to fit. The real track records that reach us almost never have them.
If you bring us this: Something useful did come out of this: a confidence interval that declares its measured coverage rather than the nominal one. A narrow interval does not mean a reliable one.
Why this is not a catalogue of things to try
The distinction matters more than it looks. A list of “try this as well” raises the number of times you have searched, and every additional search on the same data spends your statistical luck. It would make the auditor part of your search process: the same conflict of interest that separated auditing from accounting consultancy after Enron.
This is the opposite. These are ideas already buried, so that you do not spend your attempts on them.