Engineering Safety

Backtesting, Validation &
Failure Analysis

Backtesting is not proof. At best, it is a screening tool. At worst, it is a sophisticated way to lie to yourself.

Why Most Backtests Are Misleading

A backtest that looks clean, smooth, and stable should immediately raise suspicion. Most published results suffer from:

  • Implicit Lookahead Bias: Using future data in current decisions.
  • Survivorship Bias: Testing only on assets that exist today.
  • Overfitted Parameters: Optimizing variables until the chart looks perfect.
  • Static Assumptions: Applying fixed rules to dynamic, changing markets.

Backtesting Is a Filter, Not Validation

✓ What It Is

At AlgoBoy, backtesting serves one single purpose: To disqualify bad ideas early.

A method that passes a backtest has simply earned the right to be examined further.

× What It Is Not

  • It is NOT used to confirm beliefs.
  • It is NOT used for marketing performance.
  • It is NOT used to optimise parameters to exhaustion.

Forward-Only Logic & Integrity

All historical testing at AlgoBoy is built on strict forward-only logic. If a test cannot be reconstructed step-by-step as a time-ordered process, it is rejected.

# Pseudo-Code: Forward-Only Integrity Check def validate_step(current_time, decision_engine): # 1. HARD STOP: Ensure no future data leaks visible_data = market_data.filter(timestamp <= current_time) # 2. Parameters must be fixed *before* loop starts if decision_engine.parameters_changed(): raise IntegrityError("Mid-stream parameter shift detected") # 3. Generate Signal signal = decision_engine.evaluate(visible_data) return signal

Regime-Aware Testing

Markets do not behave consistently through time. Testing that ignores regimes is incomplete. We segment testing by:

Failure Analysis Is Mandatory

Failure is not an afterthought — it is a required output. Every method evaluated is accompanied by its known failure conditions.

Signal Crowding

When too many participants trade the same signal, impact cost destroys edge.

Liquidity Evaporation

When stops cannot be executed at model prices due to thin books.

Regime Misclassification

When the model assumes a mean-reverting market during a breakout.

Correlation Spikes

When "independent" signals all fail simultaneously during stress events.

Negative Results Are Published

AlgoBoy explicitly publishes failed methods, degraded approaches, and invalidated signals. Suppressing negative results creates false confidence and repeats the same mistakes.

"Backtesting does not prove an algorithm works.
It proves only that it has not yet failed under the conditions you tested."