Algometrics: Forecasting Under Algorithmic Feedback

Background

The paper addresses a fundamental problem in modern algorithmic finance and other predictive markets: the reflexivity of prediction. When predictive models are used to make trading decisions, their actions alter the very market dynamics they are trying to predict, creating a feedback loop. Traditional evaluation treats historical data as a passive, fixed environment, which fails to capture the model's impact when it is deployed as an active agent.

Key Points

The author proposes algometrics to formalize this problem and presents three central theoretical results:

1. Non-Identifiability of Deployment Risk: Even in the simplest linear model, it is impossible to determine a model's true deployment risk (its performance in the feedback loop) from historical data alone. Infinitely many different real-world environments can produce the exact same historical data, yet each would result in a different deployment risk for the same model. This means passive backtesting is fundamentally incapable of revealing real-world performance.

2. Inversion of Model Rankings under Crowding: A model's superior performance on historical data (lower "passive error") does not guarantee superior performance when deployed. If many market participants adopt similar models (crowding), the combined feedback can degrade the environment, potentially causing a historically top-ranked model to perform worse in practice than a historically inferior one. This directly challenges the validity of standard competitive model rankings.

3. Identifiability through Experimentation: The framework shows that deployment risk for short-horizon linear feedback can be estimated if the model's actions are randomized or use external instruments (like natural experiments). The author provides a finite-sample bound for this estimation, pointing toward a more rigorous, experimental method for evaluating algorithms in their native, feedback-driven environment.

Significance

This work has profound implications for quantitative finance, machine learning, and any field where predictive algorithms are deployed as active agents.

• Paradigm Shift in Benchmarking: It argues that time-series benchmarks must evolve. Instead of reporting just accuracy or loss, they must incorporate a measure of feedback sensitivity to assess how an algorithm's deployment would alter its own operating environment and subsequent performance.
• Critique of Common Practice: It formally exposes the inadequacy of standard historical backtesting for models that will influence the market, suggesting that overfitting to historical data may be even more dangerous than previously thought, as it ignores the dynamic environment the model itself creates.
• Foundation for Safer Deployment: By identifying the non-identifiability problem and offering an experimental path forward, it provides a foundation for developing more robust evaluation methodologies that account for strategic interactions and systemic risk, aiming to prevent destabilizing feedback loops in financial markets.