Pointwise Metrics Mislead: An Evaluation Protocol for Multimodal Inverse Problems

Background

In scientific reconstruction tasks, evaluation often hinges on pointwise metrics like RMSE (Root Mean Squared Error) and MAE (Mean Absolute Error). These methods assume lower error signifies superior performance. However, this assumption breaks down in complex inverse problems where posteriors exhibit multimodal distributions.

Key Points

The paper challenges the adequacy of pointwise metrics by demonstrating that they inherently produce biased estimates due to the nature of these inverse problems. Specifically:

• Pointwise Metrics Bias: The law of total variance shows that point estimators trained with MSE or MAE aim to minimize error at individual points, resulting in a narrower marginal spectrum compared to the true posterior distribution.

  • Compressed Features: This compression affects crucial spectral features such as tails, modes, and shapes, which are vital for downstream scientific measurements.

• Evaluation Protocol: To address these issues, the authors propose a three-step evaluation protocol:

  1. Per-event Distributional Accuracy (CRPS): Measures the distribution of predictions against the true posterior using Continuous Ranked Probability Score.
  2. Population-Level Marginal Accuracy (Spectrum-Fidelity Diagnostic): Evaluates the overall marginal accuracy by assessing how well the predicted spectrum matches the true one.
  3. Uncertainty Trustworthiness (Calibration): Ensures that predictions are calibrated, meaning the reported uncertainty levels match the actual spread of the data.

• Experiments: The protocol was tested on both a synthetic benchmark with an analytic posterior and a realistic many-to-one inverse problem from particle physics.

  • Model Rankings: Inverse ranking between pointwise and distributional metrics revealed that models previously ranked highly based on pointwise metrics performed poorly when evaluated using the proposed protocol.

  • Calibration Importance: Calibration further distinguished architectures that were indistinguishable under CRPS, highlighting its importance.

Significance

The findings have significant implications for scientific reconstruction tasks:

• Bias in Scientific Results: The bias introduced by pointwise metrics can lead to incorrect conclusions about model performance and may mislead the selection of models used in real-world applications.

• Comprehensive Evaluation: By introducing a more comprehensive evaluation protocol, researchers can ensure that models are not only accurate at individual points but also provide reliable distributions and calibrated uncertainties.

• Practical Implications: The proposed framework can be applied across various scientific domains where inverse problems with multimodal posteriors are common, such as medical imaging, environmental monitoring, and particle physics.