Architecture-driven Shift: towards a lightweight selector for capturing the trends of logit shift

This is a research article proposing a novel metric for model selection in Continual Learning, grounded in theoretical analysis and extensive empirical validation.

The Core Mechanism: Decoupling to Compute

The fundamental innovation is the decomposition of logit shift into two independent factors: architecture dependency and data dependency. Logit shift measures how a model's predictions on old tasks change after learning new ones, which is crucial for stability. Directly computing it is expensive. The ADS metric bypasses this by:

1. Architecture Dependency: Captured through properties like spectral norms of weight matrices and network depth/width. This part is computed once from the model's structure.
2. Data Dependency: Approximated using only a small sample of data from the prior task. This minimizes the computational burden.

The ADS value is the product of these two dependencies. The key insight is that a model's architectural design inherently influences its susceptibility to forgetting, and this influence can be quantified and combined with a lightweight data measurement to predict shift.

Validating ADS: Three Mechanistic Components

The paper provides a theoretical foundation for why ADS correlates with logit shift, breaking it down into three interacting components:

• Spectral Norm Scaling: Gradients in wider layers are scaled by the spectral norm of the weight matrix. This property, tied to architecture, directly affects how much weights change during new task training.
• Optimization Path Length: The distance the model's parameters travel to learn the new task. This is influenced by both the task's data and the model's architectural structure.
• Asymptotic Task Conflict: Even in very wide networks, some conflict between tasks persists. ADS quantifies this inherent tension.

Together, these components explain why architectural choices like layer width and depth are not neutral; they actively shape the plasticity-stability trade-off, which ADS aims to measure.

Empirical Strength and Practical Utility

The claim is backed by robust experimentation:

• Scale and Diversity: Testing on over 175 architectures (including CNNs and Transformers) with varying depths and widths demonstrates the claim's generalizability.
• Strong Correlation: The reported monotonic correlation is strong (Spearman's $r_s=0.731$ is the weakest observed), meaning as ADS increases, logit shift predictably increases across different setups.
• Direct Application: ADS is positioned not just as a theoretical concept but as a practical tool. It serves as a lightweight proxy for Expected Calibration Error (ECE), a standard metric for model reliability. This allows practitioners to select pre-trained models for CL tasks by quickly estimating their likely calibration performance after learning, without undergoing full training and evaluation on every candidate model.