Loss-Parameterized Fisher Width Along Learning Trajectories
Fisher width, measuring the Gaussian width of probe sets under local Fisher geometry deformation, can be parametrized by training loss along learning trajectories, but this relationship is branchwise rather than universal In a population Gaussian-teacher logistic model, the teacher-aligned state is extremal below loss level log(2), minimizing parameter norm while maximizing both Fisher trace and Euclidean-ball Fisher width Population gradient flow asymptotically selects the teacher-aligned branc
Analysis
TL;DR
- Fisher width, measuring the Gaussian width of probe sets under local Fisher geometry deformation, can be parametrized by training loss along learning trajectories, but this relationship is branchwise rather than universal
- In a population Gaussian-teacher logistic model, the teacher-aligned state is extremal below loss level log(2), minimizing parameter norm while maximizing both Fisher trace and Euclidean-ball Fisher width
- Population gradient flow asymptotically selects the teacher-aligned branch with explicit convergence rates for aligned and orthogonal coordinates, yielding the limit w_F(B_2^d;θ(t))/√L(θ(t)) → (√6/π)E[χ_{d-1}]
- In nonlinear MLPs with diagonal Fisher approximation, GD and SGD remain close at matched loss, while Adam follows a substantially displaced branch, suggesting optimizer-dependent loss-width relationships
- The paper establishes an exact trace-shape factorization and deterministic stability bound for fixed compact probes, supporting controlled full-Fisher experimental validation
Why It Matters
This work provides theoretical grounding for understanding how model complexity (as measured by Fisher width) evolves during training relative to loss, which is critical for generalization analysis and model selection. For practitioners, the finding that different optimizers follow distinct loss-width branches implies that loss curves alone cannot fully characterize learning dynamics, necessitating geometry-aware diagnostics.
Technical Details
- Derives an exact trace-shape factorization of Fisher width and a deterministic stability bound for fixed compact probes under Fisher geometry deformation
- Analyzes a population Gaussian-teacher logistic model where the teacher-aligned state is proven extremal on every loss level below log(2), with minimal parameter norm and maximized Fisher trace and Euclidean-ball Fisher width
- Proves that population gradient flow asymptotically selects the teacher-aligned branch, establishing explicit convergence rates for both aligned and orthogonal coordinates
- Demonstrates the universal limit relation w_F(B_2^d;θ(t))/√L(θ(t)) → (√6/π)E[χ_{d-1}] for d≥2, connecting Fisher width to chi-distribution expectations
- Validates predictions through controlled full-Fisher experiments on nonlinear MLPs with diagonal model-Fisher approximation, showing GD/SGD proximity at matched loss versus Adam's displaced trajectory
Industry Insight
- Loss-based monitoring alone is insufficient for characterizing learning dynamics; practitioners should incorporate Fisher geometry measurements to distinguish between optimizers that appear equivalent at matched loss but follow fundamentally different complexity trajectories
- The branchwise nature of loss-Fisher width parametrization suggests that hyperparameter tuning should account for optimizer-specific geometry, as Adam's displaced branch may indicate different generalization behavior despite similar loss curves
- The theoretical limit connecting Fisher width to chi-distribution expectations offers a diagnostic benchmark for validating whether training has reached the teacher-aligned extremal regime, potentially informing early stopping criteria
Disclaimer: The above content is generated by AI and is for reference only.