Predicting When Random Low-Dimensional Reparameterizations Train Neural Networks
Neural networks can be trained via random low-dimensional reparameterization, where a small latent vector is mapped into full parameter updates through a frozen random map The authors derive an orientation-resolved quadratic master formula predicting random-slice residual from curvature spectrum and displacement profile, generalizing prior Gaussian-width bounds Random Mapping Networks (RaMaN) use structured Hadamard or seed-regenerated Gaussian maps, reducing storage from O(dP) to O(d) and optim
Analysis
TL;DR
- Neural networks can be trained via random low-dimensional reparameterization, where a small latent vector is mapped into full parameter updates through a frozen random map
- The authors derive an orientation-resolved quadratic master formula predicting random-slice residual from curvature spectrum and displacement profile, generalizing prior Gaussian-width bounds
- Random Mapping Networks (RaMaN) use structured Hadamard or seed-regenerated Gaussian maps, reducing storage from O(dP) to O(d) and optimizer-state memory from O(P) to O(d)
- Matrix-free curvature approximations and sweep-free dimension selection enable practical deployment without full curvature computation
- Experiments on quadratic landscapes and end-to-end image/language models confirm sharp, protocol-dependent training transitions predicted by the theory
Why It Matters
This work bridges a critical gap between theoretical understanding and practical efficiency in neural network training, offering a principled way to determine latent dimension requirements without expensive trial-and-error. For AI practitioners working on parameter-efficient fine-tuning or memory-constrained deployment, RaMaN provides a theoretically grounded alternative to full fine-tuning with dramatically reduced memory overhead.
Technical Details
- Theoretical framework: The accessibility transition is reformulated in conic geometry, centered at the statistical dimension of the polar cone for compact convex targets. The orientation-resolved quadratic master formula predicts residual loss from both the curvature spectrum and the reference-to-solution displacement profile, yielding a self-consistent isotropic-orientation predictor that recovers the Gaussian-width quadratic bound in its radius-only specialization.
- RaMaN architecture: Structured Hadamard and seed-regenerated Gaussian random maps instantiate the predicted latent dimension, avoiding dense O(dP) storage. Optimizer-state memory scales as O(d) rather than O(P), where d is latent dimension and P is parameter count.
- Practical innovations: Matrix-free curvature approximations eliminate the need for explicit Hessian computation, and sweep-free dimension selection determines the required latent size without iterative searching.
- Empirical validation: Controlled quadratic and neural-curvature experiments show the orientation-resolved predictor closely tracks measured transition locations, outperforming orientation-agnostic approximations when displacement direction is significant. End-to-end experiments demonstrate sharp training transitions across image and language models.
Industry Insight
- Parameter-efficient fine-tuning methods could adopt RaMaN-style structured random maps to achieve comparable performance to full fine-tuning at a fraction of the memory cost, particularly valuable for edge deployment and large-scale language model adaptation.
- The orientation-resolved predictor provides a practical diagnostic tool: practitioners can estimate the minimum latent dimension needed before committing to training, avoiding wasted computation on underspecified reparameterization schemes.
- As models continue to scale, the O(d) memory reduction from RaMaN becomes increasingly impactful, potentially enabling fine-tuning of billion-parameter models on hardware that currently only supports inference.
Disclaimer: The above content is generated by AI and is for reference only.