Allocating Recurrent Compute in Looped Language Models
Looped language models apply shared computation repeatedly, but existing systems loop entire layer stacks despite mixer and dense FFN performing different operations at different costs The paper introduces Iterative Transport Rank (ITR) and marginal ITR to measure cumulative and nonredundant cross-position influence trajectories across recurrent applications MixerLoop repeats each Gated DeltaNet mixer while applying its dense FFN only once, motivated by the insight that value comes from exposing
Analysis
TL;DR
- Looped language models apply shared computation repeatedly, but existing systems loop entire layer stacks despite mixer and dense FFN performing different operations at different costs
- The paper introduces Iterative Transport Rank (ITR) and marginal ITR to measure cumulative and nonredundant cross-position influence trajectories across recurrent applications
- MixerLoop repeats each Gated DeltaNet mixer while applying its dense FFN only once, motivated by the insight that value comes from exposing new observable influence directions
- At 15M parameters, MixerLoop surpasses FullLoop on aggregate CORE benchmark; at 110M parameters, it retains 41.5% of FullLoop's CORE improvement
- MixerLoop reduces recurrent-backbone projection FLOPs by 45.9% while preserving meaningful recurrent depth benefits without repeatedly executing the dense FFN
Why It Matters
This work directly addresses the efficiency-performance tradeoff in recurrent language models, which are increasingly relevant for long-context reasoning and deployment-constrained environments. By decoupling mixer recurrence from FFN repetition, it offers a principled path to cheaper recurrent architectures without fully sacrificing the reasoning gains that looped computation provides.
Technical Details
- MixerLoop Architecture: Repeats each Gated DeltaNet mixer iteratively while applying the dense feed-forward network (FFN) only once per layer, contrasting with FullLoop which repeats the entire layer stack
- Iterative Transport Rank (ITR): A theoretical framework where ITR describes the cumulative influence trajectory of recurrent applications, and marginal ITR quantifies the nonredundant cross-position influence contributed by successive applications
- Value Criterion for Recurrence: An application is deemed valuable when it exposes a new cross-position influence direction that remains observable at the task readout, providing a principled justification for selective looping
- Experimental Setup: Compared MixerLoop against no recurrence and full-block recurrence at 15M and 110M parameter scales under identical data, initialization, and architecture conditions
- Evaluation Method: Used a finite context-off intervention to test whether later mixer applications produce distinct, non-negligible, and beneficial changes at the final language model readout, with CORE as the aggregate benchmark
Industry Insight
- Selective recurrence—looping only the mixing component rather than full layers—could become a standard optimization for efficient recurrent language models, especially as the industry pushes toward longer context windows and on-device deployment
- The ITR framework provides a measurable, theory-grounded criterion for designing recurrent architectures, suggesting that future work should quantify influence redundancy before committing to full-stack looping
- The 45.9% FLOP reduction at retained performance demonstrates that significant compute savings are achievable in recurrent models without proportional accuracy loss, making mixed-recurrence designs attractive for cost-sensitive production environments
Disclaimer: The above content is generated by AI and is for reference only.