TriRoute: Unified Learned Routing for Joint Adaptive Attention, Experts, and KV-Cache Allocation TriRoute：用于联合自适应注意力、专家和KV缓存分配的统一学习路由

TL;DR

• TriRoute introduces a unified learned routing controller that jointly optimizes attention resolution, Mixture-of-Experts (MoE) selection, and KV-cache bit-width allocation per token.
• The method addresses the strong coupling between these three decision axes, preventing the "routing-collapse cascade" observed in naive joint training through per-axis normalization and coupling-aware losses.
• TriRoute Pareto-dominates independent combinations of MoD, MoE, and KV-quantization at matched inference FLOPs and memory constraints, particularly improving robustness on rare entities, code, and arithmetic.
• The controller utilizes a heterogeneous relaxation training approach with Gumbel-Softmax and straight-through estimation, governed by a Lagrangian budget constraint to control average compute and memory costs.

Why It Matters

This research represents a significant shift from siloed efficiency optimizations to holistic, multi-axis conditional computation, offering a more efficient path to scaling language models without proportional increases in inference cost. For practitioners, it provides a concrete architectural pattern for integrating multiple sparsity and compression techniques into a single, trainable module that adapts dynamically to token complexity.

Technical Details

• Unified Controller: A lightweight network shared across all transformer layers emits a coordinated policy for each token, selecting attention mode (skip/local/full), a sparse set of FFN experts (including a null expert to mimic MoD), and KV-cache precision.
• Training Methodology: End-to-end training employs heterogeneous relaxation, combining Gumbel-Softmax with straight-through estimation for categorical choices and load-balanced top-k gating for experts, constrained by a Lagrangian multiplier to manage compute/memory budgets.
• Stability Mechanisms: The authors identified and solved a cross-axis routing-collapse cascade by implementing per-axis normalization and a specific coupling-aware balancing loss to ensure stable convergence across all three axes.
• Performance Benchmarks: Evaluated on decoder-only models ranging from 160M to 1.3B parameters, demonstrating superior performance over independent application of MoD, MoE, and KV-quantization at equivalent resource footprints.

Industry Insight

• Holistic Efficiency: Future model optimization efforts should move beyond single-axis sparsity (like just MoE) and consider joint routing strategies to maximize the utility of limited hardware resources.
• Interpretability as a Feature: The post-hoc analysis showing the controller naturally prioritizing sentence-initial positions and rare entities suggests that learned routing can align with linguistic importance, potentially reducing the need for heuristic-based pruning rules.
• Scalability Pathway: As models grow larger, the computational overhead of managing multiple independent sparsity mechanisms may become prohibitive; unified controllers like TriRoute offer a scalable alternative for maintaining performance while controlling latency and memory.

摘要

TriRoute 引入了一种统一的、可学习的路由控制器，共同优化每个 token 的注意力分辨率、混合专家（MoE）选择以及 KV 缓存位宽分配。该方法解决了这三个决策轴之间的强耦合问题，通过每轴归一化和感知耦合的损失函数，防止了在朴素联合训练中观察到的“路由崩溃级联”现象。在匹配推理 FLOPs 和内存约束的情况下，TriRoute 在帕累托前沿上优于 MoD、MoE 和 KV 量化独立组合的方案，特别是在罕见实体、代码和算术任务上提高了鲁棒性。该控制器利用带有 Gumbel-Softmax 和直通估计的异构松弛训练方法，并受拉格朗日预算约束控制，以管理平均计算和内存成本。

深度分析

简短总结

• TriRoute 引入了一种统一的、可学习的路由控制器，共同优化每个 token 的注意力分辨率、混合专家（MoE）选择以及 KV 缓存位宽分配。
• 该方法解决了这三个决策轴之间的强耦合问题，通过每轴归一化和感知耦合的损失函数，防止了在朴素联合训练中观察到的“路由崩溃级联”现象。
• 在匹配推理 FLOPs 和内存约束的情况下，TriRoute 在帕累托前沿上优于 MoD、MoE 和 KV 量化独立组合的方案，特别是在罕见实体、代码和算术任务上提高了鲁棒性。
• 该控制器利用带有 Gumbel-Softmax 和直通估计的异构松弛训练方法，并受拉格朗日预算约束控制，以管理平均计算和内存成本。

为什么重要

这项研究代表了从孤立的效率优化向整体、多轴条件计算的重大转变，为在不成比例增加推理成本的情况下扩展语言模型提供了更高效的途径。对于从业者而言，它提供了一种具体的架构模式，可将多种稀疏性和压缩技术整合到一个单一的、可训练的模块中，从而动态适应 token 的复杂性。

技术细节

• 统一控制器：一个轻量级网络在所有 Transformer 层之间共享，为每个 token 发出协调的策略，选择注意力模式（跳过/局部/完整）、一组稀疏的前馈网络专家（包括模拟 MoD 的空专家）以及 KV 缓存精度。
• 训练方法：端到端训练采用异构松弛方法，结合