Learning Compositional Meta-Routing for Agentic Workflows: An Executable Benchmark 学习组合式元路由用于智能体工作流：一个可执行基准

TL;DR

• Introduces an executable benchmark with 504 tasks across data analysis, frozen-corpus research, and document processing to evaluate agentic workflow routing
• Proposes a budget-aware meta-router using independent regularized logistic heads that compose heterogeneous operations (answer directly, decompose, retrieve, execute code, delegate, verify) from raw task text
• The learned policy achieves 100% success on held-out test data with 43% lower cost than static workflows, but drops to 75.9% on a lexical-shift challenge split
• Static routing outperforms learned routing on out-of-distribution lexical-shift tasks (93.5% vs 75.9%), revealing lexical generalization as the principal limitation
• Results establish a reproducible testbed and bounded proof of concept, not evidence of live-LLM performance

Why It Matters

This work addresses a critical gap in agentic AI: the routing problem of deciding which reasoning and execution operations to compose before producing an answer. As AI systems increasingly rely on multi-step workflows, understanding how to efficiently route tasks across heterogeneous operations becomes essential for building cost-effective, scalable agents. The benchmark also provides a standardized, machine-checkable evaluation framework that the community can build upon.

Technical Details

• Benchmark composition: 216 training, 72 development, 108 held-out test, and 108 locked lexical-shift challenge tasks spanning three domains: data analysis, frozen-corpus research, and document processing
• Meta-router architecture: Independent regularized logistic heads predict operation probabilities from word and character features, with temperature scaling performed on development data
• Greedy composition under budgets: Operations are greedily composed subject to route-cost and action-count constraints, enabling budget-aware decision-making
• Evaluation methodology: Outcomes are machine-checked after operations execute, providing objective success metrics; the lexical-shift challenge tests generalization to unseen vocabulary distributions
• Performance results: Learned policy achieves 100% success on held-out test (vs 93.5% static, 56.5% one-shot), with 43% cost reduction over static; on lexical-shift split, learned drops to 75.9% vs 93.5% static while remaining 49% cheaper

Industry Insight

• The significant performance gap between held-out test (100%) and lexical-shift challenge (75.9%) suggests that current routing approaches rely heavily on surface-level lexical patterns rather than semantic understanding—investing in vocabulary-invariant routing representations should be a priority
• Static routing policies demonstrate surprising robustness on out-of-distribution tasks, implying that simpler baselines may remain competitive in production settings where lexical diversity is high; practitioners should benchmark learned routers against static baselines before deployment
• The machine-checkable benchmark design sets a precedent for reproducible agentic evaluation; the community should adopt similar executable testbeds to move beyond self-reported metrics and enable fair comparison across routing approaches

摘要

介绍了一个包含504个任务的可执行基准测试，涵盖数据分析、冻结语料库研究和文档处理，用于评估智能体工作流路由
提出了一种预算感知的元路由器，使用独立的正则化逻辑回归头，从原始任务文本中组合异构操作（直接回答、分解、检索、执行代码、委托、验证）
学习到的策略在保留测试数据上达到100%成功率，成本比静态工作流低43%，但在词汇偏移挑战集上降至75.9%
在分布外词汇偏移任务上，静态路由优于学习路由（93.5% vs 75.9%），揭示了词汇泛化是主要局限性
结果建立了一个可复现的测试平台和有界的概念验证，而非真实LLM性能的证据

深度分析

一句话总结

• 介绍了一个包含504个任务的可执行基准测试，涵盖数据分析、冻结语料库研究和文档处理，用于评估智能体工作流路由
• 提出了一种预算感知的元路由器，使用独立的正则化逻辑回归头，从原始任务文本中组合异构操作（直接回答、分解、检索、执行代码、委托、验证）
• 学习到的策略在保留测试数据上达到100%成功率，成本比静态工作流低43%，但在词汇偏移挑战集上降至75.9%
• 在分布外词汇偏移任务上，静态路由优于学习路由（93.5% vs 75.9%），揭示了词汇泛化是主要局限性
• 结果建立了一个可复现的测试平台和有界的概念验证，而非真实LLM性能的证据

为何重要

这项工作解决了智能体AI中的一个关键空白：路由问题，即在生成答案之前决定组合哪些推理和执行操作。随着AI系统越来越多地依赖多步工作流，理解如何高效地在异构操作之间路由任务，对于构建具有成本效益且可扩展的智能体至关重要。该基准测试还提供了社区可以在此基础上构建的标准化、机器可检查的评估框架。

技术细节

• 基准测试构成：216个训练任务、72个开发任务、108个保留测试任务和108个锁定词汇偏移挑战任务，涵盖三个领域：数据分析、冻结语料库研究和文档处理
• 元路由器架构：独立的正则化逻辑回归头从单词和字符层面预测操作概率