Relay-Bench: Evaluating LLMs on Multi-Domain Reasoning Chains
Relay-Bench is introduced as an unsaturated, text-only benchmark designed to evaluate LLMs on multi-domain reasoning chains within a single prompt. The benchmark tests composite problems combining distinct domains such as coding, math, visual reasoning, and web search, with no restrictions on tool usage. Leading model GPT-5.5 (xHigh) achieves a score of only 43.3%, indicating significant difficulty in handling complex, cross-domain tasks. Problems consist of two to thirteen subproblems and inclu
Analysis
TL;DR
- Relay-Bench is introduced as an unsaturated, text-only benchmark designed to evaluate LLMs on multi-domain reasoning chains within a single prompt.
- The benchmark tests composite problems combining distinct domains such as coding, math, visual reasoning, and web search, with no restrictions on tool usage.
- Leading model GPT-5.5 (xHigh) achieves a score of only 43.3%, indicating significant difficulty in handling complex, cross-domain tasks.
- Problems consist of two to thirteen subproblems and include deliberate context bloat and prompt encoding to increase complexity.
Why It Matters
This benchmark highlights the current limitations of state-of-the-art LLMs in performing sustained, multi-step reasoning across diverse domains without multimodal inputs. For researchers and practitioners, it provides a rigorous standard for evaluating whether models can effectively integrate tools like code execution and web search in complex, real-world scenarios.
Technical Details
- Benchmark Structure: Relay-Bench uses composite problems formed by stringing together single-domain subproblems, requiring reasoning across multiple domains simultaneously.
- Domains Tested: Includes visual reasoning, coding, math, information extraction (web search focus), problem-solving, general knowledge, and data analysis.
- Complexity Factors: Challenges are augmented with prompt encoding and deliberate context bloat to test robustness against noise and complexity.
- Tool Usage: Models are explicitly encouraged to use code-execution, web searches, and other available tools, with no restrictions imposed outside the model harness.
- Performance: The top-performing model, GPT-5.5 (xHigh), scored 43.3%, demonstrating that even advanced models struggle with these multi-domain chains.
Industry Insight
- Focus on Tool Integration: The low scores suggest that future model development must prioritize better integration and orchestration of external tools (code executors, search engines) within a unified reasoning framework.
- Context Management: The inclusion of context bloat indicates a need for improved attention mechanisms or retrieval-augmented generation techniques to handle noisy or lengthy inputs effectively.
- Evaluation Standards: Benchmarking should move beyond single-domain accuracy to assess multi-step, cross-domain reasoning capabilities to better reflect real-world application demands.
Disclaimer: The above content is generated by AI and is for reference only.