BBOWP-Bench: Evaluating LLMs on Black-Box Optimization Word Problems
BBOWP-Bench is a novel benchmark evaluating LLMs on black-box optimization word problems, where the functional form is unavailable and only objective values are observable The benchmark requires systems to infer both a search space and an optimization algorithm from natural-language problem descriptions Current LLMs can select suitable optimization algorithms based on given evaluation budgets, demonstrating competence in algorithm selection LLMs struggle with search space design, particularly in
Analysis
TL;DR
- BBOWP-Bench is a novel benchmark evaluating LLMs on black-box optimization word problems, where the functional form is unavailable and only objective values are observable
- The benchmark requires systems to infer both a search space and an optimization algorithm from natural-language problem descriptions
- Current LLMs can select suitable optimization algorithms based on given evaluation budgets, demonstrating competence in algorithm selection
- LLMs struggle with search space design, particularly in identifying important variables and balancing their ranges when problem descriptions are less informative or search spaces are highly problem-specific
- The benchmark combines natural-language descriptions, executable evaluation environments, and human-designed baseline formulations for comprehensive evaluation
Why It Matters
This research addresses a critical gap in AI evaluation by introducing black-box optimization as a realistic problem setting where mathematical formulations are unavailable—a common scenario in industrial and scientific applications. For AI practitioners, it highlights that while LLMs have made progress in algorithm selection, significant challenges remain in search space design, guiding future research priorities for improving LLM-based optimization systems.
Technical Details
- BBOWP-Bench introduces Black-Box Optimization Word Problems (BBOWP), a novel setting where LLMs must infer both search space and optimization algorithm from natural-language descriptions without access to explicit mathematical expressions
- Each benchmark instance combines three components: a natural-language problem description, an executable evaluation environment, and a human-designed baseline formulation, enabling evaluation of both search-space design and algorithm selection
- The benchmark targets black-box optimization (BBO) problems where only objective values are observable and the functional form is unavailable, distinguishing it from prior benchmarks focused on explicitly formulatable objectives and constraints
- Evaluation covers algorithm selection based on evaluation budget constraints and search space design quality, including variable identification and range balancing
- The dataset and code are publicly available for further research
Industry Insight
- Organizations deploying LLMs for optimization tasks should invest in improving search space design capabilities, as this remains a significant weakness even as algorithm selection improves
- The benchmark provides a practical evaluation framework for measuring progress in LLM-based black-box optimization, which is directly applicable to industrial problems where objective functions are simulation-based or empirical
- Future LLM architectures should incorporate mechanisms for better variable importance identification and range calibration, especially for domain-specific optimization problems with limited descriptive information
Disclaimer: The above content is generated by AI and is for reference only.