JOR-Bench: Japanese Operations Research Benchmarks for Large Language Models
Introduction of JOR-Bench, a solver-independent benchmark comprising five Japanese translations of existing English Operations Research datasets. The benchmark covers 1,319 problems across linear, mixed-integer, non-linear, and combinatorial optimization, paired with expected numerical answers. Evaluation of seven LLMs reveals that strong multilingual models maintain high OR formulation capability in Japanese, with only a -0.3 percentage point accuracy drop compared to English. Error analysis id
Analysis
TL;DR
- Introduction of JOR-Bench, a solver-independent benchmark comprising five Japanese translations of existing English Operations Research datasets.
- The benchmark covers 1,319 problems across linear, mixed-integer, non-linear, and combinatorial optimization, paired with expected numerical answers.
- Evaluation of seven LLMs reveals that strong multilingual models maintain high OR formulation capability in Japanese, with only a -0.3 percentage point accuracy drop compared to English.
- Error analysis identifies specific cross-lingual pragmatic failures, such as models outputting decision-variable values instead of objective values when prompted in Japanese.
Why It Matters
This resource addresses a critical gap in evaluating non-English LLM capabilities for specialized technical domains like Operations Research. By providing a standardized, solver-independent benchmark, it enables researchers to accurately assess how well models handle complex mathematical reasoning in Japanese, facilitating better model selection and improvement for multilingual applications.
Technical Details
- Dataset Composition: JOR-Bench integrates five translated benchmarks: IndustryOR, MAMO Complex LP, NL4OPT, OptiBench, and OptMATH.
- Scope: Covers 1,319 distinct problems involving various optimization types, ensuring diverse testing of model capabilities.
- Evaluation Methodology: Uses the Python interface to OR-Tools to standardize execution, ensuring outputs are comparable and reproducible across different models and solvers.
- Model Testing: Assessed seven LLMs, including both multilingual general-purpose models and Japanese-specialized models, comparing performance across English and Japanese prompts.
Industry Insight
- Developers should implement rigorous pragmatic disambiguation checks for Japanese prompts, particularly ensuring models distinguish between decision variables and objective values in optimization tasks.
- The minimal performance drop (-0.3 pp) suggests that strong multilingual models are viable for Japanese OR tasks without extensive domain-specific fine-tuning, reducing deployment costs.
- Benchmarking efforts must prioritize solver-independent validation to ensure that performance metrics reflect the LLM's reasoning ability rather than its code generation or integration skills.
Disclaimer: The above content is generated by AI and is for reference only.