Learning to Grade Efficiently: A Bandit-Driven Prompt-Selection Framework for Low-Cost LLM Essay Scoring
Proposes a multi-armed bandit (MAB) framework for adaptive prompt selection in automated essay scoring, treating each prompt type as a bandit arm Achieves comparable scoring accuracy to exhaustive grid search while reducing LLM calls by 78.4% during prompt optimization Identifies multi-step grading with calibration examples as the highest-accuracy recipe among four tested configurations Introduces the first cost-reliability learning curves for essay scoring, tracking token usage, latency, and ag
Analysis
TL;DR
- Proposes a multi-armed bandit (MAB) framework for adaptive prompt selection in automated essay scoring, treating each prompt type as a bandit arm
- Achieves comparable scoring accuracy to exhaustive grid search while reducing LLM calls by 78.4% during prompt optimization
- Identifies multi-step grading with calibration examples as the highest-accuracy recipe among four tested configurations
- Introduces the first cost-reliability learning curves for essay scoring, tracking token usage, latency, and agreement metrics
- Transforms prompt selection from an offline hyperparameter optimization problem into an efficient online learning task
Why It Matters
This work addresses a critical bottleneck in deploying LLMs for educational assessment: the high operational cost of finding optimal prompting strategies. By reframing prompt selection as an online learning problem, it offers a scalable path for EdTech platforms to balance cost efficiency against assessment validity without exhaustive trial-and-error.
Technical Details
- Multi-Armed Bandit Controller: Each prompt type (grading recipe) is treated as an arm in a MAB framework, enabling adaptive selection during inference rather than relying on fixed prompt configurations.
- Four Grading Recipes Evaluated: Multi-step vs. single-step assessment, crossed with with vs. without calibration examples, yielding four distinct prompt configurations tested on IELTS Writing Task 2 essays.
- 78.4% Reduction in LLM Calls: The MAB framework achieves accuracy comparable to exhaustive grid search while dramatically reducing the number of LLM calls needed to identify the best grading approach.
- Cost-Reliability Learning Curves: First framework to jointly track token usage, latency, and agreement metrics, producing learning curves that visualize the cost-accuracy tradeoff for essay scoring systems.
- Dataset: Experiments conducted on IELTS Writing Task 2 essays, a standardized benchmark for automated essay scoring.
Industry Insight
- EdTech platforms can adopt bandit-driven prompt selection to significantly reduce inference costs while maintaining scoring quality, making large-scale automated assessment more economically viable.
- The cost-reliability learning curves provide a practical decision-making tool for practitioners to determine the minimum cost required to achieve target agreement levels with human raters.
- This online control paradigm generalizes beyond essay scoring and could be applied to any domain where prompt selection critically impacts both accuracy and operational cost.
Disclaimer: The above content is generated by AI and is for reference only.