Can LLMs Really Understand Item Difficulty Levels? Implications for Automated Item Generation Using LLMs
Zero-shot GPT-4.1 (temperature=0) achieved the highest item difficulty prediction accuracy among LLMs with a quadratic weighted kappa (QWK) of 0.578 Encoder-only ConvBERT outperformed all LLMs with a QWK of 0.625, surpassing even the best feature-based supervised ML model All LLMs systematically struggled to label hard items, with GPT-5.4 tending to underestimate item difficulty levels Embedding dimension reduction revealed that item embeddings from different difficulty levels were mixed togethe
Analysis
TL;DR
- Zero-shot GPT-4.1 (temperature=0) achieved the highest item difficulty prediction accuracy among LLMs with a quadratic weighted kappa (QWK) of 0.578
- Encoder-only ConvBERT outperformed all LLMs with a QWK of 0.625, surpassing even the best feature-based supervised ML model
- All LLMs systematically struggled to label hard items, with GPT-5.4 tending to underestimate item difficulty levels
- Embedding dimension reduction revealed that item embeddings from different difficulty levels were mixed together, suggesting semantic information alone is insufficient for difficulty prediction
- The study cautions against relying on LLMs for generating assessment items with targeted difficulty levels, especially as model capabilities increase
Why It Matters
This research directly challenges the growing assumption that LLMs can reliably produce educational content at specified difficulty levels, which has significant implications for automated test generation and adaptive learning systems. For AI practitioners building assessment tools, the findings suggest that semantic understanding alone does not translate to pedagogical difficulty comprehension, and specialized models may still hold an advantage over general-purpose LLMs.
Technical Details
- Models evaluated: Multiple LLMs including GPT-4.1 and GPT-5.4, encoder-only ConvBERT, and feature-based supervised machine learning models
- Benchmark dataset: Items from a large-scale Reading and Writing test
- Evaluation metric: Quadratic weighted kappa (QWK) for item difficulty level prediction accuracy
- Prompting strategies: Various zero-shot prompting approaches and parameter settings (notably temperature=0) were investigated across model architectures
- Embedding analysis: Dimension reduction of item embeddings demonstrated overlap across difficulty levels, indicating insufficient discriminative power in semantic representations alone
Industry Insight
- Organizations investing in LLM-based automated item generation for educational assessments should temper expectations; semantic fluency does not guarantee pedagogical accuracy, and difficulty calibration remains a weak point
- Hybrid approaches combining LLM generation with specialized difficulty-prediction models (like ConvBERT) may offer a more reliable pipeline than relying on LLMs alone
- As LLM capabilities advance, the bias toward underestimating difficulty may worsen, creating a growing gap between model-generated and human-calibrated assessment quality that practitioners must actively monitor and correct
Disclaimer: The above content is generated by AI and is for reference only.