ArguLens: An Open-Source System for Automated Essay Scoring and Label-Aware Feedback Generation
ArguLens is an open-source, locally deployable automated essay scoring (AES) system released under Apache 2.0 that decomposes scoring into three decoupled components for interpretability and privacy The system combines a discourse-move classifier (Qwen2.5-7B-Instruct fine-tuned with LoRA on PERSUADE 2.0), a grade-independent LightGBM scorer using 31 linguistic and discourse features, and a label-aware feedback generator (Qwen2.5-14B-Instruct via vLLM) The logitprobe classifier achieves 82.6% acc
Analysis
TL;DR
- ArguLens is an open-source, locally deployable automated essay scoring (AES) system released under Apache 2.0 that decomposes scoring into three decoupled components for interpretability and privacy
- The system combines a discourse-move classifier (Qwen2.5-7B-Instruct fine-tuned with LoRA on PERSUADE 2.0), a grade-independent LightGBM scorer using 31 linguistic and discourse features, and a label-aware feedback generator (Qwen2.5-14B-Instruct via vLLM)
- The logitprobe classifier achieves 82.6% accuracy and 0.727 macro-F1 on an essay-disjoint PERSUADE 2.0 test split
- The LightGBM scorer reaches a mean QWK of 0.813 under oracle discourse-feature protocol, with gold discourse annotations yielding a statistically significant +0.055 QWK improvement over lexical+syntactic features alone (paired t-test, p = 0.010)
- A Gradio web UI supports pluggable inference backends, single-essay and batch scoring, and downloadable per-essay breakdowns; human-rater evaluation of the feedback generator is left for future work
Why It Matters
ArguLens addresses two critical pain points in the AES landscape: the opacity of holistic scoring outputs and the data privacy/cost barriers imposed by closed-API systems. By offering a fully open-source, locally deployable architecture with interpretable component-level diagnostics, it provides a practical blueprint for educators and researchers who need transparent, auditable essay evaluation without sacrificing model performance.
Technical Details
- Architecture: Three decoupled components — (1) discourse-move classifier using Qwen2.5-7B-Instruct fine-tuned with LoRA on PERSUADE 2.0, (2) grade-independent LightGBM scorer operating over 31 engineered linguistic and discourse features, and (3) label-aware feedback generator served via vLLM with a Qwen2.5-14B-Instruct backbone
- Benchmarks: Evaluated on PERSUADE 2.0 with an essay-disjoint test split; classifier reports 82.6% accuracy and 0.727 macro-F1 via logit probing; scorer achieves mean QWK of 0.813 under prompt-grouped 5-fold cross-validation with oracle discourse features
- Ablation Study: Adding gold discourse annotations to a lexical+syntactic feature set produces a +0.055 QWK improvement, statistically significant at p = 0.010 (paired t-test), confirming discourse structure as a meaningful signal beyond surface-level features
- Deployment: Gradio-based web UI with pluggable inference backends, supporting both single-essay and batch scoring with downloadable per-essay breakdowns; licensed under Apache 2.0
- Limitation: The reported scorer results are component-level diagnostics rather than end-to-end classifier-to-scorer pipeline results; human-rater validation of the feedback generator is deferred to future work
Industry Insight
- The modular, decoupled design of ArguLens demonstrates that high-quality AES can be achieved without monolithic end-to-end models, offering a replicable pattern for building interpretable educational AI systems where auditability is paramount
- The statistically significant gain from discourse features (+0.055 QWK) suggests that investing in structured discourse analysis — rather than relying solely on lexical and syntactic signals — is a high-leverage direction for next-generation scoring systems
- The open-source, locally deployable approach directly counters the growing concern around proprietary AES APIs and data privacy, making it particularly relevant for institutional adopters in K-12 and higher education who must comply with student data protection regulations
Disclaimer: The above content is generated by AI and is for reference only.