Apr 22
Has Automated Essay Scoring Reached Sufficient Accuracy? Deriving Achievable QWK Ceilings from Classical Test Theory
★★★★★
significance 2/5
The paper investigates the theoretical limits of Automated Essay Scoring (AES) accuracy using quadratic weighted kappa (QWK). It derives two specific ceilings—theoretical and human-like—to better understand the gap between current model performance and achievable accuracy under label noise.
Why it matters
Establishes the theoretical upper bounds of model performance to distinguish between genuine algorithmic progress and the inherent noise of human grading.
Tags
#automated essay scoring #quadratic weighted kappa #classical test theory #model evaluationRelated coverage
- Global South OpportunitiesPivotal Research Fellowship 2026 (Q3): AI Safety Research Opportunity - Global South Opportunities
- arXiv cs.AIAn Intelligent Fault Diagnosis Method for General Aviation Aircraft Based on Multi-Fidelity Digital Twin and FMEA Knowledge Enhancement
- arXiv cs.AIPExA: Parallel Exploration Agent for Complex Text-to-SQL
- arXiv cs.AIThe Power of Power Law: Asymmetry Enables Compositional Reasoning
- arXiv cs.AIOn the Existence of an Inverse Solution for Preference-Based Reductions in Argumentation