Personalized Scorer Modeling: A Learning-Based Framework for Deriving Robust Sleep Stage Labels from Multiple Experts
Introduces a Learning-Based Hypnogram (LBH) framework that constructs more reliable reference sleep stage labels by modeling individual scorer behavior rather than relying on a single scorer's annotations Uses confusion matrices from ML classifiers (Random Forest, SVM, MLP) to estimate the probability of each true sleep stage given each scorer's label, then aggregates probabilities across all scorers Evaluated on DOD-H and DOD-O datasets using EEG (C3-M2) and chin EMG signals with 60 extracted f
Analysis
TL;DR
- Introduces a Learning-Based Hypnogram (LBH) framework that constructs more reliable reference sleep stage labels by modeling individual scorer behavior rather than relying on a single scorer's annotations
- Uses confusion matrices from ML classifiers (Random Forest, SVM, MLP) to estimate the probability of each true sleep stage given each scorer's label, then aggregates probabilities across all scorers
- Evaluated on DOD-H and DOD-O datasets using EEG (C3-M2) and chin EMG signals with 60 extracted features (30 per modality) in 30-second epochs
- Achieved best results with Random Forest + EEG+EMG: 86.07% accuracy, 85.46% precision, 85.29% F1 on DOD-H; 86.04% accuracy, 85.21% precision, 84.70% F1 on DOD-O
- LBH consistently outperformed both the dataset hypnogram (DH) and best-scorer hypnogram (BSH), demonstrating that collective expert behavior yields more robust labels without discarding individual scorer information
Why It Matters
This work addresses a fundamental problem in sleep staging research: inter-scorer variability undermines the reliability of ground truth labels, yet most automatic staging studies evaluate against a single reference hypnogram. By leveraging multi-scored datasets to build probabilistic reference labels, the LBH framework offers a more statistically sound evaluation standard that could improve reproducibility and fairness in sleep disorder diagnosis research.
Technical Details
- Feature extraction: EEG (C3-M2) and chin EMG signals segmented into 30-second epochs, with 30 features extracted from each modality, yielding a combined 60-dimensional feature vector for EEG+EMG settings
- LBH construction: For each scorer, confusion matrices are derived from ML classifiers; after column normalization, these matrices estimate P(true stage | scorer's label), and probabilities are aggregated across all scorers to assign the final epoch label
- Classifiers evaluated: Random Forest, Support Vector Machine (SVM), and Multilayer Perceptron (MLP), tested under both EEG-only and EEG+EMG configurations
- Datasets: DOD-H and DOD-O, publicly available sleep staging datasets with multiple expert scorers
- Baselines: Dataset Hypnogram (DH) — the standard consensus/reference label; Best-Scorer Hypnogram (BSH) — the single best-performing scorer's annotations
Industry Insight
- The LBH approach could become a new standard for constructing ground truth in sleep staging benchmarks, reducing the bias introduced by individual scorer subjectivity and enabling more reliable model comparisons across studies
- This methodology generalizes beyond sleep medicine to any domain with multiple human annotators (e.g., medical imaging, speech transcription), suggesting broader applicability for label fusion and consensus building
- Researchers should consider adopting multi-scored datasets and LBH-style label construction when developing and evaluating sleep staging models, as it may reveal performance gaps that single-hypnogram evaluations obscure
Disclaimer: The above content is generated by AI and is for reference only.