Estimating from No Data: Deriving a Continuous Score from Categories
A toy medical dataset with 8 features and 3 categorical outcomes (home recovery, hospitalization, death) is used to explore the challenge of learning a continuous severity score from discrete labels A standard 6-layer classifier achieves 98-99% validation accuracy but cannot produce a meaningful continuous risk score Naive MSE regression on encoded categories (0, 1, 2) causes the network to collapse outputs into discrete clusters around the three target values, losing granularity Reducing networ
Analysis
TL;DR
- A toy medical dataset with 8 features and 3 categorical outcomes (home recovery, hospitalization, death) is used to explore the challenge of learning a continuous severity score from discrete labels
- A standard 6-layer classifier achieves 98-99% validation accuracy but cannot produce a meaningful continuous risk score
- Naive MSE regression on encoded categories (0, 1, 2) causes the network to collapse outputs into discrete clusters around the three target values, losing granularity
- Reducing network capacity to a single linear dense layer (no activation) forces a more continuous, interpretable scoring function where the output is a weighted linear combination of inputs
- The core insight is that high classification accuracy does not guarantee a useful continuous scoring system; architectural constraints are needed to preserve output variance
Why It Matters
This work highlights a critical gap between classification performance and the practical need for continuous risk stratification in clinical decision-making. AI practitioners building medical prediction systems must recognize that standard classifier architectures, even at near-perfect accuracy, may fail to capture the nuanced severity gradients that clinicians need for triage and treatment prioritization. The approach of constraining model capacity to preserve output continuity is broadly applicable beyond healthcare to any domain requiring ordinal scoring from categorical labels.
Technical Details
- Dataset: A deterministic, non-linear toy dataset with 8 features (previous infections with Pathogen A/B, acute Pathogen B infection, cancer diagnosis, weight deviation, age, blood pressure deviation, years smoked) and three equally probable outcomes; features are uniformly sampled with age-dependent smoking distribution and 50% non-smoker cohort
- Classifier architecture: 6-layer neural network, each layer 8 units wide, ReLU activation, achieving 98-99% validation accuracy via
train.py --classifier - Failed scoring approach: Single-output network with 2 dense ReLU layers trained with MSE loss on encoded targets (0/home, 1/hospital, 2/death); produces a histogram with three sharp peaks, indicating output collapse
- Proposed solution: A degenerate linear network — a single dense layer with no activation — acting as linear regression, producing an interpretable weighted sum of inputs as a continuous severity score
- Input preprocessing: All features normalized to zero mean and unit variance; years smoked mean and variance measured from the non-smoker cohort for proper normalization
Industry Insight
- When deploying ML models for clinical triage or risk scoring, prioritize output continuity and interpretability over raw classification accuracy; a model that achieves perfect category prediction may be clinically useless if it cannot rank patients within a severity spectrum
- Architectural capacity control (e.g., bottleneck layers, linear outputs) is a practical and effective technique for forcing models to preserve information that would otherwise be discarded during categorical optimization
- For any domain where ordinal or continuous scores are needed but only categorical labels are available, consider constraining the output layer and training objective to prevent mode collapse, and validate the score distribution against domain expectations rather than relying solely on accuracy metrics
Disclaimer: The above content is generated by AI and is for reference only.