Dynamic Influence-Weighted Distillation for Single-IMU Activity Recognition
Dynamic Influence-Weighted Distillation (DIW) enables a single right-arm IMU student model to leverage multi-IMU training data from four body locations without requiring all sensors at inference time DIW introduces sample-wise adaptive gating for both logit and feature distillation losses, determined by testing one-step candidate updates on held-out fold-internal training participants On the WEAR dataset (19 activity labels, 68,298 windows, 22 participants), DIW achieves a macro-F1 of 0.638451,
Analysis
TL;DR
- Dynamic Influence-Weighted Distillation (DIW) enables a single right-arm IMU student model to leverage multi-IMU training data from four body locations without requiring all sensors at inference time
- DIW introduces sample-wise adaptive gating for both logit and feature distillation losses, determined by testing one-step candidate updates on held-out fold-internal training participants
- On the WEAR dataset (19 activity labels, 68,298 windows, 22 participants), DIW achieves a macro-F1 of 0.638451, outperforming supervised learning (0.561820) by 7.66 percentage points and fixed-weight KD (0.571623) by 6.68 percentage points
- The student model retains only 80,915 parameters at inference, using the same single right-arm IMU input across all approaches, making deployment practically unchanged
- DIW exceeds supervised performance on 18 of 19 activity labels and 21 of 22 held-out participants, demonstrating robust generalization across subjects and activities
Why It Matters
This work addresses a critical deployment bottleneck in wearable-based activity recognition: the gap between multi-sensor training setups and single-sensor inference constraints. By showing that multi-position IMU data can effectively distill into a lightweight single-IMU model through adaptive weighting, it offers a practical pathway for reducing hardware costs and user burden in real-world deployments without sacrificing accuracy.
Technical Details
- Architecture: A frozen four-IMU teacher model provides both logit and feature-level distillation targets to a smaller right-arm-only student model (80,915 parameters), with no changes to the student's forward computation graph at inference
- Dynamic Influence Weighting (DIW): For each training sample, DIW evaluates a one-step candidate parameter update on a held-out subset of fold-internal training participants, then assigns separate adaptive gates to the logit loss and feature loss components based on their individual influence
- Baseline comparison: Fixed-weight knowledge distillation applies uniform loss weights across all samples, while DIW dynamically modulates per-sample contribution, recognizing that not all training samples benefit equally from multi-IMU teacher signals
- Evaluation protocol: Subject-disjoint five-fold cross-validation on the WEAR dataset with 22 participants, 19 activity labels, and 68,298 complete windows, ensuring generalization to unseen users
- Performance gains: DIW achieves macro-F1 of 0.638451 versus 0.561820 (supervised) and 0.571623 (fixed-weight KD), with consistent improvements across nearly all labels and participants
Industry Insight
- The DIW approach demonstrates that careful sample-level weighting during distillation can unlock significant value from multi-sensor training data, suggesting that similar adaptive weighting strategies could benefit other sensor-fusion-to-single-sensor compression scenarios in wearable and IoT applications
- The near-universal improvement across 18/19 labels and 21/22 participants indicates that the method generalizes well to unseen users, a critical factor for commercial deployment where per-user calibration is impractical
- The minimal parameter footprint (80,915 parameters) combined with single-IMU inference makes this approach highly suitable for edge deployment on resource-constrained wearables, potentially reducing hardware costs while maintaining competitive accuracy
Disclaimer: The above content is generated by AI and is for reference only.