Sensitivity Analysis of GRU, LSTM and Transformer Encoder in Classification of Automated Driving Systems
Three sequence-based models (GRU, LSTM, Transformer encoder) were evaluated for classifying Level 2 ADSs using only vehicle telematics data, achieving macro F1-scores of 0.92, 0.90, and 0.93 respectively on clean data A modular robustness evaluation framework was introduced, simulating five corruption families at five severity levels (L1-L5) across continuous channels and binary event signals Threat-matched training yielded only a modest performance drop (0.904-0.916 macro F1), demonstrating res
Analysis
TL;DR
- Three sequence-based models (GRU, LSTM, Transformer encoder) were evaluated for classifying Level 2 ADSs using only vehicle telematics data, achieving macro F1-scores of 0.92, 0.90, and 0.93 respectively on clean data
- A modular robustness evaluation framework was introduced, simulating five corruption families at five severity levels (L1-L5) across continuous channels and binary event signals
- Threat-matched training yielded only a modest performance drop (0.904-0.916 macro F1), demonstrating resilience to known threat types
- Event-level corruptions caused minimal degradation (macro F1 ≥ 0.87 at L5), while temporal jitter catastrophically collapsed performance to 0.44-0.50 across all models
- The sharp failure-mode split reveals that temporal alignment is the critical vulnerability for telematics-based ADS classification systems
Why It Matters
This research addresses a critical safety and regulatory need as Software Defined Vehicles increasingly run multiple ADSs simultaneously, requiring reliable independent monitoring systems. The findings provide actionable guidance for deploying robust ADS classification in real-world telematics environments where data quality degradation is inevitable, directly impacting insurance assessment, anomaly detection, and regulatory compliance frameworks.
Technical Details
- Models evaluated: GRU, LSTM, and Transformer encoder architectures trained on vehicle telematics data to classify four driving modes: Comma Openpilot, Tesla Autopilot, Cadillac Super Cruise, and manual driving
- Clean-data performance: Macro F1-scores of 0.92 (GRU), 0.90 (LSTM), and 0.93 (Transformer encoder), with threat-matched training maintaining 0.904-0.916 across all models
- Corruption framework: Five corruption families applied at L1-L5 severity levels — continuous channels perturbed with additive white Gaussian noise (cumulative drift), correlated cross-channel noise, and temporal jitter; binary events subjected to burst loss, delayed transitions, spurious toggles, and cross-feature inconsistencies
- Evaluation metric: Macro-F1 used throughout to ensure equal weighting across classes, appropriate for the imbalanced multiclass classification scenario
- Key finding: Temporal jitter is the dominant failure mode, reducing macro-F1 by approximately 40-50% regardless of architecture choice, while event-level corruptions preserve strong performance
Industry Insight
- Telematics-based ADS monitoring systems must prioritize temporal synchronization and jitter mitigation as a first-order engineering requirement; architectures alone cannot compensate for misaligned time series data
- The modest penalty from threat-matched training suggests that adversarial-style robustness training is a viable and efficient strategy for improving real-world deployment resilience without sacrificing clean-data accuracy
- The dramatic sensitivity to temporal jitter implies that fleet operators and insurors should implement strict data quality validation pipelines and timestamp integrity checks before feeding telematics into classification models
Disclaimer: The above content is generated by AI and is for reference only.