FusionSense: Tri-Stage Near-Sensor Learning for Runtime-Adaptive Multimodal Edge Intelligence

Background

Autonomous systems and smart-industry deployments increasingly rely on distributed computation across near-sensor, edge, and cloud resources. These environments face stringent energy, latency, and reliability constraints that necessitate real-time adaptivity in decision-making processes involving multimodal sensor data (cameras, LiDAR/depth sensors). Traditional approaches either centrally fuse modalities or apply uni-modal filters, which often result in redundant transmissions or missed events.

Key Points

FusionSense addresses these issues by introducing a fusion-aware intelligent sensing framework that:

1. Trains Lightweight Classifiers Near-Sensor: A three-step procedure is employed to train classifiers at the edge.
2. Quantifies Modality Necessity via Filter-Out-Safe (FoS) Labels: This step evaluates each modality's contribution relative to the fused decision, ensuring minimal redundant transmissions.
3. Compacts Edge Fusion Models Using Near-Sensor Predictions: By integrating auxiliary signals from near-sensor predictions into edge-side fusion models, FusionSense enhances computational efficiency.

Step-by-Step Procedure:

1. Server-Side Fusion Model: A powerful server first learns the downstream task through a comprehensive fusion model.
2. Filter-Out-Safe (FoS) Labels: These labels quantify each modality's necessity in decision-making, allowing for efficient filtering at the edge.
3. Edge-Side Fusion Model Compaction: Near-sensor predictions are injected as auxiliary signals to compact edge-side fusion models.

The result is a runtime decision layer that jointly reduces both compute and communication costs while scaling linearly with sensor count.

Significance

FusionSense demonstrates significant improvements over existing approaches:

• Data Reduction Rates: Sustains task quality at substantially higher data-reduction rates than uni-modal filters.
• Energy Efficiency: Up to 33x lower energy consumption at 1% FoI prevalence and 11x at 10%.
• Quality Loss Reduction: A 92.3% reduction in quality loss when maintaining a fixed 30% data reduction rate.
• Comparative Advantage: Provides roughly 1.5x higher energy savings compared to the best prior filtering baseline.

These findings underscore FusionSense’s potential to revolutionize autonomous edge systems by optimizing both computational and communication resources effectively.