ECE MS Thesis Presentation by S.A.I. Shouborno, Interpretable and Efficient AI for Sensor-Rich Mobile and Embedded Systems

Title:

Interpretable and Efficient AI for Sensor-Rich Mobile and Embedded Systems 

Abstract:

Wearable and embedded sensing systems can deduce a vast amount of information from signals such as inertial measurement unit (IMU) data, for example by recognizing human activities, but they cannot explain the patterns they detect with contextual reasoning that is accessible to users. This thesis introduces LLaSA (Large Language and Sensor Assistant), a large language model that explains what sensors tell us about how we move. LLaSA is built on two new datasets: SensorCap, comprising 35,960 IMU-caption pairs, and OpenSQA, with 199,701 question-answer pairs grounded in motion. With an IMU encoder that embeds sensor information into a compact representation, LLaSA can fit sensor sequences into a much smaller context window than commercial general-purpose LLMs and still outperform them on benchmark and real-world activity interpretation tasks, demonstrating the effectiveness of domain supervision and model alignment for sensor reasoning.

To reduce the computational cost of deploying deep neural networks on resource-constrained hardware, the thesis also develops Variance-Optimized Layer-wise Termination (VOLT), an early-exit framework for energy-efficient on-device inference. Early-exit networks can save compute by classifying easy samples at shallow layers, but existing systems often rely on manually tuned confidence thresholds or auxiliary gating networks for exit criteria and do not consistently optimize shallow exits. VOLT introduces a variance-optimizing multi-exit loss that encourages decisive, high-certainty predictions at each exit, together with a validation-guided entropy calibration procedure that sets per-exit thresholds offline to satisfy a target accuracy without additional runtime computation. The framework further proposes Exit-population and MAC-scaled Anti-Risk (EMAR), a unified metric that links exit behavior, correctness, and compute cost. Experiments on five sensor, vision, and language datasets and three backbone architectures show up to 6.18× improvement in EMAR and up to 36.7% savings in multiply–accumulate operations compared with prior early-exit methods, with Raspberry Pi 5 and ESP32-CAM deployments confirming 1.3× to 3.6× reductions in latency and energy.

Together, these contributions advance interpretability and efficiency for sensor-rich embedded systems and motivate future work on trustworthy multimodal representation learning and AI for mobile and wearable platforms.

Research Advisor:

Prof. Bashima Islam

ECE Department, WPI

Research Committee:

Prof. Ziming Zhang

ECE Department, WPI

Prof. Fabricio Murai

DS Department, WPI