Seminar Series
“From Mindfulness Sensing to Multimodal Reasoning: Toward Sensor-Language Intelligence Beyond Vision”
Bashima Islam, PhD
Assistant Professor of Electrical and Computer Engineering
Worcester Polytechnic Institute
Abstract: The rapid growth of multimodal AI opens new opportunities for sensing, reasoning, and interaction, yet most systems still focus narrowly on vision and overlook signals critical for human-centered applications. In this talk, I will present three recent projects from my group that collectively broaden the scope of multimodal intelligence. First, I will introduce our work on mindfulness and respiration sensing, where we design smartphone-based algorithms that track respiration rate and estimate mindfulness skill progression using only accelerometer data. This study demonstrates how sensor feedback can enhance usability and engagement in digital mindfulness training, providing a compelling case for health-oriented multimodal AI. Next, I will present RAVEN, a unified architecture for multimodal question answering. At its core is QuART, a query-conditioned token gating module that learns to assign relevance scores across modalities, enabling the system to amplify informative cues while suppressing distractors. Through a staged training pipeline, RAVEN achieves robust reasoning across video, audio, and sensor streams. Finally, I will discuss LLaSA, the Large Language and Sensor Assistant, which extends multimodal research to wearable sensing. LLaSA introduces new datasets and evaluation frameworks for aligning sensor signals with language, offering the first general-purpose assistant that reasons jointly over sensor data and natural language queries. Together, these projects chart a path toward sustainable multimodal systems that hear, sense, and reason with the world, advancing both human-centered applications and core AI methods.
Bio: Dr. Bashima Islam is an Assistant Professor of Electrical and Computer Engineering at Worcester Polytechnic Institute, with affiliations in Computer Science and Data Science. Her research transforms AI through the fusion of sensors, speech, and language to build sustainable, intelligent systems for the edge. She focuses on bridging AI with real-world impact that advance acoustic understanding, behavioral health monitoring, and real-world multimodal intelligence while considering resource constrained of low-power devices. She has been recognized with several prestigious honors, including the NSF CRII Award, multiple NIH research grants, and selection to Forbes 30 Under 30 in Science.
For a zoom link please contact Kate Harrison at kharrison@wpi.edu