Ultrasound based Hand Motion and Force Tracking powered by Deep Learning for Human-Machine Interfacing
Abstract: With the advancements in artificial intelligence and machine learning, there is a growing interest in human-machine interfacing systems. This is because now it's possible to process large amounts of data to extract meaningful information. Biosignal based human-machine interfacing aims to utilize signals obtained from the human body to control digital, physical and augmented/virtual reality systems. Forearm ultrasound data is particularly attractive in this domain since it provides a visualization of the forearm musculature responsible for how the hands interact with the world.
My research focuses on utilizing machine learning to use the data provided by ultrasound to make meaningful estimations of hand movements. Specifically, the research focuses on estimating hand gestures, finger angles and finger forces. A major focus has been given to estimating multiple quantities in a single end-to-end pipelines for real-time human-machine interfacing. Towards that, simultaneously estimating finger angles and hand gestures has been explored. This kind of simultaneous estimation also finds relevance for robotic manipulation, towards which, simultaneously estimating manipulation skills and forces applied by hands on objects has been studied. Additionally, efforts have been made towards improving the design of the wearable ultrasound device, and improving the performance towards estimating these quantities. By advancing the capabilities of forearm ultrasound based hand movement and force estimation, this research is a step in the direction of utilizing a widespread non-ionizing medical imaging modality towards effective human-machine interfacing towards a future where seamless control of robots and digital and virtual interfaces becomes a necessity.
Advisor: Professor Haichong Zhang
Committee: Professor Ziming Zhang, Professor Robert D. Howe, Professor Toshiaki Koik-Akino
Zoom link: https://wpi.zoom.us/j/95317299474