PhD Speaking Qualifier | Simultaneous Estimation of Hand Configurations and Finger Joint Angles Using Forearm Ultrasound

Robotics Engineering Department PhD Speaking Qualifier

Keshav Bimbraw

Simultaneous Estimation of Hand Configurations and Finger Joint Angles 

Using Forearm Ultrasound

Thursday, March 2, 2023

12:00 PM - 1:00 PM

Virtual | Please email: Keshav Bimbraw (kbimbraw@wpi.edu) for the zoom link

Abstract: With the advancement in computing and robotics, it is necessary to develop fluent and intuitive methods for interacting with digital systems, augmented/virtual reality(AR/VR) interfaces, and physical robotic systems. Hand motion recognition is widely used to enable these interactions. Hand configuration classification and metacarpophalangeal (MCP) joint angle detection is important for a comprehensive reconstruction of hand motion. Surface electromyography (sEMG) and other technologies have been used for the detection of hand motions. Forearm ultrasound images provide a musculoskeletal visualization that can be used to understand hand motion. Recent work has shown that these ultrasound images can be classified using machine learning to estimate discrete hand configurations. Estimating both hand configuration and MCP joint angles based on forearm ultrasound has not been addressed in the literature. In this paper, we propose a convolutional neural network (CNN) based deep learning pipeline for predicting the MCP joint angles. The results for the hand configuration classification were compared by using different machine learning algorithms. Support vector classifiers with different kernels, multi-layer perception, and the proposed CNN has been used to classify the ultrasound images into 11 hand configurations based on activities of daily living. Forearm ultrasound images were acquired from 6 subjects instructed to move their hands according to predefined hand configurations. Motion capture data was acquired to get the finger angles corresponding to the hand movements at different speeds (0.5 Hz, 1 Hz, & 2 Hz) for the index, middle, ring, and pinky fingers. Average classification accuracy of 82.7 ± 9.7% for the proposed CNN and over 80% for SVC for different kernels were observed on a subset of the dataset. An average RMSE of 7.35◦±1.3◦ was obtained between the predicted and the true MCP joint angles. A low latency (6.25 - 9.1 Hz) pipeline has been proposed for estimating both MCP joint angles and hand configuration aimed at real-time control of human-machine interfaces.

Research Advisor:

Professor Haichong (Kai) Zhang, Worcester Polytechnic Institute (WPI)

Qualifier Evaluators: 

Professor Loris Fichera, Worcester Polytechnic Institute (WPI)

Professor Siavash Farzan, Worcester Polytechnic Institute (WPI)