LLM-Empowered Multimodal Human-Robot Collaborative Control for Robotic Nursing Assistance
Abstract: This thesis presents a multimodal human-robot interaction system that enables a nurse to control a remote manipulation robot through natural gaze and speech commands. The nurse wears Tobii Pro Glasses 3 and speaks commands while looking at objects on a patient's bedside table. The system fuses eye-gaze fixations with word-level speech timestamps in real time to resolve deictic references (e.g., "put that on the trolley"), then processes the grounded command through a four-agent LLM pipeline for intent resolution, dialog clarification, task planning, and plan verification. The architecture spans two compute platforms bridged by MQTT—a Jetson AGX Orin running perception and LLM agents under ROS 2, and a robot machine(IONA) running manipulation under ROS 1. We validate the system on IONA, a dual-arm mobile nursing robot, performing bedside table-clearing tasks with eight objects and two destinations, demonstrating end-to-end multimodal interaction in a realistic scenario.
Advisor: Professor Jane Li
Committee: Professor Haichong Zhang, Professor Fiona Yuan
Zoom link: https://wpi.zoom.us/j/91623889103?pwd=HbQFG1SOMgAHUdyaA1Nh34MPavTCax.1