Data-Efficient Visual-Tactile World Models for Robot Deployment in the Open World
Abstract:
Despite recent advances in robotics, the robust deployment of robots in the open world for practical tasks remains a formidable challenge. A significant roadblock is that robots lack a fundamental understanding of their interaction with the physical world, especially in new scenarios. Traditional model-based approaches utilize geometry primitives and physics models, which require significant prior knowledge and fail in the unknown. On the other hand, deep-learning-based approaches are extremely data-hungry and brittle against distribution shifts. In this talk, I will talk about how I developed robot world model representations that tightly integrate physics modeling and machine learning to allow data-efficient and accurate world models for contact-rich tasks involving rigid bodies, granular media, and deformable objects. Next, I will discuss how I tailor the world model representations to downstream active perception to allow fast adaptation of the world models with sparse online visual-tactile data. Following this, I will briefly talk about how I leveraged the learned world models for planning locomotion and manipulation tasks, and designed low-cost robot hardware with multi-sensory capabilities, which is a prerequisite towards building predictive world models from multi-modal sensing. Finally, I will conclude with my future research directions in representing, acquiring, and using robot world models for contact-rich tasks in the open world.
Bio:
Yifan Zhu is currently a postdoctoral research associate in the Mechanical Engineering and Materials Science Department at Yale University. Prior to this, he obtained a Ph.D. degree in the Computer Science Department at the University of Illinois Urbana-Champaign and a B.E. degree in the Mechanical Engineering Department at Vanderbilt University. His research centers around developing representations for robots that facilitate tight integration of physics modeling and machine learning for predictive world modeling from visual-tactile perceptions, especially in the low-data regime. His research has been featured in top-tier robotics venues such as R:SS, ICRA, IROS, RA-L, and IJSR. He also co-led the UIUC team in placing 4-th in the $10M ANA Avatar XPRIZE competition.
Zoom link: https://wpi.zoom.us/j/93413349160