Computer Science Department Colloquium: Grounding Object Semantics in Multi-Modal Interactions, by Prof. Jivko Sinapov of Tufts University

Friday, September 15, 2017
11:00 am to 12:00 pm
Floor/Room #: 


Infants use exploratory behaviors to learn about the objects around them. Psychologists have theorized that behaviors such as touching, pressing, lifting, and dropping enable infants to form grounded object representations. For example, scratching an object can provide information about its roughness, while lifting it can provide information about its weight. In a sense, the exploratory behavior acts as a ``question'' to the object, which is subsequently “answered” by the sensory stimuli produced during the execution of the behavior. In contrast, most object representations used by robots today rely solely on computer vision or laser scan data, gathered through passive observation. Such disembodied approaches to robotic perception may be useful for recognizing an object using a 3D model database, but nevertheless, will fail to infer object properties that cannot be detected using vision alone.

To bridge this gap, my research has pursued a developmental framework for object perception and exploration in which the robot's representation of objects is grounded in its own sensorimotor experience with them. In this framework, an object is represented by sensorimotor contingencies that span a diverse set of exploratory behaviors and sensory modalities. In this talk, I will highlight results from several large-scale experimental studies, which show that the behavior-grounded object representation enables a robot to solve a wide variety of perceptual and cognitive tasks relevant to object learning. I will conclude with a discussion on my ongoing work on enabling service robots to learn about objects while simultaneously operating autonomously in human-inhabited environments for extended periods of time.


Jivko Sinapov received his Ph.D. in computer science and human-computer interaction from Iowa State University (ISU) in the Fall of 2013. While working toward his Ph.D. at ISU's Developmental Robotics Lab, he developed novel methods for behavioral object exploration and multi-modal perception. He went on to be a clinical assistant professor with the Texas Institute for Discovery, Education, and Science at UT Austin and a postdoctoral associate working with Peter Stone at the UTCS Artificial Intelligence lab. Jivko Sinapov joined the Tufts University Department of Computer Science in the Fall of 2017 as an Assistant Professor. Jivko's research interests include cognitive and developmental robotics, computational perception, human-robot interaction, and reinforcement learning.

Host: Krishna Venkatasubramanian