Skip to main content

RBE Colloquium Series: Dr. Luca Carlone | Metric-Semantic Mapping and High-level Scene Understanding for Single and Multi-Robot System


Various images of robots at Robotics Engineering WPI alt
WPI Robotics Engineering
Friday, November 19, 2021
2:00 pm to 3:00 pm



Dr. Luca Carlone

Massachusetts Institute of Technology (MIT)

Metric-Semantic Mapping and High-level Scene Understanding for Single and Multi-Robot Systems


Friday, November 19, 2021

2:00 PM – 3:00 pm

Virtual |


Abstract: Spatial perception has witnessed an unprecedented progress in the last decade. Robots are now able to detect objects and create large-scale maps of an unknown environment, which are crucial capabilities for navigation and manipulation. Despite these advances, both researchers and practitioners are well aware of the brittleness of current perception systems, and a large gap still separates robot and human perception. For instance, while humans are able to quickly grasp both geometric and semantic aspects of a scene, high-level scene understanding remains a challenge for robotics. This talk discusses recent efforts targeted at bridging this gap.

I present our work on high-level scene understanding and hierarchical representations for robotics, including Kimera and 3D Dynamic Scene Graphs, and discuss their potential impact on planning and decision-making, human-robot interaction, long-term autonomy, and scene prediction. The creation of a 3D Dynamic Scene Graph requires a variety of algorithms, ranging from model-based estimation to deep learning. I provide an overview of the first generation of Spatial Perception Engines, that extend the traditional notions of mapping and SLAM, and allow a robot to build a “mental model” of the environment, including spatial concepts (e.g., humans, objects, rooms, buildings) and their relations at multiple levels of abstraction. Similar to the role played by occupancy grid maps or landmark-based maps in the past, 3D Dynamic Scene Graphs offer a new, general, and powerful representation and the grand challenge of designing algorithms that can estimate 3D Scene Graphs in real-time from sensor data has the potential to spark new research ideas and push the community outside the “SLAM comfort zone”. I also discuss recent extensions, including our work on Neural Tree architectures for semantic inference and learning in 3D Scene Graphs, and Kimera-Multi, the first distributed multi-robot mapping system that builds dense metric-semantic meshes of the environment in real-time.

Bio: Luca Carlone is the Leonardo Career Development Associate Professor in the Department of Aeronautics and Astronautics at the Massachusetts Institute of Technology, and a Principal Investigator in the Laboratory for Information & Decision Systems (LIDS). He received his PhD from the Polytechnic University of Turin in 2012. He joined LIDS as a postdoctoral associate (2015) and later as a Research Scientist (2016), after spending two years as a postdoctoral fellow at the Georgia Institute of Technology (2013-2015). His research interests include nonlinear estimation, numerical and distributed optimization, and probabilistic inference, applied to sensing, perception, and decision-making in single and multi-robot systems. His work includes seminal results on certifiably correct algorithms for localization and mapping, as well as approaches for visual-inertial navigation and distributed mapping. He is a recipient of the Best Student Paper Award at IROS 2021, the Best Paper Award in Robot Vision at ICRA 2020, a 2020 Honorable Mention from the IEEE Robotics and Automation Letters, a Track Best Paper award at the 2021 IEEE Aerospace Conference, the 2017 Transactions on Robotics King-Sun Fu Memorial Best Paper Award, the Best Paper Award at WAFR 2016, the Best Student Paper Award at the 2018 Symposium on VLSI Circuits, and he was best paper finalist at RSS 2015 and RSS 2021. He is also a recipient of the NSF CAREER Award (2021), the RSS Early Career Award (2020), the Google Daydream (2019) and the Amazon Research Award (2020), and the MIT AeroAstro Vickie Kerrebrock Faculty Award (2020). At MIT, he teaches “Robotics: Science and Systems,” the introduction to robotics for MIT undergraduates, and he created the graduate-level course “Visual Navigation for Autonomous Vehicles”, which covers mathematical foundations and fast C++ implementations of spatial perception algorithms for drones and autonomous vehicles.

Find out more: Watch previous RBE Colloquia on the Robotics Engineering Colloquium Stream channel.