Embodied Multimodal Intelligence with Foundation Models - Dr. Oier Mees (UC Berkeley)

Despite considerable progress in robot learning, most real-world robots remain confined to a narrow set of preprogrammed behaviors, falling short of public expectations. As robots become more ubiquitous in human-centered environments, the need for “generalist” robots grows: how can we scale robot learning systems to generalize and adapt, enabling them to perform a wide range of everyday tasks in unstructured settings based on arbitrary user instructions? In this talk, I will discuss the challenges and opportunities in building robot foundation models and outline the key ingredients for developing generalist robot policies—including cross-embodied learning, multimodal alignment, and scalable learning and evaluation procedures. I will present the first instantiation of such a model, capable of performing bimanual manipulation, visual navigation, quadruped locomotion, single-arm manipulation, and even aviation. I will then discuss how this model serves as a pre-trained backbone for downstream tasks, including humanoid control. Finally, I will show how incorporating intelligent reasoning not only enables robots to use common sense to think before acting, but also significantly enhances their generalization, interpretability, and ability to interact effectively with humans.

Видео Embodied Multimodal Intelligence with Foundation Models - Dr. Oier Mees (UC Berkeley) канала Soft Robotics Lab [SRL] / ETH Zurich