Towards Generalizable Zero-Shot Manipulation via Translating Human Interaction Plans
Abstract: We pursue the goal of developing robots that can interact zero-shot with generic unseen objects via a diverse repertoire of manipulation skills and show how passive human videos can serve as a rich source of data for learning such generalist robots. Unlike typical robot learning approaches which directly learn how a robot should act from interaction data, we adopt a factorized approach that can leverage large-scale human videos to learn how a human would accomplish a desired task (a human `plan'), followed by `translating’ this plan to the robot’s embodiment. Specifically, we learn a human `plan predictor’ that, given a current image of a scene and a goal image, predicts the future hand and object configurations. We combine this with a `translation’ module that learns a plan-conditioned robot manipulation policy, and allows following humans plans for generic manipulation tasks in a zero-shot manner with no deployment-time training. Importantly, while the plan predictor can leverage large-scale human videos for learning, the translation module only requires a small amount of in-domain data, and can generalize to tasks not seen during training. We show that our learned system can perform over 16 manipulation skills that generalize to 40 objects, encompassing 100 real-world tasks for table-top manipulation and diverse in-the-wild manipulation.
Website: https://homangab.github.io/hopman/
Видео Towards Generalizable Zero-Shot Manipulation via Translating Human Interaction Plans канала Homanga Bharadhwaj
Website: https://homangab.github.io/hopman/
Видео Towards Generalizable Zero-Shot Manipulation via Translating Human Interaction Plans канала Homanga Bharadhwaj
Показать
Комментарии отсутствуют
Информация о видео
Другие видео канала
Visual Affordance Prediction for Guiding Robot ExplorationLEAF: Latent Exploration Along the Frontier (ICRA 2021)Gen2Act: Human Video Generation in Novel Scenarios enables Generalizable Robot ManipulationSkill transfer via partially Amortized Hierarchical PlanningRon Dorfman's talk on Offline Meta RLAnirudh Goyal's talkConservative Safety Critics for ExplorationYana Hasson's (INRIA) talkInformation Prioritization through Empowerment in Visual Model-based RLAhmed Qureshi's talk on Motion Planning NetworksPresentation by Yunzhu Li (MIT)Zero-Shot Robot Manipulation from Passive Human VideosRika Antonova's talk on Analytic Manifold LearningZongyi Li's talk on solving PDEs from dataLearning by WatchingRoboAgent Extreme Generalization