Visually-Grounded Planning with Transformers in Procedural Tasks
In this work, we study the problem of how to leverage instructional videos to facilitate the understanding of human decision-making processes, focusing on training a model with the ability to plan a goal-directed procedure from real-world videos. Learning structured and plannable state and action spaces directly from unstructured videos is the key technical challenge of our task. There are two problems: first, the appearance gap between the training and validation datasets could be large for unstructured videos; second, these gaps lead to decision errors that compound over the steps. We address these limitations with Planning Transformer (PlaTe), which has the advantage of circumventing the compounding prediction errors that occur with single-step models during long model-based rollouts. Our method simultaneously learns the latent state and action information of assigned tasks and the representations of the decision-making process from human demonstrations. Experiments conducted on real-world instructional videos show that our method can achieve a better performance in reaching the indicated goal than previous algorithms. We also validated the possibility of applying procedural tasks on a UR-5 platform.
More Qualitative Results on CrossTask
Representative samples on CrossTask and a qualitative comparison of our methods with baselines.
More Qualitative Results on ActioNet
Representative samples on ActioNet and a qualitative comparison of our methods with baselines.
Reasons for failure cases:
- The task is complex
- The state contains multiple semantics
- The states looks very similar
This project is supported by the following.