Yunzhu Li (@YunzhuLiYZ)
2024-08-29 | โค๏ธ 275 | ๐ 48
Super excited to finally release our work, ReKep, a unified task representation using relational keypoint constraints. ๐ค https://rekep-robot.github.io/
A few key takeaways:
- Building on the success of VoxPoser, VLM-generated code has proven to be extremely versatile in task specification.
- ReKep provides a much more fine-grained task representation than the value map in VoxPoser.
- ReKep nicely connects to the Task and Motion Planning (TAMP) field, characterizing both spatial (e.g., shoe tip alignment) and temporal relationships (e.g., keeping the teapot upright during transport).
- ReKep enables real-time motion planning at 10 Hz using an off-the-shelf optimizer, allowing for closed-loop control both within and across different task stages (as shown in the video).
- Leveraging modules for their strengths (VLM for task interpretation, constrained optimization for motion planning) remains an effective way to ground foundation models in robotics.
- Our investigation complements recent trends in large-scale imitation learning, and I can already see several low-hanging opportunities to combine the best of both worlds!
Check @wenlong_huang โs thread for a more detailed walkthrough of the work! ๐
๋ฏธ๋์ด
Quoted: @wenlong_huang
What structural task representation enables multi-stage, in-the-wild, bimanual, reactive manipulation?
Introducing ReKep: LVM to label keypoints & VLM to write keypoint-based constraints, solve wโฆ
์ธ์ฉ ํธ์
Wenlong Huang (@wenlong_huang)
What structural task representation enables multi-stage, in-the-wild, bimanual, reactive manipulation?
Introducing ReKep: LVM to label keypoints & VLM to write keypoint-based constraints, solve w/ optimization for diverse tasks, w/o task-specific training or env models.
๐งต๐ https://t.co/EnAPYUTjmN
๐ฌ ์์
