Embodied AI Reading Notes (@EmbodiedAIRead)
2025-10-28 | โค๏ธ 148 | ๐ 22
GaussGym: An open-source real-to-sim framework for learning locomotion from pixels
Project:ย https://escontrela.me/gauss_gym/ Paper:ย https://arxiv.org/abs/2510.15352 Data:ย https://huggingface.co/collections/escontra/gauss-gym-datasets Code:ย https://github.com/escontra/gauss_gym
This work creates a fast open-source photorealistic robot simulation that integrates 3D Gaussian Splatting as drop-in renderer in vectorized physics simulators like IsaacGym, enabling vision-based robot policy training at 100K+ steps/second in photorealistic 3D environments captured from iPhone videos, datasets, or AI-generated worlds (e.g. via Veo).
-
GaussGym uses Visually Grounded Geometry Transformer (VGGT) to standardize and extract information from various input to initialize Gaussian Splats, and uses Neural Kernel Surface Reconstruction (NKSR) to produce high-quality meshes.
-
Once reconstructed, the highly optimized implementation of Gaussian Splats can enable render up to 4096 environments across 128 scenes at 100k steps per second in simulator on a single RTX 4090 GPU.
-
The authors show effectiveness of GaussGym by training locomotion and navigation visuomotor policies with RL directly from RGB, and successfully zero-shot transfer it to real-world stair climbing.
-
This work bridges high-throughput simulation and high-fidelity perception, which brings great potential on advancing scalable and generalizable robot learning directly from Vision, with all code and data open-sourced for the community to build upon.
๐ Related
Auto-generated bookmark