Pablo Vela (@pablovelagomez1)
2025-05-22 | โค๏ธ 179 | ๐ 20
Streaming iPhone data in real-time directly to @rerundotio ๐
The collection process is one of the most frustrating parts of building imitation-learning datasets. Iโve got a little army of sensorsโ๐ฑ iPhone, iPad, Quest 3โbut getting them temporally aligned, spatially aligned, AND seeing real-time feedback while recording is tough.
I stumbled on a great library from @wpicakelab called ARFlow. Itโs a thin client built on Unityโs ARFoundation that connects over gRPC to a server running Rerun for live data logging. I forked it to:
- Log the SLAM translation poses, and
- Upgrade rerun to v0.23 for my use case.
So far, it works well, but there are still a few hitches:
- Right now, itโs solid on iPhone and iPad; my Quest 3 client is still slow and not super reliable.
- Iโm using an older ARFlow branch focused on real-time streaming onlyโno spatial or temporal sync yet. Unity builds for iOS keep failing. ๐ ๏ธ
- Nothing is saved locally to the client, so packet loss is a risk on shaky networks.
Thereโs huge potential in tapping the ubiquitous sensors we carry around every day, and ARFlow is a big step toward making that easy
๋ฏธ๋์ด
์ธ์ฉ ํธ์
Pablo Vela (@pablovelagomez1)
As part of the robot training pipeline, I needed the egocentric perspective. Iโm building a streaming pipeline using @rerundotio and a Quest3 to fill that need.
So far, Iโve focused on the third-person (exocentric) view, so a friend and I built a simple Unity app that takes the Quest 3 passthrough camera and streams it directly to a rerun instance. This is going to be the critical component for scale. Along with this, it should be easy enough to get camera pose data + hand tracking data
This builds on top of the awesome ARFlow project, so with some more work, I should be able to extend it to work with any heterogeneous device (ARKit/ARCore/AndroidXR?)
Once I have a more stable version, Iโll release more info on it soon!
๐ฌ ์์
Tags
domain-vision-3d domain-robotics domain-ai-ml domain-dev-tools