mvp-of-multiview-video-camera-parameters-3d-keypoints

Pablo Vela (@pablovelagomez1)

2025-05-09 | ❤️ 485 | 🔁 66

---

MVP of Multiview Video → Camera parameters + 3D keypoints. Visualized with @rerundotio

The basic pipeline as of right now looks like this:

1. Capture 🔴 – Using 4 iPhones and an Insta360 Go. iPhone videos are captured via Final Cut Pro Multicam for easy sync and the exocentric view; the Insta360 Go is used for the egocentric view.
2. Sync 🕒 – Custom Gradio app using two @rerundotio viewers and callbacks for easily aligning frame timestamps so the ego and exo views are aligned.
3. Calibrate 🎯 – Use VGGT from @jianyuan_wang and @AIatMeta to get intrinsics/extrinsics for sparse cameras.
4. Estimate 3D 🕺 – Use RTMLib whole‑body keypoint estimator on each frame, then triangulate in 3D.

What’s missing?

1. No temporal coherence: I’m estimating keypoints one frame at a time and one camera at a time. This leads to a lot of jittering. For now, I plan on adding a One Euro Filter to help with jittering. Long term, I’d want to train a multiview keypoint estimator
2. Kinematic fitting is still missing; this is my next goal. The output will be joint angles, as explored in my previous posts.
3. Missing dense point cloud: VGGT seems to fail for me here. I’m looking to explore using MP‑SFM as a method for generating dense multiview depth maps + normals (plus it has a friendlier license compared to VGGT).
4. Eventually, creation of 4D Gaussian splatting using something akin to DN‑splatter—my long‑term goal is a data engine that provides poses/depths/splats/keypoints/etc.

미디어

---

🔗 Related

Auto-generated - needs manual review

인용 트윗

Pablo Vela (@pablovelagomez1)

Synchronization ✅ Calibration with VGGT ✅ Next up is hand tracking + kinematic fitting

Almost there 😮‍💨 https://t.co/de7fy0DYLr

원본 트윗

🎬 영상 🎬 영상

Tags

domain-vision-3d domain-visionos