3 ms·
What's the state of the art for structure from motion? Given video of a space, how does one practically turn it into a 3d scene today?
by sneak 3y ago
What's the state of the art for structure from motion? Given video of a space, how does one practically turn it into a 3d scene today?
- naillo 3y agoLiterally gaussian splatting (and nerfs)
- andybak 3y agoVideo is just a bunch of still images (except potentially with degraded quality from resolution and/or motion blur) so it's much the same as asking "how does one practically capture a 3d scene" without the video part. Regular photogrammetry - but with interest and practicality gradually increasing for NeRF style approaches. (Although I am suprised that there's not more attention being paid to the extra spatial relationships that can be inferred from merely knowing something is a video. Surely it could be used as an extra constraint when inferring camera poses - you know the camera can only move in certain ways from one frame to the next?)
- radarsat1 3y ago> (Although I am suprised that there's not more attention being paid to the extra spatial relationships that can be inferred from merely knowing something is a video. Surely it could be used as an extra constraint when inferring camera poses - you know the camera can only move in certain ways from one frame to the next?) Can I ask why you say this? Camera motion models including Kalman filters of various kinds and constraints on derivatives etc, are absolutely used in SLAM and photogrammetry afaik, and have been for a long time.
- andybak 3y agoOK maybe it's my limited experience but last time i tried it, video was treated as "extract frames and then only use every 50th (or so)" This was a couple of years ago probably. Meshroom and Reality Capture. Maybe I was mistaken or maybe they've improved? Can you point me to software that does use video for photogrammetry in the way you describe?
- regularfry 3y agoThe Gaussian Splatting paper itself uses COLMAP, which really isn't that new. I have thoughts about marching cubes across splatted gaussians to get a mesh out, but I don't think I've seen that done yet.
- teraflop 3y ago"Structure from motion" refers to the process of reconstructing camera poses and a sparse point cloud from a set of images (or video). That's the input to the scene reconstruction process described in this paper. As far as I know, the basic approach to SfM hasn't changed much in the last decade or so. It boils down to image feature extraction (using something like SIFT feature vectors), then a heuristic matching process, then bundle adjustment and outlier rejection.
- sneak 3y agoMy application is for taking videos of rooms and turning them into to-scale models in a 3d modeling software (to be quantized and re-textured manually). Basically a rough floor plan generator from walkthrough videos. Is there something off the shelf that does this, or a library that can generate these point clouds so I can go about trying to fit simple geometry to them to reconstruct the scene? Please forgive the noob questions; I've not worked with automated 3d stuff or photogrammetry before.
- blensor 3y agoTake a look at Meshroom from https://alicevision.org/ https://alicevision.org/ It's a pretty decent tool, although not sure if videos work well. I once tried to build a mesh from an aerial video by extracting certain frames and the result was ok-ish
- CyberDildonics 3y agointo to-scale models What does this mean? Why would scale matter in a 3D model when you can scale it however you want in two seconds?
- mkl 3y ago"Scale model" means things are correctly sized relative to each other, e.g. your square room doesn't end up rectangular.
- 3y ago
- blovescoffee 3y agoCheck out the GitHub repo hierarchal localization which uses DL + traditional SfM