3 ms·

Related question that I've had for a while. I have 25 years worth of family pictures, many of them taken at my childhood home. The house is no longer in the fam

Related question that I've had for a while. I have 25 years worth of family pictures, many of them taken at my childhood home. The house is no longer in the family, and I desperately want a photorealistic 3D model. I don't have a walkthrough video, let alone different videos; and I believe reconstruction techniques like these work on such sparse inputs. What's my best approach here? I have partial floorplans, I made a reasonable SketchUp model (I counted bricks in the pictures to get accurate measurements) but I'm nowhere near my goal, which would be having a complete, photorealistic 3D model I can drop load in Unreal or Unity and make a VR walkthrough (I can only imagine how that would blow my mum's mind!) I've thought of outsourcing this, e.g. find a talented game environment artist or architectural modeller and pay them to do this, but I fear the result will be "off". I dream of the perfect algorithm that will do this. Any ideas?

jobigoud 7y ago
I think there aren't any algorithm that can do that and won't be for any time soon. I spent several weeks last year recreating my backyard in VR, aiming at 1:1 mapping, using photogrammetry for capture and a standalone mobile headset that allow walking around untethered (Lenovo Mirage Solo). Even with photogrammetry it wasn't as accurate as required for the objects to be placed exactly at the correct position and be able to confidently walk around without fear of hitting a tree that is 20 cm more to the side in VR than in reality. It took many back and forth to get the alignment right. By the time I was done the tomatoes had grown so much the VR environment was already wrong. Outsourcing reconstruction of lost/old places is a great idea. I found that even just stereo-panoramas of old apartments seen in VR (so without positional tracking) is triggering memories in a way that simple photos can't.
- mLuby 7y ago
  It's definitely possible, if the photos are suitable. There was a burst of interest over a decade ago specifically around reconstructing a 3D model of Notre Dame from public tourist photos, so that might be something to look into. Here's an example: https://www.youtube.com/watch?v=8rG0t1hS1ms https://www.youtube.com/watch?v=8rG0t1hS1ms
  - tralarpa 7y ago
    I am wondering what has happened (or is currently happening) to this project: https://www.youtube.com/watch?v=Ur1Z72_LyTM https://www.youtube.com/watch?v=Ur1Z72_LyTM
    - ggambetta 7y ago
      OMG, that's incredible. I wanted to do something similar for my project - whenever you stepped in a location where a picture was taken, the picture would fade in, superimposed to the VR scene.
DEADBEEFC0FFEE 7y ago
Thats a fascinating project. If you think the SketchUp mod is pretty good for the dimensions, and you have a lot of photos, isn't the next task the mapping. Take the images that show the most structure, and mark points on the picture and map to points on the model. Time consuming but if you chip away at it.... Or move about in the model, and position 2d image and camera so it aligns, and project/paint the image. I'm just making stuff up.
mrfusion 7y ago
I’m interested in this too! If you do outsource maybe we could join forces and offer it as a service. It seems like there would be some good demand and folks would pay a lot for nostalgia (eg rosebud) Also if this was a reasonably popular service you’d end up with (1000s?) Of input photos with their mapped recreated environments. Perhaps that could serve as the training data for an algorithm.
GistNoesis 7y ago
Have you looked at Meshroom ? If you have enough photos maybe you can select a subset which are similar enough to get the algorithm working. It's a hot research subject combining deep learning and SLAM, to be able to estimate the locations and have a neural network hallucinate the view from a specific position. There will probably be some turn-key solution with-in a few years. But for the virtual visit you may not want no solve the full hard problem of reconstructing the 3d point cloud, if you have the position and orientation of the pictures (which you can position manually), you can have a rendering like in street-view or those point-and-click games, where you jump, to the closest picture in the right orientation. It will also have the merit of not suppressing the foreground of the pictures which in a certain sense is kind of the heart of the memory lane.

jobigoud 7y ago

I think there aren't any algorithm that can do that and won't be for any time soon. I spent several weeks last year recreating my backyard in VR, aiming at 1:1 mapping, using photogrammetry for capture and a standalone mobile headset that allow walking around untethered (Lenovo Mirage Solo). Even with photogrammetry it wasn't as accurate as required for the objects to be placed exactly at the correct position and be able to confidently walk around without fear of hitting a tree that is 20 cm more to the side in VR than in reality. It took many back and forth to get the alignment right. By the time I was done the tomatoes had grown so much the VR environment was already wrong. Outsourcing reconstruction of lost/old places is a great idea. I found that even just stereo-panoramas of old apartments seen in VR (so without positional tracking) is triggering memories in a way that simple photos can't.

mLuby 7y ago
It's definitely possible, if the photos are suitable. There was a burst of interest over a decade ago specifically around reconstructing a 3D model of Notre Dame from public tourist photos, so that might be something to look into. Here's an example: https://www.youtube.com/watch?v=8rG0t1hS1ms https://www.youtube.com/watch?v=8rG0t1hS1ms
- tralarpa 7y ago
  I am wondering what has happened (or is currently happening) to this project: https://www.youtube.com/watch?v=Ur1Z72_LyTM https://www.youtube.com/watch?v=Ur1Z72_LyTM
  - ggambetta 7y ago
    OMG, that's incredible. I wanted to do something similar for my project - whenever you stepped in a location where a picture was taken, the picture would fade in, superimposed to the VR scene.

DEADBEEFC0FFEE 7y ago

Thats a fascinating project. If you think the SketchUp mod is pretty good for the dimensions, and you have a lot of photos, isn't the next task the mapping. Take the images that show the most structure, and mark points on the picture and map to points on the model. Time consuming but if you chip away at it.... Or move about in the model, and position 2d image and camera so it aligns, and project/paint the image. I'm just making stuff up.

mrfusion 7y ago

I’m interested in this too! If you do outsource maybe we could join forces and offer it as a service. It seems like there would be some good demand and folks would pay a lot for nostalgia (eg rosebud) Also if this was a reasonably popular service you’d end up with (1000s?) Of input photos with their mapped recreated environments. Perhaps that could serve as the training data for an algorithm.

GistNoesis 7y ago

Have you looked at Meshroom ? If you have enough photos maybe you can select a subset which are similar enough to get the algorithm working. It's a hot research subject combining deep learning and SLAM, to be able to estimate the locations and have a neural network hallucinate the view from a specific position. There will probably be some turn-key solution with-in a few years. But for the virtual visit you may not want no solve the full hard problem of reconstructing the 3d point cloud, if you have the position and orientation of the pictures (which you can position manually), you can have a rendering like in street-view or those point-and-click games, where you jump, to the closest picture in the right orientation. It will also have the merit of not suppressing the foreground of the pictures which in a certain sense is kind of the heart of the memory lane.