6 ms·

Software converts 360 video into 3D model for VR

110 points by sbarre 9y ago

mh2292 9y ago
This is nothing new- Google's "Cardboard Camera" app has been around for a few years and does almost exactly this same thing. https://play.google.com/store/apps/details?id=com.google.vr.cyclops&hl=en https://play.google.com/store/apps/details?id=com.google.vr....
- throwaway2016a 9y ago
  This looks like it is actually creating a 3d model. The app you linked to I think just stitches together photos into something closer to street view. It's not actually 3d, it just looks 3d because you have enough photos to cover the whole field of vision.
  - nocut12 9y ago
    They don't do a 3d model, but it is a real stereo image. There's a bit more info in their docs: https://developers.google.com/vr/concepts/cardboard-camera-vr-photo-format#gimage https://developers.google.com/vr/concepts/cardboard-camera-v...
    - DiThi 9y ago
      > it is a real stereo image Until you look down. Or until you have a different eye distance than average. Or you try to move a bit and there are things close enough to make you dizzy. Cardboard didn't need any of those fancy things, of course. It was bad enough with abysmal latency.
qume 9y ago
This would have been news in 2005
- sbarre 9y ago
  Do you have a source for something similar from back then? Honest question because I keep up with this field and this seemed pretty novel to me (at least the quasi-DIY aspect of it)
  - willvarfar 9y ago
    The underlying problem is termed "motion matching", "motion tracking" or "camera tracking". I played around with it around 2009 or so and at that time there was scant open-source libs for bits and nothing to join it together and nothing that worked particularly well; it was the domain of high-quality high-price niche products aimed at Hollywood which you could watch product videos of on YouTube but not actually afford to use. I don't know how accessible it is now; maybe there are now working quality open-source libs and we can all start putting it into raspberry pi robots? ;)
    - danellis 9y ago
      You're talking about something different. This is a lot more than motion tracking. This is creating 3D models, not just a 3D path.
      - willvarfar 9y ago
        True! Creating a point cloud is the first step. The next step is to turn it into a mesh with faces.
  - qume 9y ago
    Snavely released bundler 9 years ago: http://www.cs.cornell.edu/~snavely/bundler/ http://www.cs.cornell.edu/~snavely/bundler/ There were a bunch of other papers around the time. I was downvoted a bunch here - which is odd as this really would have been news in 2005 when this would have been considered state of the art. Right now there is nothing here which wasn't published already a decade ago. Disclosure: I've been working on structure from motion software for the last decade.
cwe 9y ago
Reminds me of parts of photosynth[1], glad to see people are still working on this kind of thing. Was very disappointed that photosynth didn't really pan out with building 3d models off the images, they went with just panoramas. [1] https://en.wikipedia.org/wiki/Photosynth https://en.wikipedia.org/wiki/Photosynth
- greggman 9y ago
  I guess not quite the same but the majority of 3d models in Google maps 3d mode are autogenerated
PeachPlum 9y ago
Try Autodesk's Recap https://www.autodesk.com/products/recap/overview https://www.autodesk.com/products/recap/overview
- sbarre 9y ago
  This seems to require some serious non-hobbyist hardware to generate the data needed to build the models, but the results are more impressive than this demo for sure!
  - PeachPlum 9y ago
    We've made models using a cell phone camera
    - sbarre 9y ago
      Interesting! I'd love to know more about that..
      - PeachPlum 9y ago
        I'll try and post some images tomorrow, my colleague has them
GoToRO 9y ago
Very nice. It needs two 360 videos.
nkurz 9y ago
Nowadays, it is possible to create a 3D model of a space by taking hundreds of photographs of a space and using computers to analyze those photos through photogrammetry and computer vision methods. This is a laborious process, and is especially time-consuming and resource-intensive for large scenes. I was recently trying to understand what the state of the art was here, and was surprised to learn that this is true: most 3D reconstruction fares better with a smaller number of high-resolution still photos than with a larger number of lower resolution "stills" extracted from video. I'm still somewhat confused why this is the case. My limited understanding is that main advantage of working with stills is that they more commonly contain location tagging, while frames from video do not. Compared to the base level of difficulty of building a accurate 3D model from lossy 2D sources, figuring out the trajectory of camera doesn't seem too hard. Once one has this, wouldn't the video be just as easier to work with? Trying to figure this out this discrepancy, I got the impression that many researchers may actually be trying to solve the harder problem of trying to compute the camera trajectory in real time, as might be needed for a self-driving vehicle: http://webdiis.unizar.es/~raulmur/orbslam/ http://webdiis.unizar.es/~raulmur/orbslam/. This indeed does seem harder, but still leaves me wondering why a multipass approach wouldn't be feasible. What am I missing? Why can't one do a first pass to calculate camera trajectory, possibly a second pass to combine temporally adjacent frames for greater resolution, then create a better model from the resulting wealth of data? Alternatively stated, why does the quality of the 3D model seem to depend more on the resolution and quality of the input 2D images rather than on the number of these images?
- ndh2 9y ago
  I've been wondering about this as well. Why are they using so few images? To me that is like reverse blinking: Instead of keeping your eyes open, and blinking for a short amount of time every now and then, you're walking around with your eyes closed, and open them only for a short time. I believe the main problem is knowing which data to trust and which data to discard. Reflections (water, shiny surfaces), moving objects (leaves in the wind), over-exposure, and lens flares are already pretty hard to deal with. But with low resolution data, you make it even more difficult because there's more data to discard for being too inaccurate.
mcoliver 9y ago
Check out Reality Capture by Capturing Reality (yeah..really..they reversed the name of the company and the software). https://www.capturingreality.com https://www.capturingreality.com We have used this in conjunction with screen captures from google earth to regenerate environments
drcross 9y ago
Things like this give hefty credence to simulation theory.
duiker101 9y ago
This is really nice but PLEASE don't put an image of a youtube video with the play button at the beginning of the article just to post the video later... it took me way too long to realize I was clicking on an image.
- residude 9y ago
  I clicked 15 times until using inspect element and realized no link in it.

110 points by sbarre 9y ago

This is nothing new- Google's "Cardboard Camera" app has been around for a few years and does almost exactly this same thing. https://play.google.com/store/apps/details?id=com.google.vr.cyclops&hl=en https://play.google.com/store/apps/details?id=com.google.vr....

throwaway2016a 9y ago
This looks like it is actually creating a 3d model. The app you linked to I think just stitches together photos into something closer to street view. It's not actually 3d, it just looks 3d because you have enough photos to cover the whole field of vision.
- nocut12 9y ago
  They don't do a 3d model, but it is a real stereo image. There's a bit more info in their docs: https://developers.google.com/vr/concepts/cardboard-camera-vr-photo-format#gimage https://developers.google.com/vr/concepts/cardboard-camera-v...
  - DiThi 9y ago
    > it is a real stereo image Until you look down. Or until you have a different eye distance than average. Or you try to move a bit and there are things close enough to make you dizzy. Cardboard didn't need any of those fancy things, of course. It was bad enough with abysmal latency.

qume 9y ago

This would have been news in 2005

sbarre 9y ago
Do you have a source for something similar from back then? Honest question because I keep up with this field and this seemed pretty novel to me (at least the quasi-DIY aspect of it)
- willvarfar 9y ago
  The underlying problem is termed "motion matching", "motion tracking" or "camera tracking". I played around with it around 2009 or so and at that time there was scant open-source libs for bits and nothing to join it together and nothing that worked particularly well; it was the domain of high-quality high-price niche products aimed at Hollywood which you could watch product videos of on YouTube but not actually afford to use. I don't know how accessible it is now; maybe there are now working quality open-source libs and we can all start putting it into raspberry pi robots? ;)
  - danellis 9y ago
    You're talking about something different. This is a lot more than motion tracking. This is creating 3D models, not just a 3D path.
    - willvarfar 9y ago
      True! Creating a point cloud is the first step. The next step is to turn it into a mesh with faces.
- qume 9y ago
  Snavely released bundler 9 years ago: http://www.cs.cornell.edu/~snavely/bundler/ http://www.cs.cornell.edu/~snavely/bundler/ There were a bunch of other papers around the time. I was downvoted a bunch here - which is odd as this really would have been news in 2005 when this would have been considered state of the art. Right now there is nothing here which wasn't published already a decade ago. Disclosure: I've been working on structure from motion software for the last decade.

cwe 9y ago

Reminds me of parts of photosynth[1], glad to see people are still working on this kind of thing. Was very disappointed that photosynth didn't really pan out with building 3d models off the images, they went with just panoramas. [1] https://en.wikipedia.org/wiki/Photosynth https://en.wikipedia.org/wiki/Photosynth

greggman 9y ago
I guess not quite the same but the majority of 3d models in Google maps 3d mode are autogenerated

PeachPlum 9y ago

Try Autodesk's Recap https://www.autodesk.com/products/recap/overview https://www.autodesk.com/products/recap/overview

sbarre 9y ago
This seems to require some serious non-hobbyist hardware to generate the data needed to build the models, but the results are more impressive than this demo for sure!
- PeachPlum 9y ago
  We've made models using a cell phone camera
  - sbarre 9y ago
    Interesting! I'd love to know more about that..
    - PeachPlum 9y ago
      I'll try and post some images tomorrow, my colleague has them

GoToRO 9y ago

Very nice. It needs two 360 videos.

nkurz 9y ago

Nowadays, it is possible to create a 3D model of a space by taking hundreds of photographs of a space and using computers to analyze those photos through photogrammetry and computer vision methods. This is a laborious process, and is especially time-consuming and resource-intensive for large scenes. I was recently trying to understand what the state of the art was here, and was surprised to learn that this is true: most 3D reconstruction fares better with a smaller number of high-resolution still photos than with a larger number of lower resolution "stills" extracted from video. I'm still somewhat confused why this is the case. My limited understanding is that main advantage of working with stills is that they more commonly contain location tagging, while frames from video do not. Compared to the base level of difficulty of building a accurate 3D model from lossy 2D sources, figuring out the trajectory of camera doesn't seem too hard. Once one has this, wouldn't the video be just as easier to work with? Trying to figure this out this discrepancy, I got the impression that many researchers may actually be trying to solve the harder problem of trying to compute the camera trajectory in real time, as might be needed for a self-driving vehicle: http://webdiis.unizar.es/~raulmur/orbslam/ http://webdiis.unizar.es/~raulmur/orbslam/. This indeed does seem harder, but still leaves me wondering why a multipass approach wouldn't be feasible. What am I missing? Why can't one do a first pass to calculate camera trajectory, possibly a second pass to combine temporally adjacent frames for greater resolution, then create a better model from the resulting wealth of data? Alternatively stated, why does the quality of the 3D model seem to depend more on the resolution and quality of the input 2D images rather than on the number of these images?

ndh2 9y ago
I've been wondering about this as well. Why are they using so few images? To me that is like reverse blinking: Instead of keeping your eyes open, and blinking for a short amount of time every now and then, you're walking around with your eyes closed, and open them only for a short time. I believe the main problem is knowing which data to trust and which data to discard. Reflections (water, shiny surfaces), moving objects (leaves in the wind), over-exposure, and lens flares are already pretty hard to deal with. But with low resolution data, you make it even more difficult because there's more data to discard for being too inaccurate.

mcoliver 9y ago

Check out Reality Capture by Capturing Reality (yeah..really..they reversed the name of the company and the software). https://www.capturingreality.com https://www.capturingreality.com We have used this in conjunction with screen captures from google earth to regenerate environments

drcross 9y ago

Things like this give hefty credence to simulation theory.

duiker101 9y ago

This is really nice but PLEASE don't put an image of a youtube video with the play button at the beginning of the article just to post the video later... it took me way too long to realize I was clicking on an image.

residude 9y ago
I clicked 15 times until using inspect element and realized no link in it.