5 ms·

Denser 3D Point Clouds in OpenSfM

117 points by yubin 10y ago

Pica_soO 10y ago
Do they use the same NeuralNets that are used to improve videos on these pictures to reconstruct the details?
yummybear 10y ago
I wonder if you wouldn't get cleaner results if you eliminated points that don't have an adjacent point nearer than a certain distance?
- yubin 10y ago
  Yes. We can apply different heuristics to remove outlier points. One interesting way to do it will be to train a neural network to refine the dense point cloud.
zubairq 10y ago
Pretty amazing stuff!!
washwash 10y ago
great
eriknstr 10y ago
Never heard of OpenSfM before but have been looking for something like it.
- spacemanmatt 10y ago
  I have personal and professional projects that involve mundane data feeding and service-building around this core capability. This totally fills a niche that's been on my tech radar for a while.
dharma1 10y ago
very cool. that's quite dense. So the segmentation convnet is applied on the 2d image pixels before SfM, rather than on the point cloud data? I've been looking for neural net semantic segmentation for 3D point cloud data
- yubin 10y ago
  Exactly, the convnet is applied on 2d pixels instead of on 3D point clouds. If one have labelled 3D points (e.g. if a point is part of a car or not), one can train a network for recognition directly on 3D points.
  - dharma1 10y ago
    Nice. I wonder if this could be used for robot navigation. The segmented point clouds especially are useful. Been looking forward to a DIY Roomba project using point clouds and neural nets for navigation
deckar01 10y ago
I discovered VisualSfM this weekend after trying Adobe ReCap 360. The photo reconstruction functionality of ReCap is cloud based, which was a blocker for me, because I have a beefy graphics rig and want to script it into a pipeline. I am curious how OpenSfM compares to VisualSfM and if they are related in any way. Edit: The VisualSfM license is only open for non-commercial uses. OpenSfM is BSD licensed, so I can safely build my business on top of it.
- yubin 10y ago
  Both VisualSfM and OpenSfM use similar methodology (incremental structure from motion) to reconstruct 3D scene points and camera positions from photos. And yes, you can build your own tools (even commercially) with OpenSfM.
  - deckar01 10y ago
    I am curious how the quality compares and if they have common library dependencies.
- yorwba 10y ago
  I am curious, did you try to build VisualSfM yourself? I had to use it for a project at Uni a few years ago and the build process was quite adventurous, without this[0] guide I would have been completely lost. Would be interesting to know whether it has gotten any easier. [0] http://www.10flow.com/2012/08/15/building-visualsfm-on-ubuntu-12-04-precise-pangolin-desktop-64-bit/ http://www.10flow.com/2012/08/15/building-visualsfm-on-ubunt...
  - deckar01 10y ago
    No, I used the windows binaries during evaluation.
- exthaicitizen 10y ago
  Adobe doesn't make ReCap 360, Autodesk does.
efkv 10y ago
For those who have not tried it, OpenSfM is an amazing tool. Can't wait to play with dense reconstructions. Congrats mapillary team, I know this has been a long time coming!
MichailP 10y ago
Is it possible to use images from say Google Street View to reconstruct 3D city model with this tool? Maybe it could fall into fair use for small part of city...
- maxerickson 10y ago
  It would likely violate their TOS. No creation or augmentation of data sets based on Google’s Content or Services. You will not use Google’s Content or Services to create or augment your own mapping-related dataset (or that of a third party), including a mapping or navigation dataset, business listings database, mailing list, or telemarketing list. There's probably some other issues. https://developers.google.com/maps/terms https://developers.google.com/maps/terms
- rasmi 10y ago
  You would need a significant number of images for a particular scene (more than Google Street View would provide -- maybe in the tens or hundreds depending on the complexity -- but perhaps mining Google Maps photo uploads for a particular location would suffice). A Google Image search for a particular landmark is likely to yield better results ([1], [2], [3]). Also, stitching together multiple scenes after reconstructing each individual scene may prove difficult. Apple Maps does a sort of satellite view projection on to 3D building models ([4]), but this is far from a dense reconstruction. [1]http://grail.cs.washington.edu/projects/timelapse/ http://grail.cs.washington.edu/projects/timelapse/ [2]http://grail.cs.washington.edu/projects/timelapse3d/ http://grail.cs.washington.edu/projects/timelapse3d/ [3]https://grail.cs.washington.edu/rome/ https://grail.cs.washington.edu/rome/ [4]http://www.theverge.com/2015/9/29/9413583/apple-maps-3d-flyover-view-new-cities http://www.theverge.com/2015/9/29/9413583/apple-maps-3d-flyo...
- toomuchtodo 10y ago
  You'll want to use the OpenStreetView dataset for that, which has a much more liberal license for collected images than Mapillary. (OSV is a direct competitor to Mapillary, with OSV being sponsored by Navteq) http://openstreetview.com/terms/ http://openstreetview.com/terms/ You might also consider mining Flickr by geotag, constraining to CC licenses that allow for use.
data0815 10y ago
How well would this work in a situation with artificial (moving) lighting? I have seen reconstructions for wrecks underwater, and we are interested in doing something similar.
- rasmi 10y ago
  It is possible to correct for all sorts of different lighting conditions. See here: http://grail.cs.washington.edu/projects/timelapse/ http://grail.cs.washington.edu/projects/timelapse/
jsch 10y ago
COLMAP is an open source structure-from-motion and multi-view stereo software and is hopefully useful for people here as well: https://colmap.github.io/ https://colmap.github.io/, http://people.inf.ethz.ch/jschoenb/colmap/ http://people.inf.ethz.ch/jschoenb/colmap/
BHSPitMonkey 10y ago
What techniques are available to transform point clouds like these into closed polygonal meshes (like the ones you see in Google Earth where 3D imagery is available)?
zump 10y ago
So why can't this be used instead of LIDAR for Self-Driving cars? Experts?

117 points by yubin 10y ago

Do they use the same NeuralNets that are used to improve videos on these pictures to reconstruct the details?

I wonder if you wouldn't get cleaner results if you eliminated points that don't have an adjacent point nearer than a certain distance?

yubin 10y ago
Yes. We can apply different heuristics to remove outlier points. One interesting way to do it will be to train a neural network to refine the dense point cloud.

zubairq 10y ago

Pretty amazing stuff!!

washwash 10y ago

great

eriknstr 10y ago

Never heard of OpenSfM before but have been looking for something like it.

spacemanmatt 10y ago
I have personal and professional projects that involve mundane data feeding and service-building around this core capability. This totally fills a niche that's been on my tech radar for a while.

dharma1 10y ago

very cool. that's quite dense. So the segmentation convnet is applied on the 2d image pixels before SfM, rather than on the point cloud data? I've been looking for neural net semantic segmentation for 3D point cloud data

yubin 10y ago
Exactly, the convnet is applied on 2d pixels instead of on 3D point clouds. If one have labelled 3D points (e.g. if a point is part of a car or not), one can train a network for recognition directly on 3D points.
- dharma1 10y ago
  Nice. I wonder if this could be used for robot navigation. The segmented point clouds especially are useful. Been looking forward to a DIY Roomba project using point clouds and neural nets for navigation

deckar01 10y ago

I discovered VisualSfM this weekend after trying Adobe ReCap 360. The photo reconstruction functionality of ReCap is cloud based, which was a blocker for me, because I have a beefy graphics rig and want to script it into a pipeline. I am curious how OpenSfM compares to VisualSfM and if they are related in any way. Edit: The VisualSfM license is only open for non-commercial uses. OpenSfM is BSD licensed, so I can safely build my business on top of it.

yubin 10y ago
Both VisualSfM and OpenSfM use similar methodology (incremental structure from motion) to reconstruct 3D scene points and camera positions from photos. And yes, you can build your own tools (even commercially) with OpenSfM.
- deckar01 10y ago
  I am curious how the quality compares and if they have common library dependencies.
yorwba 10y ago
I am curious, did you try to build VisualSfM yourself? I had to use it for a project at Uni a few years ago and the build process was quite adventurous, without this[0] guide I would have been completely lost. Would be interesting to know whether it has gotten any easier. [0] http://www.10flow.com/2012/08/15/building-visualsfm-on-ubuntu-12-04-precise-pangolin-desktop-64-bit/ http://www.10flow.com/2012/08/15/building-visualsfm-on-ubunt...
- deckar01 10y ago
  No, I used the windows binaries during evaluation.
exthaicitizen 10y ago
Adobe doesn't make ReCap 360, Autodesk does.

efkv 10y ago

For those who have not tried it, OpenSfM is an amazing tool. Can't wait to play with dense reconstructions. Congrats mapillary team, I know this has been a long time coming!

MichailP 10y ago

Is it possible to use images from say Google Street View to reconstruct 3D city model with this tool? Maybe it could fall into fair use for small part of city...

maxerickson 10y ago
It would likely violate their TOS. No creation or augmentation of data sets based on Google’s Content or Services. You will not use Google’s Content or Services to create or augment your own mapping-related dataset (or that of a third party), including a mapping or navigation dataset, business listings database, mailing list, or telemarketing list. There's probably some other issues. https://developers.google.com/maps/terms https://developers.google.com/maps/terms
rasmi 10y ago
You would need a significant number of images for a particular scene (more than Google Street View would provide -- maybe in the tens or hundreds depending on the complexity -- but perhaps mining Google Maps photo uploads for a particular location would suffice). A Google Image search for a particular landmark is likely to yield better results ([1], [2], [3]). Also, stitching together multiple scenes after reconstructing each individual scene may prove difficult. Apple Maps does a sort of satellite view projection on to 3D building models ([4]), but this is far from a dense reconstruction. [1]http://grail.cs.washington.edu/projects/timelapse/ http://grail.cs.washington.edu/projects/timelapse/ [2]http://grail.cs.washington.edu/projects/timelapse3d/ http://grail.cs.washington.edu/projects/timelapse3d/ [3]https://grail.cs.washington.edu/rome/ https://grail.cs.washington.edu/rome/ [4]http://www.theverge.com/2015/9/29/9413583/apple-maps-3d-flyover-view-new-cities http://www.theverge.com/2015/9/29/9413583/apple-maps-3d-flyo...
toomuchtodo 10y ago
You'll want to use the OpenStreetView dataset for that, which has a much more liberal license for collected images than Mapillary. (OSV is a direct competitor to Mapillary, with OSV being sponsored by Navteq) http://openstreetview.com/terms/ http://openstreetview.com/terms/ You might also consider mining Flickr by geotag, constraining to CC licenses that allow for use.

data0815 10y ago

How well would this work in a situation with artificial (moving) lighting? I have seen reconstructions for wrecks underwater, and we are interested in doing something similar.

rasmi 10y ago
It is possible to correct for all sorts of different lighting conditions. See here: http://grail.cs.washington.edu/projects/timelapse/ http://grail.cs.washington.edu/projects/timelapse/

jsch 10y ago

COLMAP is an open source structure-from-motion and multi-view stereo software and is hopefully useful for people here as well: https://colmap.github.io/ https://colmap.github.io/, http://people.inf.ethz.ch/jschoenb/colmap/ http://people.inf.ethz.ch/jschoenb/colmap/

BHSPitMonkey 10y ago

What techniques are available to transform point clouds like these into closed polygonal meshes (like the ones you see in Google Earth where 3D imagery is available)?

zump 10y ago

So why can't this be used instead of LIDAR for Self-Driving cars? Experts?