6 ms·

Meshroom: Open Source 3D Reconstruction Software (2018) [video]

262 points by hereiskkb 7y ago

pierrec 7y ago
They link to the full Mikros showreel [1], implying that it's been used in a bunch of blockbusters already. But I couldn't find any information on how it's been used and in which projects, it would be cool to have more details on that. It's always interesting to see the film industry working with open source software, especially in cases like this where production companies are sponsoring development. [1]: https://www.youtube.com/watch?v=pBy1g7gbXCQ https://www.youtube.com/watch?v=pBy1g7gbXCQ
- pfranz 7y ago
  > it's hard to tell how it's been used and in which projects That's always the case in vfx. The best source is contemporaneous interviews directly with the people involved, which is sporadic and very rare. Most demo reels, even for big software packages, are incredibly misleading. Company A might have some partnership with a production studio, slap their name in the credits, use their work in the demo reel, but the output is junk so the artist scraps it and does it the "old fashioned way" from scratch. Or the software gets used in a very specific, tangential way like roughing something out or converting file formats back and forth. VFX companies and software packages are trying to sell themselves for future work. So they tend to gloss over details and present themselves much better than they are in practice. After a show wraps, it's really difficult to get good information. The files on disk may not even open anymore and the artists have moved on.
corysama 7y ago
btw: https://www.reddit.com/r/photogrammetry/ https://www.reddit.com/r/photogrammetry/
hbosch 7y ago
Meshroom has been very fun to work with for what I can only refer to as "closed-shell" or fully convex shapes. I tried my damnedest to collect a good photogrammetry-based model of a flower, and the mesh fidelity I got from my 100 or so photos was pretty terrible... And I think probably because of the leafs and the bowl shape of the petals on top. If I had time, I probably would have separated the flower into distinct pieces (leafs, stem, top of flower) and do more assemblage in my 3D program with discretely generated meshes... However, stones and plant bulbs came out great. Edit: After some of the other links in this discussion, I definitely need to revisit this project! Some people have had great success with foliage and flowers it seems like.
- sdwisely 7y ago
  I tried a wicker basket as a test mesh this week and had a great deal of trouble with the handles. For closed objects though the results are probably better than I've found in commercial software.
sitdownyoungman 7y ago
so...are shiny black objects better scanned into 3d objects now?
carreau 7y ago
My organization recently considered buying more licenses for photoscan/metashape (https://www.agisoft.com/ https://www.agisoft.com/) but it was too expensive. I'm curious as to how it compares (I'm not a user myself so can't judge), and I couldn't find wether this can run on a distributed GPU cluster. If that's the case I would love to push for replacing photoscan by this.
jefft255 7y ago
Really cool, especially now that Autodesk's photogrammetry software is no longer available for free. I think this is a good example of academic/public research resulting in useful free software!
- anjneymidha 7y ago
  +1! There's such a wealth of computer vision approaches of practical value locked up in CVPR and other research papers that could be exposed this way to enthusiasts and semi-pro users
  - jeffchuber 7y ago
    +1 to this comment
sherr 7y ago
A useful addition to that page would be a short summary of what "photogrammetry" is and a description of what this software does. It need only be a couple of sentences. Otherwise, looks very interesting.
- ungzd 7y ago
  Underlying photogrammetry library, https://alicevision.github.io/ https://alicevision.github.io/, has more informative landing page.
- tacotime 7y ago
  It's described right on the page: "Meshroom is a ... 3D Reconstruction Software..." 1 minute into the 6 minute video you can see that the program generates a point cloud from a set of photos.
- mkesper 7y ago
  Yes, the description right now is equivalent to "does magic with photographs". It's a lost opportunity for introducing the uninitiated reader, thereby reducing the chance of them watching the video.
- notthemessiah 7y ago
  If you somehow stumble onto the page and are too lazy to Google the term, I don't think you're in the target audience.
nobbis 7y ago
Another great free alternative is COLMAP (https://demuc.de/colmap https://demuc.de/colmap). And for those interested in doing this on mobile in real-time, we've just released a beta SDK (https://abound.dev https://abound.dev) that lets you add real-time photogrammetry to any iOS app.
- Fission 7y ago
  Very cool stuff! Since you're using volumetric integration/fixed voxel size, one trick to get significantly improved visual texturing quality is to low-pass/smooth the mesh right before texturing.
  - nobbis 7y ago
    Yes, we support Laplacian filtering of the level set prior to meshing in the SDK (plus other mesh smoothing techniques.) There are trade-offs involved.
- obviuosly 7y ago
  How do COLMAP, Meshroom and VisualSFM compare to one another?
  - atkbrah 7y ago
    Some stuff that comes into mind: * COLMAP and Meshroom are able to create a mesh from the pointcloud whereas VisualSFM is not. If you wish to create a mesh out of VisualSFM end result, you could use OpenMVS[1] * COLMAP does not create a texture file for the resulting mesh - instead it uses vertex color for "texturing" * COLMAP and Meshroom licenses permit commercial usage - VisualSFM does not (you need to license it) From these three I've found that COLMAP works the best and VisualSFM comes second. Meshroom has a nice feature that allows you to add images into your model incrementally (I think without complete recompute) but it didn't work that well yet. What comes to open source photogrammetry in general, my opinion is that COLMAP + OpenMVS or OpenMVG[2] + OpenMVS are the tools to go albeit the command line interface with huge amount of possible options can be a bit tricky at first. 1. https://github.com/cdcseacave/openMVS https://github.com/cdcseacave/openMVS 2. https://github.com/openMVG/openMVG https://github.com/openMVG/openMVG
h2odragon 7y ago
Still CUDA only? Didn't see an answer to that in a few minutes of clicking around their site.
- microcolonel 7y ago
  Seems like some of it is still CUDA-only, from the AliceVision INSTALL.md [0]. I have to wonder how AMD is letting NVIDIA win so big in academia, when their GPUs are actually quite competitive. I'm beginning to think that they have some management practice or culture which makes it difficult for them to retain software talent, otherwise I can't really understand how they always seem to have a glaring weakness somewhere that they're not addressing. [0]: https://github.com/alicevision/AliceVision/blob/develop/INSTALL.md https://github.com/alicevision/AliceVision/blob/develop/INST...
  - TomVDB 7y ago
    Nvidia had (has?) CUDA centers of excellence for more than a decade, they gave GPUs to universities, they've had their own compute related conference for years, they have a huge set of libraries. During all the time, AMD did essentially nothing. (They weren't financially in the position to do so.) Then they missed the AI boom. Now they are trying, right at the time where the compute specific performance lead that they once had is gone and when their perf/W is terrible (which is not a good thing for datacenters.) There is really no mystery here.
  - errantspark 7y ago
    I don't think it's AMD's fault really, CUDA just as an insane amount of inertia in the space and there's a whole lot of incentive for anyone to do anything about it. As much as I hate it I've been vendor locked to NV for years and as a result I don't really care anymore. I'll never buy a AMD card because I need to get work done and get paid. It sucks, but not enough for me to go through the pain of trying to work with the far far inferior OpenCL based software for rendering/photogrammetry etc. I don't even touch ML, but I assume the story is the same there. What's the value proposition for writing software that works with OpenCL when your entire market already has NV cards? The price differential is tiny too, maybe if AMD came down like 30% for the equivalent computational power.

262 points by hereiskkb 7y ago

They link to the full Mikros showreel [1], implying that it's been used in a bunch of blockbusters already. But I couldn't find any information on how it's been used and in which projects, it would be cool to have more details on that. It's always interesting to see the film industry working with open source software, especially in cases like this where production companies are sponsoring development. [1]: https://www.youtube.com/watch?v=pBy1g7gbXCQ https://www.youtube.com/watch?v=pBy1g7gbXCQ

pfranz 7y ago
> it's hard to tell how it's been used and in which projects That's always the case in vfx. The best source is contemporaneous interviews directly with the people involved, which is sporadic and very rare. Most demo reels, even for big software packages, are incredibly misleading. Company A might have some partnership with a production studio, slap their name in the credits, use their work in the demo reel, but the output is junk so the artist scraps it and does it the "old fashioned way" from scratch. Or the software gets used in a very specific, tangential way like roughing something out or converting file formats back and forth. VFX companies and software packages are trying to sell themselves for future work. So they tend to gloss over details and present themselves much better than they are in practice. After a show wraps, it's really difficult to get good information. The files on disk may not even open anymore and the artists have moved on.

corysama 7y ago

btw: https://www.reddit.com/r/photogrammetry/ https://www.reddit.com/r/photogrammetry/

hbosch 7y ago

Meshroom has been very fun to work with for what I can only refer to as "closed-shell" or fully convex shapes. I tried my damnedest to collect a good photogrammetry-based model of a flower, and the mesh fidelity I got from my 100 or so photos was pretty terrible... And I think probably because of the leafs and the bowl shape of the petals on top. If I had time, I probably would have separated the flower into distinct pieces (leafs, stem, top of flower) and do more assemblage in my 3D program with discretely generated meshes... However, stones and plant bulbs came out great. Edit: After some of the other links in this discussion, I definitely need to revisit this project! Some people have had great success with foliage and flowers it seems like.

sdwisely 7y ago
I tried a wicker basket as a test mesh this week and had a great deal of trouble with the handles. For closed objects though the results are probably better than I've found in commercial software.

sitdownyoungman 7y ago

so...are shiny black objects better scanned into 3d objects now?

carreau 7y ago

My organization recently considered buying more licenses for photoscan/metashape (https://www.agisoft.com/ https://www.agisoft.com/) but it was too expensive. I'm curious as to how it compares (I'm not a user myself so can't judge), and I couldn't find wether this can run on a distributed GPU cluster. If that's the case I would love to push for replacing photoscan by this.

jefft255 7y ago

Really cool, especially now that Autodesk's photogrammetry software is no longer available for free. I think this is a good example of academic/public research resulting in useful free software!

anjneymidha 7y ago
+1! There's such a wealth of computer vision approaches of practical value locked up in CVPR and other research papers that could be exposed this way to enthusiasts and semi-pro users
- jeffchuber 7y ago
  +1 to this comment

sherr 7y ago

A useful addition to that page would be a short summary of what "photogrammetry" is and a description of what this software does. It need only be a couple of sentences. Otherwise, looks very interesting.

ungzd 7y ago
Underlying photogrammetry library, https://alicevision.github.io/ https://alicevision.github.io/, has more informative landing page.
tacotime 7y ago
It's described right on the page: "Meshroom is a ... 3D Reconstruction Software..." 1 minute into the 6 minute video you can see that the program generates a point cloud from a set of photos.
mkesper 7y ago
Yes, the description right now is equivalent to "does magic with photographs". It's a lost opportunity for introducing the uninitiated reader, thereby reducing the chance of them watching the video.
notthemessiah 7y ago
If you somehow stumble onto the page and are too lazy to Google the term, I don't think you're in the target audience.

nobbis 7y ago

Another great free alternative is COLMAP (https://demuc.de/colmap https://demuc.de/colmap). And for those interested in doing this on mobile in real-time, we've just released a beta SDK (https://abound.dev https://abound.dev) that lets you add real-time photogrammetry to any iOS app.

Fission 7y ago
Very cool stuff! Since you're using volumetric integration/fixed voxel size, one trick to get significantly improved visual texturing quality is to low-pass/smooth the mesh right before texturing.
- nobbis 7y ago
  Yes, we support Laplacian filtering of the level set prior to meshing in the SDK (plus other mesh smoothing techniques.) There are trade-offs involved.
obviuosly 7y ago
How do COLMAP, Meshroom and VisualSFM compare to one another?
- atkbrah 7y ago
  Some stuff that comes into mind: * COLMAP and Meshroom are able to create a mesh from the pointcloud whereas VisualSFM is not. If you wish to create a mesh out of VisualSFM end result, you could use OpenMVS[1] * COLMAP does not create a texture file for the resulting mesh - instead it uses vertex color for "texturing" * COLMAP and Meshroom licenses permit commercial usage - VisualSFM does not (you need to license it) From these three I've found that COLMAP works the best and VisualSFM comes second. Meshroom has a nice feature that allows you to add images into your model incrementally (I think without complete recompute) but it didn't work that well yet. What comes to open source photogrammetry in general, my opinion is that COLMAP + OpenMVS or OpenMVG[2] + OpenMVS are the tools to go albeit the command line interface with huge amount of possible options can be a bit tricky at first. 1. https://github.com/cdcseacave/openMVS https://github.com/cdcseacave/openMVS 2. https://github.com/openMVG/openMVG https://github.com/openMVG/openMVG

h2odragon 7y ago

Still CUDA only? Didn't see an answer to that in a few minutes of clicking around their site.

microcolonel 7y ago
Seems like some of it is still CUDA-only, from the AliceVision INSTALL.md [0]. I have to wonder how AMD is letting NVIDIA win so big in academia, when their GPUs are actually quite competitive. I'm beginning to think that they have some management practice or culture which makes it difficult for them to retain software talent, otherwise I can't really understand how they always seem to have a glaring weakness somewhere that they're not addressing. [0]: https://github.com/alicevision/AliceVision/blob/develop/INSTALL.md https://github.com/alicevision/AliceVision/blob/develop/INST...
- TomVDB 7y ago
  Nvidia had (has?) CUDA centers of excellence for more than a decade, they gave GPUs to universities, they've had their own compute related conference for years, they have a huge set of libraries. During all the time, AMD did essentially nothing. (They weren't financially in the position to do so.) Then they missed the AI boom. Now they are trying, right at the time where the compute specific performance lead that they once had is gone and when their perf/W is terrible (which is not a good thing for datacenters.) There is really no mystery here.
- errantspark 7y ago
  I don't think it's AMD's fault really, CUDA just as an insane amount of inertia in the space and there's a whole lot of incentive for anyone to do anything about it. As much as I hate it I've been vendor locked to NV for years and as a result I don't really care anymore. I'll never buy a AMD card because I need to get work done and get paid. It sucks, but not enough for me to go through the pain of trying to work with the far far inferior OpenCL based software for rendering/photogrammetry etc. I don't even touch ML, but I assume the story is the same there. What's the value proposition for writing software that works with OpenCL when your entire market already has NV cards? The price differential is tiny too, maybe if AMD came down like 30% for the equivalent computational power.