5 ms·

Question related to 3D mesh models in general: has any significant work been done on models oriented towards photogrammetry? Case in point, I have a series of

by geuis 2y ago

Question related to 3D mesh models in general: has any significant work been done on models oriented towards photogrammetry? Case in point, I have a series of photos (48) that capture a small statue. The photos are high quality, the object was on a rotating platform. Lighting is consistent. The background is solid black. These normally are ideal variables for photogrammetry but none of the various common applications and websites do a very good job creating a mesh out of it that isn't super low poly and/or full of holes. I've been casually scanning huggingface for relevant models to try out but haven't really found anything.

tzumby 2y ago
I’m not an expert, only dabbled in photogrammetry, but it seems to me that the crux of that problem is identifying common pixels across images in order to sort of triangulate a point in the 3D space. It doesn’t sound like something an LLM would be good at.
jocaal 2y ago
Recently, a lot of development in this area has been in gaussian splatting and from what I have seen, the new methods are super effective. https://en.wikipedia.org/wiki/Gaussian_splatting https://en.wikipedia.org/wiki/Gaussian_splatting https://www.youtube.com/watch?v=6dPBaV6M9u4 https://www.youtube.com/watch?v=6dPBaV6M9u4
- geuis 2y ago
  Yeah some very impressive stuff with splats going on. But I haven't seen much about going from splats to high quality 3D meshes. I've tried one or two with pretty poor results.
  - jiggawatts 2y ago
    There have been a few papers published on the topic, but it's "early days". Expect a lot of progress over the next couple of years.
    - SR2Z 2y ago
      Splats to meshes is an isosurface extraction problem, fundamentally. It's one of the great unsolved problems of computer graphics and having a good general algorithm would have massive ripple effects for any problem involving meshes. It's a rabbit hole and I only really understood it when I realized that the minimum time between any GH committer's hobby example and implementing the 2003 state-of-the-art is ~4 years. Fingers crossed that Gaussian Splatting makes the rewards high enough that resources get poured on this.
- meindnoch 2y ago
  The parent explicitly asked for a mesh.
  - jocaal 2y ago
    The second link I posted contains n flow from splats to meshes
  - andybak 2y ago
    You can never be sure what someone's real intent is. They might mean "something meshlike". Personally I usually reply by asking for more info (I always have the XY Problem in my mind) but that is time consuming and some people assume you're being pendantic (I am however correct more often than not - people have posed the wrong question or haven't given critical parts of the context)
    - geuis 2y ago
      Yeah, I am explicitly asking about meshes, which is why I said that and also referenced photogrammetry. Sometimes people know what they're asking for help with. Thanks for the links. Going to check them out this morning.
      - andybak 2y ago
        Just to be clear I wasn't singling you out. I don't know anything about you. And furthermore I also often post questions that lack sufficient context. My point was that it's always okay to ask for clarification or to assume that there was maybe some broader contact context or to offer a suggestion that doesn't follow the most literal interpretation of the question as asked.
falloon 2y ago
Kiri engine is pretty easy to use and just released a good update for their 3DGS pipeline, and they have one of the better 3DGS to mesh options. https://kiri-innovation.github.io/3DGStoMesh2/ https://kiri-innovation.github.io/3DGStoMesh2/
troymc 2y ago
Check out RealityCapture [1]. I think it's what's used to create the Quixel Megascans [2]. (They're both under the Epic corporate umbrella now.) [1] https://www.capturingreality.com/realitycapture https://www.capturingreality.com/realitycapture [2] https://quixel.com/megascans/ https://quixel.com/megascans/
Joel_Mckay 2y ago
COLMAP + CloudCompare with a good CUDA GPU (more VRAM is better) card will give reasonable results for large textured objects like buildings. Glass/Water/Mirror/Gloss will need coated to scan, dry spray on Dr.scholls foot deodorant seems to work fine for our object scans. There are now more advanced options than Gaussian splatting, and these can achieve normal playback speeds rather than hours of filtering. I'll drop a citation if I recall the recent paper and example code. However, note this style of 3D scene recovery tends to be heavily 3D location dependent. Best of luck, =3
Elucalidavah 2y ago
> the object was on a rotating platform Isn't a static-object-rotating-camera basically a requirement for photogrammetry?
- Mashimo 2y ago
  Why would that make a difference?
  - SequoiaHope 2y ago
    Photogrammetry generally assumes a fully static scene. If there are static parts of the scene which the camera can see and also rotating parts, the algorithm may struggle to properly match features between images.
    - Mashimo 2y ago
      i think it's common to have dots on the rotating disk where the object is placed on.
      - SequoiaHope 2y ago
        Sure, but if the background has a lot of features it will still confuse the algorithm unless it has some special settings for ignoring the background.
  - deleted 2y ago
    [deleted]
  - addandsubtract 2y ago
    The OC mentioned "static lighting". If they meant static, while the platform was spinning, then the lighting would be inconsistent, because the object would change lighting with each photo. You would have to fix the lighting to the platform to spin with the object, while taking the pictures to get consistent lighting.
    - 2y ago
archerx 2y ago
>The background is solid black. >These normally are ideal variables for photogrammetry Actually no, my friend learned this the hard way during a photogrammetry project, he rented a photo studio, and made sure the background were perfectly black and took the photos but the photogrammetry program (Meshroom I think) was struggling to reconstruct the mesh. I did some research and I learned that it uses features in the background to help position itself to make the meshes. So he redid his tests outside with "messy" backgrounds and it worked much much better. This was a few years ago so I don't know if things are different now.
Broussebar 2y ago
For this exact use case I used instant-ngp[0] recently and was really pleased with the results. There's an article[1] explaining how to prepare your data. [0] https://github.com/NVlabs/instant-ngp https://github.com/NVlabs/instant-ngp [1] https://github.com/NVlabs/instant-ngp/blob/master/docs/nerf_dataset_tips.md https://github.com/NVlabs/instant-ngp/blob/master/docs/nerf_...
GistNoesis 2y ago
>full of holes On the geometry side from the theoretical point of view you can repair meshes, [1], by inferring a signed or unsigned distance field from your existing mesh, then you contour this distance field. If you like the distance field approach, there are also research work [2], to estimate neural unsigned distance fields directly, (kind of a similar way as Gaussian splats). [1] https://github.com/nzfeng/signed-heat-3d https://github.com/nzfeng/signed-heat-3d [it works but it's research code, so buggy, not user friendly, and mostly on toy problems because complexity explode very quickly when using a grid the number of cells grows as a n^3, and then they solve a sparse linear system on top (so total complexity bounded by n^6), but tolerating approximations and writing things properly practical complexity should be on par with methods like finite element method in Computational Fluid Dynamics. [2] https://virtualhumans.mpi-inf.mpg.de/ndf/ https://virtualhumans.mpi-inf.mpg.de/ndf/
MrSkelter 2y ago
48 images is an incredibly small number for high quality photogrammetry. 480 wouldn’t be overkill. A couple of hundred would be considered normal.

by geuis 2y ago

tzumby 2y ago

I’m not an expert, only dabbled in photogrammetry, but it seems to me that the crux of that problem is identifying common pixels across images in order to sort of triangulate a point in the 3D space. It doesn’t sound like something an LLM would be good at.

jocaal 2y ago

Recently, a lot of development in this area has been in gaussian splatting and from what I have seen, the new methods are super effective. https://en.wikipedia.org/wiki/Gaussian_splatting https://en.wikipedia.org/wiki/Gaussian_splatting https://www.youtube.com/watch?v=6dPBaV6M9u4 https://www.youtube.com/watch?v=6dPBaV6M9u4

geuis 2y ago
Yeah some very impressive stuff with splats going on. But I haven't seen much about going from splats to high quality 3D meshes. I've tried one or two with pretty poor results.
- jiggawatts 2y ago
  There have been a few papers published on the topic, but it's "early days". Expect a lot of progress over the next couple of years.
  - SR2Z 2y ago
    Splats to meshes is an isosurface extraction problem, fundamentally. It's one of the great unsolved problems of computer graphics and having a good general algorithm would have massive ripple effects for any problem involving meshes. It's a rabbit hole and I only really understood it when I realized that the minimum time between any GH committer's hobby example and implementing the 2003 state-of-the-art is ~4 years. Fingers crossed that Gaussian Splatting makes the rewards high enough that resources get poured on this.
meindnoch 2y ago
The parent explicitly asked for a mesh.
- jocaal 2y ago
  The second link I posted contains n flow from splats to meshes
- andybak 2y ago
  You can never be sure what someone's real intent is. They might mean "something meshlike". Personally I usually reply by asking for more info (I always have the XY Problem in my mind) but that is time consuming and some people assume you're being pendantic (I am however correct more often than not - people have posed the wrong question or haven't given critical parts of the context)
  - geuis 2y ago
    Yeah, I am explicitly asking about meshes, which is why I said that and also referenced photogrammetry. Sometimes people know what they're asking for help with. Thanks for the links. Going to check them out this morning.
    - andybak 2y ago
      Just to be clear I wasn't singling you out. I don't know anything about you. And furthermore I also often post questions that lack sufficient context. My point was that it's always okay to ask for clarification or to assume that there was maybe some broader contact context or to offer a suggestion that doesn't follow the most literal interpretation of the question as asked.

falloon 2y ago

Kiri engine is pretty easy to use and just released a good update for their 3DGS pipeline, and they have one of the better 3DGS to mesh options. https://kiri-innovation.github.io/3DGStoMesh2/ https://kiri-innovation.github.io/3DGStoMesh2/

troymc 2y ago

Check out RealityCapture [1]. I think it's what's used to create the Quixel Megascans [2]. (They're both under the Epic corporate umbrella now.) [1] https://www.capturingreality.com/realitycapture https://www.capturingreality.com/realitycapture [2] https://quixel.com/megascans/ https://quixel.com/megascans/

Joel_Mckay 2y ago

COLMAP + CloudCompare with a good CUDA GPU (more VRAM is better) card will give reasonable results for large textured objects like buildings. Glass/Water/Mirror/Gloss will need coated to scan, dry spray on Dr.scholls foot deodorant seems to work fine for our object scans. There are now more advanced options than Gaussian splatting, and these can achieve normal playback speeds rather than hours of filtering. I'll drop a citation if I recall the recent paper and example code. However, note this style of 3D scene recovery tends to be heavily 3D location dependent. Best of luck, =3

Elucalidavah 2y ago

> the object was on a rotating platform Isn't a static-object-rotating-camera basically a requirement for photogrammetry?

Mashimo 2y ago
Why would that make a difference?
- SequoiaHope 2y ago
  Photogrammetry generally assumes a fully static scene. If there are static parts of the scene which the camera can see and also rotating parts, the algorithm may struggle to properly match features between images.
  - Mashimo 2y ago
    i think it's common to have dots on the rotating disk where the object is placed on.
    - SequoiaHope 2y ago
      Sure, but if the background has a lot of features it will still confuse the algorithm unless it has some special settings for ignoring the background.
- deleted 2y ago
  [deleted]
- addandsubtract 2y ago
  The OC mentioned "static lighting". If they meant static, while the platform was spinning, then the lighting would be inconsistent, because the object would change lighting with each photo. You would have to fix the lighting to the platform to spin with the object, while taking the pictures to get consistent lighting.
  - 2y ago

archerx 2y ago

>The background is solid black. >These normally are ideal variables for photogrammetry Actually no, my friend learned this the hard way during a photogrammetry project, he rented a photo studio, and made sure the background were perfectly black and took the photos but the photogrammetry program (Meshroom I think) was struggling to reconstruct the mesh. I did some research and I learned that it uses features in the background to help position itself to make the meshes. So he redid his tests outside with "messy" backgrounds and it worked much much better. This was a few years ago so I don't know if things are different now.

Broussebar 2y ago

For this exact use case I used instant-ngp[0] recently and was really pleased with the results. There's an article[1] explaining how to prepare your data. [0] https://github.com/NVlabs/instant-ngp https://github.com/NVlabs/instant-ngp [1] https://github.com/NVlabs/instant-ngp/blob/master/docs/nerf_dataset_tips.md https://github.com/NVlabs/instant-ngp/blob/master/docs/nerf_...

GistNoesis 2y ago

>full of holes On the geometry side from the theoretical point of view you can repair meshes, [1], by inferring a signed or unsigned distance field from your existing mesh, then you contour this distance field. If you like the distance field approach, there are also research work [2], to estimate neural unsigned distance fields directly, (kind of a similar way as Gaussian splats). [1] https://github.com/nzfeng/signed-heat-3d https://github.com/nzfeng/signed-heat-3d [it works but it's research code, so buggy, not user friendly, and mostly on toy problems because complexity explode very quickly when using a grid the number of cells grows as a n^3, and then they solve a sparse linear system on top (so total complexity bounded by n^6), but tolerating approximations and writing things properly practical complexity should be on par with methods like finite element method in Computational Fluid Dynamics. [2] https://virtualhumans.mpi-inf.mpg.de/ndf/ https://virtualhumans.mpi-inf.mpg.de/ndf/

MrSkelter 2y ago

48 images is an incredibly small number for high quality photogrammetry. 480 wouldn’t be overkill. A couple of hundred would be considered normal.