5 ms·

I wonder how this compares to LGM outputs which also came out recently - https://github.com/3DTopia/LGM https://github.com/3DTopia/LGM - specifically LGM looks

I wonder how this compares to LGM outputs which also came out recently - https://github.com/3DTopia/LGM https://github.com/3DTopia/LGM - specifically LGM looks like it's generating the additional viewpoints from text or an image and then creating a mesh whereas GaussianObject seems to require 4 viewpoints. We're getting closer to workflows that can generate good 3d objects and texture them with diffusion models, there's a bit of work in integrating these into existing tooling though - also, animations and rigging for characters still seems to be a big issue (at least I haven't seen any good 3d generation models that can create rigged characters yet).

jsheard 3y ago
Animation has always been the achilles heel of new 3D representations, the first thing you should ask when someone claims to have replaced triangles is "how do I animate this?". The answer is generally "you can't" and then the technique fades into obscurity. A decade ago it was voxels coming to replace triangle meshes and that went nowhere. It is possible to turn a NeRF into a mesh, but the quality of the meshes produced is currently so poor that it doesn't really get us any closer to being able to animate or edit the model.
- tdudhhu 3y ago
  Depends on how you are going to use the result. This method is great for creating assets. Photogrammetry is already used a lot. The rule for a lot if 3D work is: if it looks good, it is good. But I agree this method won't be suitable for all 3D work.
  - jsheard 3y ago
    Photogrammetry requires a lot of manual cleanup and retopology work, but it's worth the trouble because it captures a lot of extremely fine detail which can be baked down into textures and applied to the manually refined model. It's not really clear to me that NeRFs are producing enough detail to warrant that cleanup effort, rather than modelling something from scratch. I would be interested to see a comparison of conventional SOTA photogrammetry pipelines versus NeRF techniques solving a 3D model from the same set of photos, but these papers only ever seem to compare themselves to other NeRF papers, rather than the software that the industry is already using to solve this exact problem.
    - tdudhhu 3y ago
      Well again: if it looks good it is good. When you are not creating assets for games or close-up details it can still be very usefull to take a couple of pictures of a tree stump and use it in your environment.
      - matt3D 3y ago
        Also photogrammetry from an industry level. I work in construction, we take photos of the building in situ as a lot of the important elements are encased later (eg. Reinforcement inside concrete) This level of 3D NeRF is great for that as it's much easier to parse than a folder full of images.
- regularfry 3y ago
  https://github.com/Anttwo/SuGaR https://github.com/Anttwo/SuGaR looks non-poor to me, but I've not had a chance to play with it properly yet.
  - kevingadd 3y ago
    They don't show any of the topology, UVs, texture maps, skeletons of the extracted meshes, which makes me skeptical about what they're actually doing to animate them, and whether they've gone through the process of making multiple animations for a moderately complex mesh generated from this tool or just have a few toy examples. It's still very impressive research, at least.
    - regularfry 3y ago
      I can well imagine that reliably getting good topology out is a Hard Problem(tm).
- zokier 3y ago
  There is work ongoing to animate gaussians: https://www.youtube.com/watch?v=C4IT1gnkaF0 https://www.youtube.com/watch?v=C4IT1gnkaF0 D3GA: Drivable 3D Gaussian Avatars https://www.youtube.com/watch?v=q5hHZrRUV-k https://www.youtube.com/watch?v=q5hHZrRUV-k Animatable Gaussians: Learning Pose-dependent Gaussian Maps for High-fidelity Human Avatar Modeling
  - jsheard 3y ago
    It's something, but I find it difficult to get excited about "we trained another NN to perform this specific transformation on this specific type of asset". Does it not generalize? What are you supposed to do if the NN doesn't get it right? It's an inscrutable black box on top of a stack of other inscrutable black boxes, which maybe make things easy if you hit the happy path but make it impossible to reason about the problem if it fails. I can't say I'm a fan of replacing well defined systems with a big blob of weights that nobody really understands.
    - kajecounterhack 3y ago
      This is kind of the attitude that folks at companies had last decade (e.g. early 2010s) which prevented the adoption of deep neural networks until people could no longer overlook their efficacy. If the NN doesn't get it right, it's absolutely not the end of the world. First of all, triaging the failures usually turns up something interesting / some connecting thread. And as for fixing issues, your options are typically: 1. Retrain with more examples to help it get it right (easier these days since you can take advantage of synthetic data / self- or semi-supervised approaches). Specifically it will help to generate or find examples that approximate the failures. 2. Shape your loss to penalize undesirable behavior / tendencies you've identified in triage 3. Add inputs or outputs that help your net solve the problem better (e.g. side inputs that may help the net not overfit to its available data) These "inscrutable black boxes," while hard to edit directly, are not impossible to evaluate which means you can just measure to see if they're better than more explicit systems. And most real world systems are patchworks of ML + heuristics to give some desired level of performance. Furthermore there is a big field of folks dedicated to elucidating what your net is learning (e.g. saliency). The tools are pretty cool -- for example in computer vision you can visualize whether your net is overfitting to background data in your training examples. A valid critique of all of this is simply that it's expensive. Very expensive to build teams to warehouse and clean huge amounts of data, then build unbiased / fair model evaluations, then build models, then integrate them into real world stuff, usually to be used in conjunction with non-learned algos. Another valid critique is that many companies have started using models they can't retrain or fix (e.g. off the shelf LLMs). This is dangerous territory since you lose so many of the options I outlined above.
- spookie 3y ago
  Yup, just the fact that topology is often rather poor, you're making it impossible to translate motion capture data for a lot of things. Facial expressions come to mind. I understand the reasons for different representations for reconstruction, but what we really need is to translate that into a mesh that is actually useful. Its great to have something as a reference, don't get me wrong, but we are still far away from even being able to discreetly assign materials to the object's various parts. Another big issue is licensing, most of these are not allowed commercially.
  - fragmede 3y ago
    they're not free for commercial use. If you want to use them commercially, you get in contact with the research group that came up with it and pay them a pile of money to use it in your commercial product. For someone like Pixar, I can see them ponying up the cash for it if they think it'll allow them to save on animator hour costs and/or iterate faster.
- lmpdev 3y ago
  I fear mesh maybe the modelling equivalent of OOP. Not because it is the best paradigm for all situations, but just through inertia of use over decades
- LarsDu88 3y ago
  I do a lot of Blender 3d modeling as part of my hobby gamedev stuff. I think one thing that a lot of folks tend to forget is that quad and tri meshes are only one of SEVERAL 3d modeling techniques. It just so happens that this is the technique that the world landed on in the 90s due to how flexible it is, how performant it is, and algorithms like Catmull-Clark subdivision. Remember, Toy Story 1 actually used NURB surfaces for all the characters! For a Bug's Life and Toy Story 2, they redid all the meshes as catmull-clark subdividable surfaces. And guess who was CEO and founder of Pixar? There also techniques like NURBs, signed distance fields, and metaballs (all stuff you can actually use in Blender) for generating 3d surfaces. And of course the techniques used in CAD software. With the right algorithm and sufficient compute, there could be some sort of completely alternate 3d representation in the future for creating virtual spaces.
  - opyate 3y ago
    This fictional documentary title just popped into my head: "Cloudy with a Chance of Metaballs: A history of 3D modelling in animated film."

jsheard 3y ago

Animation has always been the achilles heel of new 3D representations, the first thing you should ask when someone claims to have replaced triangles is "how do I animate this?". The answer is generally "you can't" and then the technique fades into obscurity. A decade ago it was voxels coming to replace triangle meshes and that went nowhere. It is possible to turn a NeRF into a mesh, but the quality of the meshes produced is currently so poor that it doesn't really get us any closer to being able to animate or edit the model.

tdudhhu 3y ago
Depends on how you are going to use the result. This method is great for creating assets. Photogrammetry is already used a lot. The rule for a lot if 3D work is: if it looks good, it is good. But I agree this method won't be suitable for all 3D work.
- jsheard 3y ago
  Photogrammetry requires a lot of manual cleanup and retopology work, but it's worth the trouble because it captures a lot of extremely fine detail which can be baked down into textures and applied to the manually refined model. It's not really clear to me that NeRFs are producing enough detail to warrant that cleanup effort, rather than modelling something from scratch. I would be interested to see a comparison of conventional SOTA photogrammetry pipelines versus NeRF techniques solving a 3D model from the same set of photos, but these papers only ever seem to compare themselves to other NeRF papers, rather than the software that the industry is already using to solve this exact problem.
  - tdudhhu 3y ago
    Well again: if it looks good it is good. When you are not creating assets for games or close-up details it can still be very usefull to take a couple of pictures of a tree stump and use it in your environment.
    - matt3D 3y ago
      Also photogrammetry from an industry level. I work in construction, we take photos of the building in situ as a lot of the important elements are encased later (eg. Reinforcement inside concrete) This level of 3D NeRF is great for that as it's much easier to parse than a folder full of images.
regularfry 3y ago
https://github.com/Anttwo/SuGaR https://github.com/Anttwo/SuGaR looks non-poor to me, but I've not had a chance to play with it properly yet.
- kevingadd 3y ago
  They don't show any of the topology, UVs, texture maps, skeletons of the extracted meshes, which makes me skeptical about what they're actually doing to animate them, and whether they've gone through the process of making multiple animations for a moderately complex mesh generated from this tool or just have a few toy examples. It's still very impressive research, at least.
  - regularfry 3y ago
    I can well imagine that reliably getting good topology out is a Hard Problem(tm).
zokier 3y ago
There is work ongoing to animate gaussians: https://www.youtube.com/watch?v=C4IT1gnkaF0 https://www.youtube.com/watch?v=C4IT1gnkaF0 D3GA: Drivable 3D Gaussian Avatars https://www.youtube.com/watch?v=q5hHZrRUV-k https://www.youtube.com/watch?v=q5hHZrRUV-k Animatable Gaussians: Learning Pose-dependent Gaussian Maps for High-fidelity Human Avatar Modeling
- jsheard 3y ago
  It's something, but I find it difficult to get excited about "we trained another NN to perform this specific transformation on this specific type of asset". Does it not generalize? What are you supposed to do if the NN doesn't get it right? It's an inscrutable black box on top of a stack of other inscrutable black boxes, which maybe make things easy if you hit the happy path but make it impossible to reason about the problem if it fails. I can't say I'm a fan of replacing well defined systems with a big blob of weights that nobody really understands.
  - kajecounterhack 3y ago
    This is kind of the attitude that folks at companies had last decade (e.g. early 2010s) which prevented the adoption of deep neural networks until people could no longer overlook their efficacy. If the NN doesn't get it right, it's absolutely not the end of the world. First of all, triaging the failures usually turns up something interesting / some connecting thread. And as for fixing issues, your options are typically: 1. Retrain with more examples to help it get it right (easier these days since you can take advantage of synthetic data / self- or semi-supervised approaches). Specifically it will help to generate or find examples that approximate the failures. 2. Shape your loss to penalize undesirable behavior / tendencies you've identified in triage 3. Add inputs or outputs that help your net solve the problem better (e.g. side inputs that may help the net not overfit to its available data) These "inscrutable black boxes," while hard to edit directly, are not impossible to evaluate which means you can just measure to see if they're better than more explicit systems. And most real world systems are patchworks of ML + heuristics to give some desired level of performance. Furthermore there is a big field of folks dedicated to elucidating what your net is learning (e.g. saliency). The tools are pretty cool -- for example in computer vision you can visualize whether your net is overfitting to background data in your training examples. A valid critique of all of this is simply that it's expensive. Very expensive to build teams to warehouse and clean huge amounts of data, then build unbiased / fair model evaluations, then build models, then integrate them into real world stuff, usually to be used in conjunction with non-learned algos. Another valid critique is that many companies have started using models they can't retrain or fix (e.g. off the shelf LLMs). This is dangerous territory since you lose so many of the options I outlined above.
spookie 3y ago
Yup, just the fact that topology is often rather poor, you're making it impossible to translate motion capture data for a lot of things. Facial expressions come to mind. I understand the reasons for different representations for reconstruction, but what we really need is to translate that into a mesh that is actually useful. Its great to have something as a reference, don't get me wrong, but we are still far away from even being able to discreetly assign materials to the object's various parts. Another big issue is licensing, most of these are not allowed commercially.
- fragmede 3y ago
  they're not free for commercial use. If you want to use them commercially, you get in contact with the research group that came up with it and pay them a pile of money to use it in your commercial product. For someone like Pixar, I can see them ponying up the cash for it if they think it'll allow them to save on animator hour costs and/or iterate faster.
lmpdev 3y ago
I fear mesh maybe the modelling equivalent of OOP. Not because it is the best paradigm for all situations, but just through inertia of use over decades
LarsDu88 3y ago
I do a lot of Blender 3d modeling as part of my hobby gamedev stuff. I think one thing that a lot of folks tend to forget is that quad and tri meshes are only one of SEVERAL 3d modeling techniques. It just so happens that this is the technique that the world landed on in the 90s due to how flexible it is, how performant it is, and algorithms like Catmull-Clark subdivision. Remember, Toy Story 1 actually used NURB surfaces for all the characters! For a Bug's Life and Toy Story 2, they redid all the meshes as catmull-clark subdividable surfaces. And guess who was CEO and founder of Pixar? There also techniques like NURBs, signed distance fields, and metaballs (all stuff you can actually use in Blender) for generating 3d surfaces. And of course the techniques used in CAD software. With the right algorithm and sufficient compute, there could be some sort of completely alternate 3d representation in the future for creating virtual spaces.
- opyate 3y ago
  This fictional documentary title just popped into my head: "Cloudy with a Chance of Metaballs: A history of 3D modelling in animated film."