30 ms·

I've been bullish[1] on this as a major aspect of generative AI for a while now, so it's great to see this paper published. 3D has an extremely steep learning

I've been bullish[1] on this as a major aspect of generative AI for a while now, so it's great to see this paper published. 3D has an extremely steep learning curve once you try to do anything non-trivial, especially in terms of asset creation for VR etc. but my real interest is where this leads in terms of real-world items. One of the major hurdles is that in the real-world we aren't as forgiving as we are in VR/games. I'm not entirely surprised to see that most of the outputs are "artistic" ones, but I'm really interested to see where this ends up when we can give AI combined inputs from text/photos/LIDAR etc and have it make the model for a physical item that can be 3D printed. [1] https://www.technicalchops.com/articles/ai-inputs-and-outputs-in-the-real-world/ https://www.technicalchops.com/articles/ai-inputs-and-output...

milofeynman 2y ago
Autodesk has been building practical 3d models for years with generative design. I have to imagine it's only getting better with these recent advances, but I'm not immersed in the space.
slidehero 2y ago
>I've been bullish[1] on this as a major aspect of generative AI for a while now, so it's great to see this paper published. Me too. My first thought when seeing 2D AI generated images was that 3D would be a logical next step. Compared to pixels, there's so much additional data to work with when training these models I just assumed that 3D would be an easier problem to solve than 2D image generation. You have 3D point data, edges, edge loops, bone systems etc and a lot of the existing data is pretty well labeled too.
lallysingh 2y ago
I'm excited to see the main deterrents to Indy gave dev: art and sound, get AI'd away. A single developer could use an off-the-shelf game engine and some AI generated assets (perhaps combining with whatever they can buy cheap) to develop some fun games. Still, getting from still models to something that animates is necessary:(
- vermilingua 2y ago
  This is exactly what I’m not excited for: indie game discovery is already hard enough, Steam widening the floodgates has not been a positive experience. Reducing the effort to create games even further is going to DoS the indie game market as we see the same studios that pump out hundreds of hentai games suddenly able to broaden their audiences significantly.
  - drschwabe 2y ago
    That's the idea, you won't have to discover - instead you can just create the game you want.
    - latentsea 2y ago
      Be careful what you wish for.
  - livrem 2y ago
    I think having modern game engines reducing the need for game programmers to almost zero caused much of this, but it also resulted in some interesting games when artists could create games without a need to hire programmers. It will be interesting to see if AI art (and AI 3D models) will mean that we see interesting games instead created by programmers without having to hire any artists. What I do not look forward to is the predictable spam flood of games created without both artists and programmers.
    - l33tman 2y ago
      To be fair, this is already the case on all the platforms, as you can easily put together a game with free assets from the assetstores (or pay a few dollars for pretty high quality assets). For every standard game genre you can imagine I'm sure there are thousands of generic games released every year on every playform (don't have any real numbers but I get that feeling...) Rendering the assets by AI or buying them from the asset store is not going to change the number of generic games put out there I think, maybe AI gen can make some of them a bit more unique at best.
aabhay 2y ago
I have to be a soggy blanket person, but there are some pretty strong reasons why 3D generative AI is going to have a much shallower adoption cycle than 2D. (I founded a company that was likely the first generative AI company for 3D assets) 1. 3D is actually a broad collection of formats and not a single thing or representation. This is because of the deep relation between surface topology and render performance. 2. 3D is much more challenging to use in any workflow. Its much more physical, and the ergonomics of 3DOF makes it naturally hard to place in as many places as 2D 3. 3D is much more expensive to produce per unit value, in many ways. This is why, for example, almost every indie web comic artist draws in 2D instead of 3D. In an ai first world it might be less “work” but will still be leaps and bounds more expensive. In my opinion, the media that have the most appeal for genAI are basically (in order) - images - videos - music - general audio - 2D animation - tightly scoped 3D experiences such as avatars - games - general 3D models. My conclusion from being in this space was that there’s likely a world where 3D-style videos generated from pixels are more poised to take off than 3D as a data type.
- noduerme 2y ago
  You skipped an important market segment. Porn. The history of technology for making and distributing images, from cave paintings to ancient Rome through the Renaissance, Betamax, Photoshop, Tomb Raider and Instagram filters, has always been driven by improving the rendering of boobs.
- fartfeatures 2y ago
  Doesn't Nvidia's USD format solve the multiple formats issue?
  - unconed 2y ago
    I think you mean Pixar's. USD was designed for letting very large teams collaborate on making animated movies. It's actually terrible as an interchange format. e.g. The materials/shading system is both overengineered and underdefined. It is entirely application specific. For the parts of USD that do work well, we have better and/or more widely supported standards, like GLTF and PBR. It was a very dumb choice to push this anywhere.

milofeynman 2y ago

Autodesk has been building practical 3d models for years with generative design. I have to imagine it's only getting better with these recent advances, but I'm not immersed in the space.

slidehero 2y ago

>I've been bullish[1] on this as a major aspect of generative AI for a while now, so it's great to see this paper published. Me too. My first thought when seeing 2D AI generated images was that 3D would be a logical next step. Compared to pixels, there's so much additional data to work with when training these models I just assumed that 3D would be an easier problem to solve than 2D image generation. You have 3D point data, edges, edge loops, bone systems etc and a lot of the existing data is pretty well labeled too.

lallysingh 2y ago

I'm excited to see the main deterrents to Indy gave dev: art and sound, get AI'd away. A single developer could use an off-the-shelf game engine and some AI generated assets (perhaps combining with whatever they can buy cheap) to develop some fun games. Still, getting from still models to something that animates is necessary:(

vermilingua 2y ago
This is exactly what I’m not excited for: indie game discovery is already hard enough, Steam widening the floodgates has not been a positive experience. Reducing the effort to create games even further is going to DoS the indie game market as we see the same studios that pump out hundreds of hentai games suddenly able to broaden their audiences significantly.
- drschwabe 2y ago
  That's the idea, you won't have to discover - instead you can just create the game you want.
  - latentsea 2y ago
    Be careful what you wish for.
- livrem 2y ago
  I think having modern game engines reducing the need for game programmers to almost zero caused much of this, but it also resulted in some interesting games when artists could create games without a need to hire programmers. It will be interesting to see if AI art (and AI 3D models) will mean that we see interesting games instead created by programmers without having to hire any artists. What I do not look forward to is the predictable spam flood of games created without both artists and programmers.
  - l33tman 2y ago
    To be fair, this is already the case on all the platforms, as you can easily put together a game with free assets from the assetstores (or pay a few dollars for pretty high quality assets). For every standard game genre you can imagine I'm sure there are thousands of generic games released every year on every playform (don't have any real numbers but I get that feeling...) Rendering the assets by AI or buying them from the asset store is not going to change the number of generic games put out there I think, maybe AI gen can make some of them a bit more unique at best.

aabhay 2y ago

I have to be a soggy blanket person, but there are some pretty strong reasons why 3D generative AI is going to have a much shallower adoption cycle than 2D. (I founded a company that was likely the first generative AI company for 3D assets) 1. 3D is actually a broad collection of formats and not a single thing or representation. This is because of the deep relation between surface topology and render performance. 2. 3D is much more challenging to use in any workflow. Its much more physical, and the ergonomics of 3DOF makes it naturally hard to place in as many places as 2D 3. 3D is much more expensive to produce per unit value, in many ways. This is why, for example, almost every indie web comic artist draws in 2D instead of 3D. In an ai first world it might be less “work” but will still be leaps and bounds more expensive. In my opinion, the media that have the most appeal for genAI are basically (in order) - images - videos - music - general audio - 2D animation - tightly scoped 3D experiences such as avatars - games - general 3D models. My conclusion from being in this space was that there’s likely a world where 3D-style videos generated from pixels are more poised to take off than 3D as a data type.

noduerme 2y ago
You skipped an important market segment. Porn. The history of technology for making and distributing images, from cave paintings to ancient Rome through the Renaissance, Betamax, Photoshop, Tomb Raider and Instagram filters, has always been driven by improving the rendering of boobs.
fartfeatures 2y ago
Doesn't Nvidia's USD format solve the multiple formats issue?
- unconed 2y ago
  I think you mean Pixar's. USD was designed for letting very large teams collaborate on making animated movies. It's actually terrible as an interchange format. e.g. The materials/shading system is both overengineered and underdefined. It is entirely application specific. For the parts of USD that do work well, we have better and/or more widely supported standards, like GLTF and PBR. It was a very dumb choice to push this anywhere.