11 ms·

Stable Fast 3D: Rapid 3D Asset Generation from Single Images

321 points by meetpateltech 2y ago

talldayo 2y ago
> 0.5 seconds per 3D asset generation on a GPU with 7GB VRAM Holy cow - I was thinking this might be one of those datacenter-only models but here I am proven wrong. 7GB of VRAM suggests this could run on a lot of hardware that 3D artists own already.
msp26 2y ago
You can interact with the models on their project page: https://stable-fast-3d.github.io/ https://stable-fast-3d.github.io/
calini 2y ago
I'm going to 3D print so much dumb stuff with this.
- jsheard 2y ago
  They're still hesitant to show the untextured version of the models so I would assume it's like previous efforts where most of the detail is in the textures, and the model itself, the part you would 3D print, isn't so impressive.
  - yazzku 2y ago
    I was going to comment on the same; these 3d reconstructions often generate a mess of a topology, and this post does not show any of the mesh triangulations, so I assume they're still not good. Arguably, the meshes are bad even for rendering.
    - dlivingston 2y ago
      Presumably, these meshes can be cleaned up using standard mesh refinement algorithms, like those found in MeshLab: https://www.meshlab.net/#features https://www.meshlab.net/#features
      - Keyframe 2y ago
        Hopefully that's in the (near) future, but as of now there still exists 'retopo' in 3D work for a reason. Just like roto and similar menial tasks. We're getting there with automation though.
  - mft_ 2y ago
    You can download a .glb file (from the HuggingFace demo page) and open it locally (e.g. in MS 3D Viewer). I'm looking at a mesh from one of the better examples I tried and it's actually pretty good...
  - jayd16 2y ago
    You know I do wonder about this. If its just for static assets does it really matter? In something like Unreal, the textures are going to be virtualized and the geometry is going to be turned in to LODed triangle soup anyway. Has anyone tried to build an Unreal scene with these generated meshes?
    - jsheard 2y ago
      Usually the problem is the model itself is severely lacking in detail, sure Nanite could make light work of a poorly optimized model but it's not going to fix the model being a vague blob which doesn't hold up to close scrutiny.
      - kaibee 2y ago
        Generate the accompanying normal map and then just tesselate it?
      - andybak 2y ago
        So don't use them in a context where they require close scrutiny?
- fragmede 2y ago
  hueforge
bloopernova 2y ago
Closer and closer to the automatic mapping drones from Prometheus. I wonder what the optimum group of technologies is that would enable that kind of mapping? Would you pile on LIDAR, RADAR, this tech, ultrasound, magnetic sensing, etc etc. Although, you're then getting a flying tricorder. Which could enable some cool uses even outside the stereotypical search and rescue.
- nycdatasci 2y ago
  High-res images from multiple perspectives should be sufficient. If you have a consumer drone, this product (no affiliation) is extremely impressive: https://www.dronedeploy.com/ https://www.dronedeploy.com/ You basically select an area on a map that you want to model in 3d, it flies your drone (take-off, flight path, landing), takes pictures, uploads to their servers for processing, generates point cloud, etc. Very powerful.
  - thetoon 2y ago
    What you could do with WebODM is already quite impressive
- alsodumb 2y ago
  Are you talking about mapping tunnels with drones? That's already done and it doesn't really need any 'AI': it's plain old SLAM. DARPA's subterranean challenge had many teams that did some pretty cool stuff in this direction: https://spectrum.ieee.org/darpa-subterranean-challenge-2657170650 https://spectrum.ieee.org/darpa-subterranean-challenge-26571...
- sorenjan 2y ago
  You don't need or want generative AI for mapping, you "just" need lidar and drones for slam. https://www.youtube.com/watch?v=1CWWP9jb4cE https://www.youtube.com/watch?v=1CWWP9jb4cE
- pzo 2y ago
  You already have depth anything v2 that can generate depthmap in realtime even on iPhone. Quality is pretty good but probably will be even improved in the future. Actually in many ways those depthmaps are much better quality than iPhone Lidar or Truedepth camera (that cannot handle transparent, metalic, reflective surfaces and also they have a big noise). https://github.com/DepthAnything/Depth-Anything-V2 https://github.com/DepthAnything/Depth-Anything-V2 https://huggingface.co/spaces/pablovela5620/depth-compare https://huggingface.co/spaces/pablovela5620/depth-compare https://huggingface.co/apple/coreml-depth-anything-v2-small https://huggingface.co/apple/coreml-depth-anything-v2-small
specproc 2y ago
Be still my miniature-painting heart.
timr 2y ago
For all of the hype around LLMs, this general area (image generation and graphical assets) seems to me to be the big long-term winner of current-generation AI. It hits the sweet spot for the fundamental limitations of the methods: * so-called "hallucination" (actually just how generative models work) is a feature, not a bug. * anyone can easily see the unrealistic and biased outputs without complex statistical tests. * human intuition is useful for evaluation, and not fundamentally misleading (i.e. the equivalent of "this text sounds fluent, so the generator must be intelligent!" hype doesn't really exist for imagery. We're capable of treating it as technology and evaluating it fairly, because there's no equivalent human capability.) * even lossy, noisy, collapsed and over-trained methods can be valuable for different creative pursuits. * perfection is not required. You can easily see distorted features in output, and iteratively try to improve them. * consistency is not required (though it will unlock hugely valuable applications, like video, should it ever arrive). * technologies like LoRA allow even unskilled users to train character-, style- or concept-specific models with ease. I've been amazed at how much better image / visual generation models have become in the last year, and IMO, the pace of improvement has not been slowing as much as text models. Moreover, it's becoming increasingly clear that the future isn't the wholesale replacement of photographers, cinematographers, etc., but rather, a generation of crazy AI-based power tools that can do things like add and remove concepts to imagery with a few text prompts. It's insanely useful, and just like Photoshop in the 90s, a new generation of power-users is already emerging, and doing wild things with the tools.
- CuriouslyC 2y ago
  Image models are a great way to understand generate AI. It's like surveying a battlefield from the air as opposed to the ground.
- ibash 2y ago
  > anyone can easily see the unrealistic outputs without complex statistical tests. This is key, we’re all pre-wired with fast correctness tests. Are there other data types that match this?
  - batch12 2y ago
quantumwoke 2y ago
Great result. Just had a play around with the demo models and they preserve structure really nicely; although the textures are still not great. It's kind of a voxelized version of the input image
deleted 2y ago
[deleted]
mft_ 2y ago
I'm really excited for something in this area to really deliver, and it's really cool that I can just drag pictures into the demo on HuggingFace [0] to try it. However... mixed success. It's not good with (real) cats yet - which was obvs the first thing I tried. It did reasonably well with a simple image of an iPhone, and actually pretty impressively with a pancake with fruit on top, terribly with a rocket, and impressively again with a rack of pool balls. [0] https://huggingface.co/spaces/stabilityai/stable-fast-3d https://huggingface.co/spaces/stabilityai/stable-fast-3d
kleiba 2y ago
This is good news for the indie game dev scene, I suppose?
- jayd16 2y ago
  The models aren't really optimized for game dev. Fine for machinima, probably.
ww520 2y ago
This is a great step forward. I wonder whether RAG based 3D animation generation can be done with this. 1. Textual description of a story. 2. Extract/generate keywords from the story using LLM. 3. Search and look up 2D images by the keywords. 4. Generate 3D models from the 2D images using Stable Fast 3D. 5. Extract/generate path description from the story using LLM. 6. Generate movement/animation/gait using some AI. ... 7. Profit??
- nwoli 2y ago
  Pre generate a bunch of images via sdxl and convert to 3d and then serve nearest mesh after querying
nwoli 2y ago
Huggingface space to try it https://huggingface.co/spaces/stabilityai/stable-fast-3d https://huggingface.co/spaces/stabilityai/stable-fast-3d
Y_Y 2y ago
It really looks like they've been doing that classic infomercial tactic of desaturating the images of the things they're comparing against to make theirs seen better.
woolion 2y ago
This is the third image to 3D AI I've tested, and in all cases the examples they give look like 2D renders of 3D models already. My tests were with cel-shaded images (cartoony, not with realistic lighting) and the model outputs something very flat but with very bad topology, which is worse than starting with a low poly or extruding the drawing. I suspect it is unable to give decent results without accurate shadows from which the normal vectors could be recomputed and thus lacks any 'understanding' of what the structure would be from the lines and forms. In any case it would be cool if they specified the set of inputs that is expected to give decent results.
- quitit 2y ago
  It might not just be your tests. All of my tests of img2mesh technologies have produced poor results, even when using images that are very similar to the ones featured in their demo. I’ve never got fidelity like what they’ve shown. I’ll give this a whirl and see if it performs better.
  - woolion 2y ago
    All right, I was hesitating to try shading some images to see if that improves the quality. It's probably still too early.
  - quitit 2y ago
    Tried it with a collection of images, and in my opinion it performs -worse- than earlier releases. It is however fast.
- diggan 2y ago
  What stuck out to me from this release was this: > Optional quad or triangle remeshing (adding only 100-200ms to processing time) But it seems to have been optional. Did you try it with that turned on? I'd be very interested in those results, as I had the same experience as you, the models don't generate good enough meshes, so was hoping this one would be a bit better at that. Edit: I just tried it out myself on their Huggingface demo and even with the predefined images they have there, the mesh output is just not good enough. https://i.imgur.com/e6voLi6.png https://i.imgur.com/e6voLi6.png
nextworddev 2y ago
For those reading from Stability - just tried it - API seems to be down and the notebook doesn't have the example code it claimed to have.
hansc 2y ago
Looks very good on examples, but testing a few Ikea chairs or a Donald Duck image gives very wrong results. You can test here: https://huggingface.co/spaces/stabilityai/stable-fast-3d https://huggingface.co/spaces/stabilityai/stable-fast-3d
ksec 2y ago
Given the Graphics Asset part of AA or AAA Games are the most expensive, i wonder if 3D Asset Generation could perhaps drastically lower that by 50% or more? At least in terms of same output. Because in reality I guess artist will just spend more time in other areas.
causi 2y ago
Man it would be so cool to get AI-assisted photogrammetry. Imagine that instead of taking a hundred photos or a slow scan and having to labor over a point cloud, you could just take like three pictures and then go down a checklist. "Is this circular? How long is this straight line? Is this surface flat? What's the angle between these two pieces?" and getting a perfect replica or even a STEP file out of it. Heaven for 3D printers.
deleted 2y ago
[deleted]
puppycodes 2y ago
I really can't wait for this technology to improve. Unfortunately just from testing this it seems not very useful. It takes more work to modify the bad model it approximates from the image output than starting with a good foundation from scratch. I would rather see something that took a series of steps to reach a higher quality end product more slowly instead of expecting everything to come from one image. Perhaps i'm missing the use case?
- MrTrvp 2y ago
  Perhaps it'll require a series of segmentation and transforms that improves individual components and then works up towards the full 3d model of the image.
- andybak 2y ago
  > not very useful Useful for what? I think use cases will emerge. A lot of critiques assume you're working in VFX or game development. Making image to 3d (and by extension text to image to 3d) effortless a whole host of new applications open up - which might not be anywhere near so demanding.
fsloth 2y ago
Not the holy grail yet, but pretty cool! I see these usable not as main assets, but as something you would add as a low effort embellishment to add complexity to the main scene. The fact they maintain profile makes them usable for situations where mere 2d billboard impostor (i.e the original image always oriented towards the camera) would not cut it. You can totally create a figure image (Midjourney|Bing|Dalle3) and drag and drop it to the image input and get a surprising good 3d presentation that is not a highly detailed model, but something you could very well put to a shelf in a 3d scene as an embellishment where the camera never sees the back of it, and the model is never at the center of attention.
abidlabs 2y ago
Official Gradio demo is here: https://huggingface.co/spaces/stabilityai/stable-fast-3d https://huggingface.co/spaces/stabilityai/stable-fast-3d
voidUpdate 2y ago
What I'd really like to see in these kinds of articles is examples of it not working as well. I don't necessarily want to see it being perfect, I'd quite like to see its limitations too

321 points by meetpateltech 2y ago

> 0.5 seconds per 3D asset generation on a GPU with 7GB VRAM Holy cow - I was thinking this might be one of those datacenter-only models but here I am proven wrong. 7GB of VRAM suggests this could run on a lot of hardware that 3D artists own already.

msp26 2y ago

You can interact with the models on their project page: https://stable-fast-3d.github.io/ https://stable-fast-3d.github.io/

calini 2y ago

I'm going to 3D print so much dumb stuff with this.

jsheard 2y ago
They're still hesitant to show the untextured version of the models so I would assume it's like previous efforts where most of the detail is in the textures, and the model itself, the part you would 3D print, isn't so impressive.
- yazzku 2y ago
  I was going to comment on the same; these 3d reconstructions often generate a mess of a topology, and this post does not show any of the mesh triangulations, so I assume they're still not good. Arguably, the meshes are bad even for rendering.
  - dlivingston 2y ago
    Presumably, these meshes can be cleaned up using standard mesh refinement algorithms, like those found in MeshLab: https://www.meshlab.net/#features https://www.meshlab.net/#features
    - Keyframe 2y ago
      Hopefully that's in the (near) future, but as of now there still exists 'retopo' in 3D work for a reason. Just like roto and similar menial tasks. We're getting there with automation though.
- mft_ 2y ago
  You can download a .glb file (from the HuggingFace demo page) and open it locally (e.g. in MS 3D Viewer). I'm looking at a mesh from one of the better examples I tried and it's actually pretty good...
- jayd16 2y ago
  You know I do wonder about this. If its just for static assets does it really matter? In something like Unreal, the textures are going to be virtualized and the geometry is going to be turned in to LODed triangle soup anyway. Has anyone tried to build an Unreal scene with these generated meshes?
  - jsheard 2y ago
    Usually the problem is the model itself is severely lacking in detail, sure Nanite could make light work of a poorly optimized model but it's not going to fix the model being a vague blob which doesn't hold up to close scrutiny.
    - kaibee 2y ago
      Generate the accompanying normal map and then just tesselate it?
    - andybak 2y ago
      So don't use them in a context where they require close scrutiny?
fragmede 2y ago
hueforge

bloopernova 2y ago

Closer and closer to the automatic mapping drones from Prometheus. I wonder what the optimum group of technologies is that would enable that kind of mapping? Would you pile on LIDAR, RADAR, this tech, ultrasound, magnetic sensing, etc etc. Although, you're then getting a flying tricorder. Which could enable some cool uses even outside the stereotypical search and rescue.

nycdatasci 2y ago
High-res images from multiple perspectives should be sufficient. If you have a consumer drone, this product (no affiliation) is extremely impressive: https://www.dronedeploy.com/ https://www.dronedeploy.com/ You basically select an area on a map that you want to model in 3d, it flies your drone (take-off, flight path, landing), takes pictures, uploads to their servers for processing, generates point cloud, etc. Very powerful.
- thetoon 2y ago
  What you could do with WebODM is already quite impressive
alsodumb 2y ago
Are you talking about mapping tunnels with drones? That's already done and it doesn't really need any 'AI': it's plain old SLAM. DARPA's subterranean challenge had many teams that did some pretty cool stuff in this direction: https://spectrum.ieee.org/darpa-subterranean-challenge-2657170650 https://spectrum.ieee.org/darpa-subterranean-challenge-26571...
sorenjan 2y ago
You don't need or want generative AI for mapping, you "just" need lidar and drones for slam. https://www.youtube.com/watch?v=1CWWP9jb4cE https://www.youtube.com/watch?v=1CWWP9jb4cE
pzo 2y ago
You already have depth anything v2 that can generate depthmap in realtime even on iPhone. Quality is pretty good but probably will be even improved in the future. Actually in many ways those depthmaps are much better quality than iPhone Lidar or Truedepth camera (that cannot handle transparent, metalic, reflective surfaces and also they have a big noise). https://github.com/DepthAnything/Depth-Anything-V2 https://github.com/DepthAnything/Depth-Anything-V2 https://huggingface.co/spaces/pablovela5620/depth-compare https://huggingface.co/spaces/pablovela5620/depth-compare https://huggingface.co/apple/coreml-depth-anything-v2-small https://huggingface.co/apple/coreml-depth-anything-v2-small

specproc 2y ago

Be still my miniature-painting heart.

timr 2y ago

For all of the hype around LLMs, this general area (image generation and graphical assets) seems to me to be the big long-term winner of current-generation AI. It hits the sweet spot for the fundamental limitations of the methods: * so-called "hallucination" (actually just how generative models work) is a feature, not a bug. * anyone can easily see the unrealistic and biased outputs without complex statistical tests. * human intuition is useful for evaluation, and not fundamentally misleading (i.e. the equivalent of "this text sounds fluent, so the generator must be intelligent!" hype doesn't really exist for imagery. We're capable of treating it as technology and evaluating it fairly, because there's no equivalent human capability.) * even lossy, noisy, collapsed and over-trained methods can be valuable for different creative pursuits. * perfection is not required. You can easily see distorted features in output, and iteratively try to improve them. * consistency is not required (though it will unlock hugely valuable applications, like video, should it ever arrive). * technologies like LoRA allow even unskilled users to train character-, style- or concept-specific models with ease. I've been amazed at how much better image / visual generation models have become in the last year, and IMO, the pace of improvement has not been slowing as much as text models. Moreover, it's becoming increasingly clear that the future isn't the wholesale replacement of photographers, cinematographers, etc., but rather, a generation of crazy AI-based power tools that can do things like add and remove concepts to imagery with a few text prompts. It's insanely useful, and just like Photoshop in the 90s, a new generation of power-users is already emerging, and doing wild things with the tools.

CuriouslyC 2y ago
Image models are a great way to understand generate AI. It's like surveying a battlefield from the air as opposed to the ground.
ibash 2y ago
> anyone can easily see the unrealistic outputs without complex statistical tests. This is key, we’re all pre-wired with fast correctness tests. Are there other data types that match this?
- batch12 2y ago

quantumwoke 2y ago

Great result. Just had a play around with the demo models and they preserve structure really nicely; although the textures are still not great. It's kind of a voxelized version of the input image

deleted 2y ago

[deleted]

mft_ 2y ago

I'm really excited for something in this area to really deliver, and it's really cool that I can just drag pictures into the demo on HuggingFace [0] to try it. However... mixed success. It's not good with (real) cats yet - which was obvs the first thing I tried. It did reasonably well with a simple image of an iPhone, and actually pretty impressively with a pancake with fruit on top, terribly with a rocket, and impressively again with a rack of pool balls. [0] https://huggingface.co/spaces/stabilityai/stable-fast-3d https://huggingface.co/spaces/stabilityai/stable-fast-3d

kleiba 2y ago

This is good news for the indie game dev scene, I suppose?

jayd16 2y ago
The models aren't really optimized for game dev. Fine for machinima, probably.

ww520 2y ago

This is a great step forward. I wonder whether RAG based 3D animation generation can be done with this. 1. Textual description of a story. 2. Extract/generate keywords from the story using LLM. 3. Search and look up 2D images by the keywords. 4. Generate 3D models from the 2D images using Stable Fast 3D. 5. Extract/generate path description from the story using LLM. 6. Generate movement/animation/gait using some AI. ... 7. Profit??

nwoli 2y ago
Pre generate a bunch of images via sdxl and convert to 3d and then serve nearest mesh after querying

nwoli 2y ago

Huggingface space to try it https://huggingface.co/spaces/stabilityai/stable-fast-3d https://huggingface.co/spaces/stabilityai/stable-fast-3d

Y_Y 2y ago

It really looks like they've been doing that classic infomercial tactic of desaturating the images of the things they're comparing against to make theirs seen better.

woolion 2y ago

This is the third image to 3D AI I've tested, and in all cases the examples they give look like 2D renders of 3D models already. My tests were with cel-shaded images (cartoony, not with realistic lighting) and the model outputs something very flat but with very bad topology, which is worse than starting with a low poly or extruding the drawing. I suspect it is unable to give decent results without accurate shadows from which the normal vectors could be recomputed and thus lacks any 'understanding' of what the structure would be from the lines and forms. In any case it would be cool if they specified the set of inputs that is expected to give decent results.

quitit 2y ago
It might not just be your tests. All of my tests of img2mesh technologies have produced poor results, even when using images that are very similar to the ones featured in their demo. I’ve never got fidelity like what they’ve shown. I’ll give this a whirl and see if it performs better.
- woolion 2y ago
  All right, I was hesitating to try shading some images to see if that improves the quality. It's probably still too early.
- quitit 2y ago
  Tried it with a collection of images, and in my opinion it performs -worse- than earlier releases. It is however fast.
diggan 2y ago
What stuck out to me from this release was this: > Optional quad or triangle remeshing (adding only 100-200ms to processing time) But it seems to have been optional. Did you try it with that turned on? I'd be very interested in those results, as I had the same experience as you, the models don't generate good enough meshes, so was hoping this one would be a bit better at that. Edit: I just tried it out myself on their Huggingface demo and even with the predefined images they have there, the mesh output is just not good enough. https://i.imgur.com/e6voLi6.png https://i.imgur.com/e6voLi6.png

nextworddev 2y ago

For those reading from Stability - just tried it - API seems to be down and the notebook doesn't have the example code it claimed to have.

hansc 2y ago

Looks very good on examples, but testing a few Ikea chairs or a Donald Duck image gives very wrong results. You can test here: https://huggingface.co/spaces/stabilityai/stable-fast-3d https://huggingface.co/spaces/stabilityai/stable-fast-3d

ksec 2y ago

Given the Graphics Asset part of AA or AAA Games are the most expensive, i wonder if 3D Asset Generation could perhaps drastically lower that by 50% or more? At least in terms of same output. Because in reality I guess artist will just spend more time in other areas.

causi 2y ago

Man it would be so cool to get AI-assisted photogrammetry. Imagine that instead of taking a hundred photos or a slow scan and having to labor over a point cloud, you could just take like three pictures and then go down a checklist. "Is this circular? How long is this straight line? Is this surface flat? What's the angle between these two pieces?" and getting a perfect replica or even a STEP file out of it. Heaven for 3D printers.

deleted 2y ago

[deleted]

puppycodes 2y ago

I really can't wait for this technology to improve. Unfortunately just from testing this it seems not very useful. It takes more work to modify the bad model it approximates from the image output than starting with a good foundation from scratch. I would rather see something that took a series of steps to reach a higher quality end product more slowly instead of expecting everything to come from one image. Perhaps i'm missing the use case?

MrTrvp 2y ago
Perhaps it'll require a series of segmentation and transforms that improves individual components and then works up towards the full 3d model of the image.
andybak 2y ago
> not very useful Useful for what? I think use cases will emerge. A lot of critiques assume you're working in VFX or game development. Making image to 3d (and by extension text to image to 3d) effortless a whole host of new applications open up - which might not be anywhere near so demanding.

fsloth 2y ago

Not the holy grail yet, but pretty cool! I see these usable not as main assets, but as something you would add as a low effort embellishment to add complexity to the main scene. The fact they maintain profile makes them usable for situations where mere 2d billboard impostor (i.e the original image always oriented towards the camera) would not cut it. You can totally create a figure image (Midjourney|Bing|Dalle3) and drag and drop it to the image input and get a surprising good 3d presentation that is not a highly detailed model, but something you could very well put to a shelf in a 3d scene as an embellishment where the camera never sees the back of it, and the model is never at the center of attention.

abidlabs 2y ago

Official Gradio demo is here: https://huggingface.co/spaces/stabilityai/stable-fast-3d https://huggingface.co/spaces/stabilityai/stable-fast-3d

voidUpdate 2y ago

What I'd really like to see in these kinds of articles is examples of it not working as well. I don't necessarily want to see it being perfect, I'd quite like to see its limitations too