6 ms·

Shap-E: Generate 3D objects conditioned on text or images

288 points by tim_sw 3y ago

marviel 3y ago
https://github.com/openai/shap-e/blob/main/shap_e/examples/sample_text_to_3d.ipynb https://github.com/openai/shap-e/blob/main/shap_e/examples/s... ^ Single file Jupyter Notebook for Text Into: (1) Gif (derived from NERFs I suppose) (2) .PLY
jamilton 3y ago
The models seem charmingly low-poly from the gifs.
qup 3y ago
I just got a 3D printer but I'm a total noob. How many more steps have to occur between these models and me printing them?
- moffkalast 3y ago
  If you check the issues, the latest one is from Jozef Prusa interested in doing straight up Printables integration... so not that many.
- dceddia 3y ago
  I'm not sure what format it's generating these models in, but if they're SDL, you should be able to drop them into a slicer program (like PrusaSlicer, Cura, etc) and generate the gcode there (the 3D printer movement instructions). Then load that gcode into your printer (via SD card, wifi, whatever it supports) and go! If these aren't in SDL, but are some format Blender can open, then Blender can turn them into SDL. It looks from the description like it's compatible with Blender.
  - kuida0r3 3y ago
    From the jupyter notebook in the repo [0], the output is in .ply or polygon file format which is used by some 3D scanners and can be converted to STL supported by most 3D printers. 0-https://github.com/openai/shap-e/blob/main/shap_e/examples/sample_text_to_3d.ipynb https://github.com/openai/shap-e/blob/main/shap_e/examples/s...
- krisoft 3y ago
  > How many more steps have to occur between these models and me printing them? Shouldn't be too hard. Based on the code of the examples it seems it can output polygonal meshes in ply format. Either your slicer natively supports that, or you can use something (for example blender) to convert the ply to a format your slicer supports. What can go wrong if the model "looks okay" but have some subtle geometric issue (like if it has a missing face, which makes it non-manifold or non-watertight) Nowadays many slicers have tools to heal such problems, but there can be dragons that way. I looked for an example object in their repo and found this one: https://github.com/openai/shap-e/blob/main/shap_e/examples/example_data/cactus/object.obj https://github.com/openai/shap-e/blob/main/shap_e/examples/e... If that is representative it is a very nice clean geometry. I have sliced it with prusa slicer no problem and I could be printing it within a minute of downloading it if I would need a 3d printed cactus in my life. > I just got a 3D printer but I'm a total noob. How noob you mean? If this would be literally your first print I would recommend printing first a sample file which came with your printer. (just to make sure everything is okay with the printer itself) But as a second print you can totally try to print one of these of course. Just don't give up the hobby if something goes wrong :) Something always goes wrong.
  - qup 3y ago
    > How noob you mean? I successfully printed a hollow cube yesterday, my first ever print, and failed on about 6 attempts at a "mario ring" file. It's PETG and I'm just fiddling with settings, getting a lot of stringing etc. Anyway, very new, but I'm prepared for the learning curve. I'm a good debugger, and I don't give up easily, so hopefully that'll carry through to the printing.
    - wkat4242 3y ago
      Like the others said, forget about petg until you know what you're doing. Just use PLA :)
    - filoleg 3y ago
      I got into 3D printing from zero knowledge a couple of months ago, and i found PETG an absolute nightmare to deal with compared to PLA. Mind you, i only bought filament from Prusa and AtomicFilament, two very respectable manufacturers. Had zero issues with their PLA, and it looked beautiful. With PETG, no matter how much i tried adjusting live-z levels or playing in PrusaSlicer with print speed levels, temperature, layer height, etc., it all would mess up at some point. Out of 5 prints of the same object, I finally managed to get one great print out of PETG. But when I tried printing another one, I would start having various issues again and would have to adjust things again in hopes of eventually getting it right. I just went back to PLA and had zero issues at all since then. Even though I am aware of PLA disadvantages compared to PETG, I simply realized that for my purposes, they didn't matter nearly as much as consistency and ease of good prints.
deleted 3y ago
[deleted]
deleted 3y ago
[deleted]
appleaday1 3y ago
TIL what a Ube ice cream cone is
- astrange 3y ago
  it's popular in the Philippines and it's an impressive purple color. Taro and pandan are good in desserts too. And then there's durian.
  - uhhyeahdude 3y ago
    Ube has been an increasingly trendy/popular flavor for all sorts of dessert food in America; this is perhaps most evident if you visit Hawaii, but it can also be found in pretty much any bubble tea shop as a flavor option. ubaehawaii makes an ube crinkle cookie that is as delicious as it is purple. I’m not affiliated or anything, just a fan of good cookies (and ube!)
alexmolas 3y ago
Finally some open AI from OpenAI
- juunpp 3y ago
  Is it? The code in the repository is MIT, but the models are downloaded from Azure: https://github.com/openai/shap-e/blob/main/shap_e/models/download.py https://github.com/openai/shap-e/blob/main/shap_e/models/dow... The code license in such cases would not extend to binary blobs downloaded from the Internet. And the code is basically useless without the blob.
  - pabs3 3y ago
    It definitely wouldn't pass Debian's machine learning policy: https://salsa.debian.org/deeplearning-team/ml-policy/ https://salsa.debian.org/deeplearning-team/ml-policy/ https://deepdive.opensource.org/podcast/why-debian-wont-distribute-ai-models-any-time-soon/ https://deepdive.opensource.org/podcast/why-debian-wont-dist...
  - newswasboring 3y ago
    I can't figure it out, what's the license on the blob?
    - juunpp 3y ago
      Not a lawyer, but at least for code, no license means you have no rights to that code, even if it's just one download way: https://opensource.stackexchange.com/questions/1720/what-can-i-assume-if-a-publicly-published-project-has-no-license https://opensource.stackexchange.com/questions/1720/what-can... I don't know if the same is true of blobs, and I also haven't checked whether the blobs are licensed anywhere on their site.
- speedgoose 3y ago
  OpenAi also released Whisper in September. They do open stuff once in a while.
chaxor 3y ago
Seems extremely similar to the Point-e work done a few months ago. I would imagine that just doing Veitoris Rips on the point-e would get close to this solution as well, although that would be a bit too logic based for the ML world we live in now.
- jahewson 3y ago
  That would get you a mesh but not a NeRF.
krisoft 3y ago
Tangentially related: A hobby I found immensely rewarding as of late is to ask a generative model to draw something pretty and then try to make the object in real life. I have previously asked dall-e to draw me steampunk watch-like gizmos, selected the prettiest and then I went and carved it out of wax. And right now I'm 3d modelling a copper owl jewellery design dreamed up by Midjourney. It is a really exciting exercise because it makes me think about what do I like about the image, what is the "essence of it". It is also stretches my maker muscles because when I design things for myself from scratch I always keep in mind the constraints of my tools and techniques. But of course the AI has no such compunctions. It just draws something pretty, manufacturability be damned.
- vertis 3y ago
  I've found it very rewarding as a source of fantastical ideas. I like that generating items in mid-journey can break me out of one line of thinking even if that's because it's result is absurd.
- matlo 3y ago
  [flagged]
  - cinntaile 3y ago
    190 GBP for a spoon. It reminds me of a certain expression, something about fools and money...
    - dkural 3y ago
      Eating is important. Life, is all about eating, and life gave rise to intelligence. /que dramatic music.
  - cinntaile 3y ago
    You should mention that this is your project by the way. You're pretending like it's not by writing they instead of we, while on your own website you state that you're co-founder of oio.
startupsfail 3y ago
There was something like that, but seemingly (comparing demo videos) much higher quality released a few months ago from NVIDIA - https://research.nvidia.com/labs/dir/magic3d/ https://research.nvidia.com/labs/dir/magic3d/
- ehsankia 3y ago
  I don't understand, if you have a 3d model, why do these gifs need to be so low res and aliased? The AI isn't generated gifs, why can't they properly render the previews in higher quality? The magic3d models don't look that much more detailed, they're just rendered properly...
  - mysterydip 3y ago
    perhaps to hide imperfections/rough edges and make it look nicer than it is?
- astrange 3y ago
  The colors on those are strange and unpleasant. I wonder if they forgot something basic like gamma correction.
- Racing0461 3y ago
  maybe thats why they released at on github instead of adding it as a feature on openai.com, it sucks.
  - orliesaurus 3y ago
    I actually thought this initially, but I also believe that given the nature of where we are going (in the 3d space at least) it makes sense to open source something that isn't commercially resellable (thinking all the people who just repackage openai APIs) and let the community iterate over it and make it better perhaps?
deleted 3y ago
[deleted]
bemmu 3y ago
HuggingFace space: https://huggingface.co/spaces/hysts/Shap-E https://huggingface.co/spaces/hysts/Shap-E
- newswasboring 3y ago
  Did it get hugged to death? I only see a runtime error.
personjerry 3y ago
Every time I click Samples.md and then click back, GitHub tells me "Access to this site has been restricted." and times me out for a while. Anybody have the same thing or know why?
- knobhandle 3y ago
  Had the same issue. On other browser I was able to browse the site. Just clear recent history and cookies.
cl42 3y ago
I've been playing with this a bit. Sharing some examples that aren't from their repo, in case people are curious. Simple objects are surprisingly good. Examples of a viking helmet below: -- https://phasellm.com/static/img/helmet-01.png https://phasellm.com/static/img/helmet-01.png -- https://phasellm.com/static/img/helmet-02.png https://phasellm.com/static/img/helmet-02.png More complex objects, e.g., "a viking" are harder though you can clearly see what the model is trying to do (cape, sword, person, etc.)... -- https://phasellm.com/static/img/viking-01.png https://phasellm.com/static/img/viking-01.png Maybe more iterations of the model could work. I'm sure it's only a matter of time before this gets much better, more details, etc.
- amelius 3y ago
  Wouldn't a more promising approach be to go through NeRFs first? So first figure out what you want the model to look like from various angles, and then try to make a textured 3d model out of it.
georyb 3y ago
I'm curios how much more difficult is generative 3D? We are barely able to generate 2D 1080p images, without artefacts using diffusion methods. How much more complex is the jump to production grade 3D? As someone creating re-usable 3D product models on Polymock @ https://polymock.com https://polymock.com the bigest issue for me, is that you want your models to be editable. 3D modellers want to post-process and add tweaks to existing 3D models. Current AI generators lack this layering structures. What I actually want, is a 3D generator for textures / materials. It should be a much simpler task for current AI. At the moment, the state of 3D generators is slowly ripening for indie game developers. I'm sure that in the near future, the models will be high-enough quality to generate a basic character set. Others have mentioned it, but I think NVIDIA has an edge in 3D synthesis & rendering. OpenAI is arguable the leader in consumer text-transformers - I wonder the goal for releasing this POC (I don't think they will allocate any serious resource here)
- placeholder_0 3y ago
  Have you tried texturelab.xyz?
  - georyb 3y ago
    Thanks for sharing - looks handy for games
- prox 3y ago
  Someone made a SD/blender hybrid add-on where texturing will be based on a prompt.
joeld42 3y ago
Tip: If you're loading these .glbs in blender, add an "Attribute" node in the shader editor, put "Col" in the name field, and connect "Color" to "Base Color" in the material to see the colors.
EamonnMR 3y ago
The next super Scribblenauts game is going to be wild.

288 points by tim_sw 3y ago

https://github.com/openai/shap-e/blob/main/shap_e/examples/sample_text_to_3d.ipynb https://github.com/openai/shap-e/blob/main/shap_e/examples/s... ^ Single file Jupyter Notebook for Text Into: (1) Gif (derived from NERFs I suppose) (2) .PLY

jamilton 3y ago

The models seem charmingly low-poly from the gifs.

qup 3y ago

I just got a 3D printer but I'm a total noob. How many more steps have to occur between these models and me printing them?

moffkalast 3y ago
If you check the issues, the latest one is from Jozef Prusa interested in doing straight up Printables integration... so not that many.
dceddia 3y ago
I'm not sure what format it's generating these models in, but if they're SDL, you should be able to drop them into a slicer program (like PrusaSlicer, Cura, etc) and generate the gcode there (the 3D printer movement instructions). Then load that gcode into your printer (via SD card, wifi, whatever it supports) and go! If these aren't in SDL, but are some format Blender can open, then Blender can turn them into SDL. It looks from the description like it's compatible with Blender.
- kuida0r3 3y ago
  From the jupyter notebook in the repo [0], the output is in .ply or polygon file format which is used by some 3D scanners and can be converted to STL supported by most 3D printers. 0-https://github.com/openai/shap-e/blob/main/shap_e/examples/sample_text_to_3d.ipynb https://github.com/openai/shap-e/blob/main/shap_e/examples/s...
krisoft 3y ago
> How many more steps have to occur between these models and me printing them? Shouldn't be too hard. Based on the code of the examples it seems it can output polygonal meshes in ply format. Either your slicer natively supports that, or you can use something (for example blender) to convert the ply to a format your slicer supports. What can go wrong if the model "looks okay" but have some subtle geometric issue (like if it has a missing face, which makes it non-manifold or non-watertight) Nowadays many slicers have tools to heal such problems, but there can be dragons that way. I looked for an example object in their repo and found this one: https://github.com/openai/shap-e/blob/main/shap_e/examples/example_data/cactus/object.obj https://github.com/openai/shap-e/blob/main/shap_e/examples/e... If that is representative it is a very nice clean geometry. I have sliced it with prusa slicer no problem and I could be printing it within a minute of downloading it if I would need a 3d printed cactus in my life. > I just got a 3D printer but I'm a total noob. How noob you mean? If this would be literally your first print I would recommend printing first a sample file which came with your printer. (just to make sure everything is okay with the printer itself) But as a second print you can totally try to print one of these of course. Just don't give up the hobby if something goes wrong :) Something always goes wrong.
- qup 3y ago
  > How noob you mean? I successfully printed a hollow cube yesterday, my first ever print, and failed on about 6 attempts at a "mario ring" file. It's PETG and I'm just fiddling with settings, getting a lot of stringing etc. Anyway, very new, but I'm prepared for the learning curve. I'm a good debugger, and I don't give up easily, so hopefully that'll carry through to the printing.
  - wkat4242 3y ago
    Like the others said, forget about petg until you know what you're doing. Just use PLA :)
  - filoleg 3y ago
    I got into 3D printing from zero knowledge a couple of months ago, and i found PETG an absolute nightmare to deal with compared to PLA. Mind you, i only bought filament from Prusa and AtomicFilament, two very respectable manufacturers. Had zero issues with their PLA, and it looked beautiful. With PETG, no matter how much i tried adjusting live-z levels or playing in PrusaSlicer with print speed levels, temperature, layer height, etc., it all would mess up at some point. Out of 5 prints of the same object, I finally managed to get one great print out of PETG. But when I tried printing another one, I would start having various issues again and would have to adjust things again in hopes of eventually getting it right. I just went back to PLA and had zero issues at all since then. Even though I am aware of PLA disadvantages compared to PETG, I simply realized that for my purposes, they didn't matter nearly as much as consistency and ease of good prints.

deleted 3y ago

[deleted]

deleted 3y ago

[deleted]

appleaday1 3y ago

TIL what a Ube ice cream cone is

astrange 3y ago
it's popular in the Philippines and it's an impressive purple color. Taro and pandan are good in desserts too. And then there's durian.
- uhhyeahdude 3y ago
  Ube has been an increasingly trendy/popular flavor for all sorts of dessert food in America; this is perhaps most evident if you visit Hawaii, but it can also be found in pretty much any bubble tea shop as a flavor option. ubaehawaii makes an ube crinkle cookie that is as delicious as it is purple. I’m not affiliated or anything, just a fan of good cookies (and ube!)

alexmolas 3y ago

Finally some open AI from OpenAI

juunpp 3y ago
Is it? The code in the repository is MIT, but the models are downloaded from Azure: https://github.com/openai/shap-e/blob/main/shap_e/models/download.py https://github.com/openai/shap-e/blob/main/shap_e/models/dow... The code license in such cases would not extend to binary blobs downloaded from the Internet. And the code is basically useless without the blob.
- pabs3 3y ago
  It definitely wouldn't pass Debian's machine learning policy: https://salsa.debian.org/deeplearning-team/ml-policy/ https://salsa.debian.org/deeplearning-team/ml-policy/ https://deepdive.opensource.org/podcast/why-debian-wont-distribute-ai-models-any-time-soon/ https://deepdive.opensource.org/podcast/why-debian-wont-dist...
- newswasboring 3y ago
  I can't figure it out, what's the license on the blob?
  - juunpp 3y ago
    Not a lawyer, but at least for code, no license means you have no rights to that code, even if it's just one download way: https://opensource.stackexchange.com/questions/1720/what-can-i-assume-if-a-publicly-published-project-has-no-license https://opensource.stackexchange.com/questions/1720/what-can... I don't know if the same is true of blobs, and I also haven't checked whether the blobs are licensed anywhere on their site.
speedgoose 3y ago
OpenAi also released Whisper in September. They do open stuff once in a while.

chaxor 3y ago

Seems extremely similar to the Point-e work done a few months ago. I would imagine that just doing Veitoris Rips on the point-e would get close to this solution as well, although that would be a bit too logic based for the ML world we live in now.

jahewson 3y ago
That would get you a mesh but not a NeRF.

krisoft 3y ago

Tangentially related: A hobby I found immensely rewarding as of late is to ask a generative model to draw something pretty and then try to make the object in real life. I have previously asked dall-e to draw me steampunk watch-like gizmos, selected the prettiest and then I went and carved it out of wax. And right now I'm 3d modelling a copper owl jewellery design dreamed up by Midjourney. It is a really exciting exercise because it makes me think about what do I like about the image, what is the "essence of it". It is also stretches my maker muscles because when I design things for myself from scratch I always keep in mind the constraints of my tools and techniques. But of course the AI has no such compunctions. It just draws something pretty, manufacturability be damned.

vertis 3y ago
I've found it very rewarding as a source of fantastical ideas. I like that generating items in mid-journey can break me out of one line of thinking even if that's because it's result is absurd.
matlo 3y ago
[flagged]
- cinntaile 3y ago
  190 GBP for a spoon. It reminds me of a certain expression, something about fools and money...
  - dkural 3y ago
    Eating is important. Life, is all about eating, and life gave rise to intelligence. /que dramatic music.
- cinntaile 3y ago
  You should mention that this is your project by the way. You're pretending like it's not by writing they instead of we, while on your own website you state that you're co-founder of oio.

startupsfail 3y ago

There was something like that, but seemingly (comparing demo videos) much higher quality released a few months ago from NVIDIA - https://research.nvidia.com/labs/dir/magic3d/ https://research.nvidia.com/labs/dir/magic3d/

ehsankia 3y ago
I don't understand, if you have a 3d model, why do these gifs need to be so low res and aliased? The AI isn't generated gifs, why can't they properly render the previews in higher quality? The magic3d models don't look that much more detailed, they're just rendered properly...
- mysterydip 3y ago
  perhaps to hide imperfections/rough edges and make it look nicer than it is?
astrange 3y ago
The colors on those are strange and unpleasant. I wonder if they forgot something basic like gamma correction.
Racing0461 3y ago
maybe thats why they released at on github instead of adding it as a feature on openai.com, it sucks.
- orliesaurus 3y ago
  I actually thought this initially, but I also believe that given the nature of where we are going (in the 3d space at least) it makes sense to open source something that isn't commercially resellable (thinking all the people who just repackage openai APIs) and let the community iterate over it and make it better perhaps?

deleted 3y ago

[deleted]

bemmu 3y ago

HuggingFace space: https://huggingface.co/spaces/hysts/Shap-E https://huggingface.co/spaces/hysts/Shap-E

newswasboring 3y ago
Did it get hugged to death? I only see a runtime error.

personjerry 3y ago

Every time I click Samples.md and then click back, GitHub tells me "Access to this site has been restricted." and times me out for a while. Anybody have the same thing or know why?

knobhandle 3y ago
Had the same issue. On other browser I was able to browse the site. Just clear recent history and cookies.

cl42 3y ago

I've been playing with this a bit. Sharing some examples that aren't from their repo, in case people are curious. Simple objects are surprisingly good. Examples of a viking helmet below: -- https://phasellm.com/static/img/helmet-01.png https://phasellm.com/static/img/helmet-01.png -- https://phasellm.com/static/img/helmet-02.png https://phasellm.com/static/img/helmet-02.png More complex objects, e.g., "a viking" are harder though you can clearly see what the model is trying to do (cape, sword, person, etc.)... -- https://phasellm.com/static/img/viking-01.png https://phasellm.com/static/img/viking-01.png Maybe more iterations of the model could work. I'm sure it's only a matter of time before this gets much better, more details, etc.

amelius 3y ago
Wouldn't a more promising approach be to go through NeRFs first? So first figure out what you want the model to look like from various angles, and then try to make a textured 3d model out of it.

georyb 3y ago

I'm curios how much more difficult is generative 3D? We are barely able to generate 2D 1080p images, without artefacts using diffusion methods. How much more complex is the jump to production grade 3D? As someone creating re-usable 3D product models on Polymock @ https://polymock.com https://polymock.com the bigest issue for me, is that you want your models to be editable. 3D modellers want to post-process and add tweaks to existing 3D models. Current AI generators lack this layering structures. What I actually want, is a 3D generator for textures / materials. It should be a much simpler task for current AI. At the moment, the state of 3D generators is slowly ripening for indie game developers. I'm sure that in the near future, the models will be high-enough quality to generate a basic character set. Others have mentioned it, but I think NVIDIA has an edge in 3D synthesis & rendering. OpenAI is arguable the leader in consumer text-transformers - I wonder the goal for releasing this POC (I don't think they will allocate any serious resource here)

placeholder_0 3y ago
Have you tried texturelab.xyz?
- georyb 3y ago
  Thanks for sharing - looks handy for games
prox 3y ago
Someone made a SD/blender hybrid add-on where texturing will be based on a prompt.

joeld42 3y ago

Tip: If you're loading these .glbs in blender, add an "Attribute" node in the shader editor, put "Col" in the name field, and connect "Color" to "Base Color" in the material to see the colors.

EamonnMR 3y ago

The next super Scribblenauts game is going to be wild.