5 ms·

this is cool, but hm, how should I put it... Ultimately there's a relationship between the preciseness in which you want to control something and the underlyin

this is cool, but hm, how should I put it... Ultimately there's a relationship between the preciseness in which you want to control something and the underlying information, as conveyed in language, to describe such precision. Whether you use plain english, or code - ultimately to do things of sufficient precision you will have to be equally precise in your description. I'm sure someone with more time and more knowledge on such things have already formalized this in some information theory paper, but... The point I'm making here is that this is great because a lot of people are doing "simple" things, and now they will be able to do those things without understanding the idiosyncrasies of Blender APIs, but I'm convinced that this will ultimately turn into something equally difficult as blender APIs to do novel things. and WHEN (not if) that happens, I hope users are prepared to learn the Blender APIs, because it will be inevitable. edit: one other thought. I think "language models" are not the right solution ultimately. I think kind of like AI didn't boom until the proper compute was available even though theoretical models and algorithms existed, language models are the crud solution. once we have a loseless way to simply "think" what we want, then a "large thought model" will have less trouble, as there will be less ambiguity in what you want to what is said. right now it's thought -> language -> model. later it will be thought -> model.

capableweb 4y ago
It could also be that something like this is just another tool in the toolbox. Sure, you could spend time trying to understand Blender's Python API, but lots of Blender users are not programmers. It could be really helpful for them to be able to say "Place 10 spot lights at random position within the boundaries of Mesh 3 with random intensity and color" and just have that appear, rather than having to go looking for a plugin that does it for them.
- lwhi 4y ago
  What's the benefit in not being specific, aside from it being easier to request?
  - whitemary 4y ago
    The benefit is that it's easier to request.
  - Morgawr 4y ago
    When it comes to art (audio or visual especially) iterating over a concept with some air of "generality" and "good enough" can have very interesting emerging properties. You get a "vibe" for what you want, but you might not have a perfectly clear idea of the entire composition until you actually see it in front of you, and then you iterate over that. When I write a song I usually just noodle on my guitar with some pre-programmed drums in the background, I just play whatever comes to mind at the time, record it, then listen back to it, change a few things, add a few accents, decide to add another guitar line, maybe shift in fifths or sevenths to add more voices, add a few instruments that fade in and out like strings or brass, etc. Some people might have more methodical approaches to art, and that's fine too, but in my case it's absolutely a 100% exploration effort about stuff that I don't even know I want until I see it in front of my eyes. These tools are amazing for this.
- therouwboat 4y ago
  I dont know why people make this harder than it needs to be, make group of lights, change colors and duplicate, its good enough, no need to use python api.
  - capableweb 4y ago
    Obviously just an example, but even so, unless you already have the workflow established, writing human text describing something could be easier than manually doing it.
crooked-v 4y ago
Or in other words, the "create an adversary capable of defeating Data" problem... just a bit less of an issue (for now) if you get the instructions wrong.
ericd 4y ago
Well, you get precision with imprecise communication via iteration. You walk together toward the correct mutual understanding by successive clarifications.
- nemo44x 4y ago
  There’s also a parameter you can set called “temperature”. This is how you control the free-form aspect of the model. Set it close to 1 and it will bullshit a lot, or rather it will take bigger chances in filling in outside of context and maybe a bit more hyperactive. A value closer to 0 keeps it tight by taking less risk off the context, but can also miss out a bit and loses color.
- roflyear 4y ago
  Sure but GPT certainly cannot actually do this. It can iterate, but in GPT-world that means "give me another response" not "learn from what i'm saying, and give me another response" The issue is, sometimes those two things seem to be the same thing!
  - ericd 4y ago
    Eh? GPT definitely uses the past comments as context, so as you clarify, its responses get much closer to the target. If you haven't tried GPT-4, you absolutely should before having strong opinions about it.
    - serf 4y ago
      parent is only half wrong; given a sufficiently long iteration process any of the chatGPT versions would surely start losing coherency regarding far past requests -- this may be less evident when used within a well confined and (somewhat) easily self-referenced system (like Blender, for example) -- but it's especially evident when trying to prompt GPT to write a fiction story from basics or otherwise work entirely on its own without a place to store outputs and then refer back. tl;dr: it's easier to tell chatgpt to "Rewrite this story: " and then feed back previous outputs when writing a story than it is to get to an acceptable output from massively detailed prompts or long chains of iteration; this trait has far-reaching consequence rather than just writing fiction. I do understand , however, that 'long-term memory' is a very active point of discussion and development.
samstave 4y ago
"Zoom Enhance!" -- Hear me out: One interesting thing would be to describe a scene and get the rough print. But do it in sections, such that you can select and begin to refine sections and elements within the scene to whittle down to the preciseness that pleases you for each element... What would be really interesting will be how geometry nodes can be managed using gpt.
skybrian 4y ago
Chatbots use random number generators to create a variety of output, so it would be a terrible idea to use natural language as "source code" for a code generation tool. Running a chatbot shouldn't happen within a build process, because the results aren't reproducible. Once you have the source code though, you can use a variety of tools to manipulate it and save the result. Using chatbots to make modifications under supervision is fine. You discard bad modifications and save the good ones. This is using natural language for one-offs and source code for reproducible results. It's looking like they will go well together.
- FractalHQ 4y ago
  Can you not provide a seed to an LLM the way you can with some of the image / video generation models I’ve seen?
  - skybrian 4y ago
    Yes, probably, if you got one that runs locally. OpenAI doesn’t have a way to do it in their API. There are other reasons not to do it, like it being an external API that charges money. And even if you got something local and deterministic, the generated code would be less easily tweaked by editing the original prompt than by asking the LLM to make the change you want, or by editing the code directly.
guyomes 4y ago
If the command is not restricted to text only, but use text and geometrical context, you can remove a lot of ambiguities. This is often done in video games with contextual interactions. After all, using the Blender GUI, you can do a lot using only a 2D mouse coordinate and two boutons. So 2D mouse coordinates and text could be better. A nice evolution would be an AI model that can understand natural language instructions, while taking into account where your mouse pointer, how the model is zoomed and oriented, and that has geometric insight of the 3D scene built so far.
sharemywin 4y ago
With a shared vector database and semantic search plugin you can see bunch of prompts other people have created. with a search plugin you can have it find the api docs and have it output interesting parameters and how to use them with examples. with a python REPL plugin you can have it generate 10 variations and run the code for each. with GPT4 and plugins you could describe the output you want to midjourney or something and give the prompt to it to generate it in blender(feed the outputs to some image similarity vector to compare) and have it search through parameter space(or vector space of the prompt) until it finds something pretty close to what you want. given your budget of course.

capableweb 4y ago

It could also be that something like this is just another tool in the toolbox. Sure, you could spend time trying to understand Blender's Python API, but lots of Blender users are not programmers. It could be really helpful for them to be able to say "Place 10 spot lights at random position within the boundaries of Mesh 3 with random intensity and color" and just have that appear, rather than having to go looking for a plugin that does it for them.

lwhi 4y ago
What's the benefit in not being specific, aside from it being easier to request?
- whitemary 4y ago
  The benefit is that it's easier to request.
- Morgawr 4y ago
  When it comes to art (audio or visual especially) iterating over a concept with some air of "generality" and "good enough" can have very interesting emerging properties. You get a "vibe" for what you want, but you might not have a perfectly clear idea of the entire composition until you actually see it in front of you, and then you iterate over that. When I write a song I usually just noodle on my guitar with some pre-programmed drums in the background, I just play whatever comes to mind at the time, record it, then listen back to it, change a few things, add a few accents, decide to add another guitar line, maybe shift in fifths or sevenths to add more voices, add a few instruments that fade in and out like strings or brass, etc. Some people might have more methodical approaches to art, and that's fine too, but in my case it's absolutely a 100% exploration effort about stuff that I don't even know I want until I see it in front of my eyes. These tools are amazing for this.
therouwboat 4y ago
I dont know why people make this harder than it needs to be, make group of lights, change colors and duplicate, its good enough, no need to use python api.
- capableweb 4y ago
  Obviously just an example, but even so, unless you already have the workflow established, writing human text describing something could be easier than manually doing it.

crooked-v 4y ago

Or in other words, the "create an adversary capable of defeating Data" problem... just a bit less of an issue (for now) if you get the instructions wrong.

ericd 4y ago

Well, you get precision with imprecise communication via iteration. You walk together toward the correct mutual understanding by successive clarifications.

nemo44x 4y ago
There’s also a parameter you can set called “temperature”. This is how you control the free-form aspect of the model. Set it close to 1 and it will bullshit a lot, or rather it will take bigger chances in filling in outside of context and maybe a bit more hyperactive. A value closer to 0 keeps it tight by taking less risk off the context, but can also miss out a bit and loses color.
roflyear 4y ago
Sure but GPT certainly cannot actually do this. It can iterate, but in GPT-world that means "give me another response" not "learn from what i'm saying, and give me another response" The issue is, sometimes those two things seem to be the same thing!
- ericd 4y ago
  Eh? GPT definitely uses the past comments as context, so as you clarify, its responses get much closer to the target. If you haven't tried GPT-4, you absolutely should before having strong opinions about it.
  - serf 4y ago
    parent is only half wrong; given a sufficiently long iteration process any of the chatGPT versions would surely start losing coherency regarding far past requests -- this may be less evident when used within a well confined and (somewhat) easily self-referenced system (like Blender, for example) -- but it's especially evident when trying to prompt GPT to write a fiction story from basics or otherwise work entirely on its own without a place to store outputs and then refer back. tl;dr: it's easier to tell chatgpt to "Rewrite this story: " and then feed back previous outputs when writing a story than it is to get to an acceptable output from massively detailed prompts or long chains of iteration; this trait has far-reaching consequence rather than just writing fiction. I do understand , however, that 'long-term memory' is a very active point of discussion and development.

samstave 4y ago

"Zoom Enhance!" -- Hear me out: One interesting thing would be to describe a scene and get the rough print. But do it in sections, such that you can select and begin to refine sections and elements within the scene to whittle down to the preciseness that pleases you for each element... What would be really interesting will be how geometry nodes can be managed using gpt.

skybrian 4y ago

Chatbots use random number generators to create a variety of output, so it would be a terrible idea to use natural language as "source code" for a code generation tool. Running a chatbot shouldn't happen within a build process, because the results aren't reproducible. Once you have the source code though, you can use a variety of tools to manipulate it and save the result. Using chatbots to make modifications under supervision is fine. You discard bad modifications and save the good ones. This is using natural language for one-offs and source code for reproducible results. It's looking like they will go well together.

FractalHQ 4y ago
Can you not provide a seed to an LLM the way you can with some of the image / video generation models I’ve seen?
- skybrian 4y ago
  Yes, probably, if you got one that runs locally. OpenAI doesn’t have a way to do it in their API. There are other reasons not to do it, like it being an external API that charges money. And even if you got something local and deterministic, the generated code would be less easily tweaked by editing the original prompt than by asking the LLM to make the change you want, or by editing the code directly.

guyomes 4y ago

If the command is not restricted to text only, but use text and geometrical context, you can remove a lot of ambiguities. This is often done in video games with contextual interactions. After all, using the Blender GUI, you can do a lot using only a 2D mouse coordinate and two boutons. So 2D mouse coordinates and text could be better. A nice evolution would be an AI model that can understand natural language instructions, while taking into account where your mouse pointer, how the model is zoomed and oriented, and that has geometric insight of the 3D scene built so far.

sharemywin 4y ago

With a shared vector database and semantic search plugin you can see bunch of prompts other people have created. with a search plugin you can have it find the api docs and have it output interesting parameters and how to use them with examples. with a python REPL plugin you can have it generate 10 variations and run the code for each. with GPT4 and plugins you could describe the output you want to midjourney or something and give the prompt to it to generate it in blender(feed the outputs to some image similarity vector to compare) and have it search through parameter space(or vector space of the prompt) until it finds something pretty close to what you want. given your budget of course.