5 ms·

> The reason [LLMs are not a good tool to do truly effective brainstorming] is LLMs are trained to follow existing patterns in the human-produced corpus, and no

by dahart 2y ago

> The reason [LLMs are not a good tool to do truly effective brainstorming] is LLMs are trained to follow existing patterns in the human-produced corpus, and not natively taught to “brainstorm”. The problem with this argument is that people do the same thing, we’re not that great at brainstorming either. When we brainstorm in groups, we’re just bringing multiple points of view together. The more data LLMs are trained on, the more viewpoints it might be able to bring that you haven’t considered. That said, LLMs and all NNs so far are built to interpolate, and they are bad and have unbounded error when extrapolating outside their training examples. That is a good reason to not expect today’s AI to come up with new ideas.

trueismywork 2y ago
We don't need more data, we need independent LLMs with slight randomness.
- dahart 2y ago
  Aren’t independent LLMs just more data? Some AI researchers are arguing that more data is the primary thing that got us this far, and the primary thing that will improve AI from here. Here’s an example (and a very good talk whether you believe my summary) https://www.youtube.com/live/a13aqr07tJ4?si=FZO5m_XzrfpDhyQP https://www.youtube.com/live/a13aqr07tJ4?si=FZO5m_XzrfpDhyQP As a part-time generative artist for several decades, a user of Monte-Carlo methods on a daily basis, and author of some papers on the topic, I personally believe that randomness is not a good answer to anything creative. Randomness is boring, and average. Randomness only helps you when you have a high quality Markov model constraining what your RNG chooses between, and that’s more or less all LLMs actually are. Adding more randomness to creative works in general makes creative works muddy and lowers quality, it needs to be guided. Randomness is a useful tool, but is widely misunderstood IMO and not very effective at brainstorming style exploration; brainstorming is about solving problems in interesting ways, i.e., there is reasoning behind it, not slightly more random word salad.
nequo 2y ago
Setting the temperature to a higher value would emulate this, wouldn't it? Then the model would be more willing to deviate from the most likely next token.
- blackle 2y ago
  Increasing temperature just makes more uncommon tokens more likely to appear. This can include both uncommon ideas and uncommon spelling and grammatical mistakes. It also won't make ideas that the LLM isn't capable of thinking appear, unless you crank the temperature way up and get lucky (like monkeys on typewriters lucky)
  - ein0p 2y ago
    Not just “uncommon tokens”. In the case when generation does weighted random sampling, it autoregressively takes its own past tokens into account. It generates whole different sequences. So if it’s diversity of output you want, you will get it.
  - lanstin 2y ago
    That is also true for creative people. Generating interesting ideas is one thing then filtering out impractical or unaesthetic ideas is also needed.
    - dragonwriter 2y ago
      Right, so in rough outline an LLM-based brainstorming framework might generate multiple responses to a question with multiple different combinations of temperature, top_p, top_k, etc., to get a mix of dull-but-baseline (you don’t want to miss those options) and more off-the-beaten path responses, then take each one and send it back to the LLM with a prompt to evaluate “does this provide a coherent answer – however unconventional or bad of an idea – to the question asked?” to filter out the cases where it was asked for summer vacation ideas and responded with something like “Banana banana banana banana” (this request probably with default tuning.)
XorNot 2y ago
> and have unbounded error when extrapolating outside their training examples I would say, try reading a conspiracy theory forum - or simply wait for one to get posted on HN, and then revisit this particular conclusion. I don't think this is a unique failure of LLMs, except insofar as LLMs have less of a physical grounding in reality then humans do: but we have people out there who have convinced themselves they can project forcefields, and take that all the way through to getting punched in the face by a martial artist when it turns out that, no, they can't.
password_ 2y ago
>people do the same thing how do you know?
- jhbadger 2y ago
  Nothing is really original. That's why you can make finite lists of the possible plots of books and movies. It's why TVtropes is a thing. Everything is just a mashup of things before.
  - hobobaggins 2y ago
    Ah, yes, Everything that can be invented has been invented. - apocryphally attributed to Charles H. Duell, Commissioner of US Patent Office way back in 1889. This is a myth. Just because you don't really see blinding insight every day doesn't mean it doesn't exist (and it's more spectacular when you do see it!) Another reason that this seems to happen is that innovation and invention tend to happen in more esoteric and technical realms today, and so they're only really fully understood by experts in that field.
    - galdosdi 2y ago
      Even Charles Duell wasn't speaking originally here: "What has been will be again, what has been done will be done again; there is nothing new under the sun." - Ecclesiastes 1:9 (circa 250 BC)
    - jhbadger 2y ago
      I happen to actually be a a computational biologist whose work could be considered "esoteric and technical" and whose papers are only fully understood by others in my field. But the fact is every paper of mine (and everybody else's) has 50 to 100 references. Because research is pretty much just reusing existing knowledge in the papers cited. The main source of "innovation", if it can be called that, is taking an idea from another field and applying it to your own. The idea of a scientist or inventor who comes up with an idea completely out the blue with no context is common in fiction, but that isn't how it works in reality.
  - 2y ago
hobobaggins 2y ago
> people do the same thing You're right, most people do that, but that's because they haven't trained themselves to be inventors. This is a skill that definitely needs to be developed more, but finding truly creative (and sometimes backwards-seeming) solutions can sometimes mean thinking about problems in a completely different way. However, all LLM's do this. That's one of the points of the article.
- refulgentis 2y ago
  There was an actual peer-reviewed article from an institution yesterday showing, numerically, robustly, human evaluated, that LLMs are better at creative thinking than humans. I don't think handwaving about exclusively outputting training data and random punditry via article really holds up here. It's wrong for so many reasons, from the training data having more perspectives, to the "Fat Tony" test, empirically, we can go make it output novel combinations right now, to the CS test, we can have it emulate arbitrary Turing machines.
  - wrycoder 2y ago
    Might you possibly have a link?
    - refulgentis 2y ago
      My bad, looked it up and meant to edit it in, dunno what happened: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10858891/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10858891/
      - barfbagginus 2y ago
        The methodology of this paper is seriously suspect. First they have a very small sample size of around n = 150. Second, they don't represent average husband, they use people working on mechanical Turk for $9 an hour. Finally, they don't have any standard measures of divergent thought - they just use 4 different types of question answering problems, and those are supposed to measure creativity. Whether they do or don't would require us to evaluate the questions. It's important to be clear that this is saying that LLMs might be more creative than low paid workers on mechanical Turk, given no training time. That's not a very good representation of the average human, or the fact that humans need training. So for example, an average doctor or scientist might still be more creative than LLMs. Tldr; this is not a good paper to throw around if you want to convince people. It's truly awful in its design, and the way it's presented is not always honest.
pixelsort 2y ago
I came here to say pretty much the same. It is so tiring to hear opinions like these from technically-minded people who clearly lack "better-than-average" creativity and lateral thinking skills. LLMs are garbage-in garbage-out in terms of brainstorming. If you aren't good at brainstorming without LLMs then they won't be much use. So many people are just seeing their own limits reflected back at them. When I'm brainstorming math or cryptography or philosophy with an LLM it is only because of the connections and leaps that I introduce that leads to novel and amusing discoveries. You have to develop successful patterns. Coaxing and condensing. Drilling into numbered lists. Asking for tables with progressively more columns and altered sorting. You have to know the spatial reasoning limits of the model. Asking for enumeration and disabling of social niceties. The number of skills needed is probably beyond most people's imaginations.

by dahart 2y ago

trueismywork 2y ago

We don't need more data, we need independent LLMs with slight randomness.

dahart 2y ago
Aren’t independent LLMs just more data? Some AI researchers are arguing that more data is the primary thing that got us this far, and the primary thing that will improve AI from here. Here’s an example (and a very good talk whether you believe my summary) https://www.youtube.com/live/a13aqr07tJ4?si=FZO5m_XzrfpDhyQP https://www.youtube.com/live/a13aqr07tJ4?si=FZO5m_XzrfpDhyQP As a part-time generative artist for several decades, a user of Monte-Carlo methods on a daily basis, and author of some papers on the topic, I personally believe that randomness is not a good answer to anything creative. Randomness is boring, and average. Randomness only helps you when you have a high quality Markov model constraining what your RNG chooses between, and that’s more or less all LLMs actually are. Adding more randomness to creative works in general makes creative works muddy and lowers quality, it needs to be guided. Randomness is a useful tool, but is widely misunderstood IMO and not very effective at brainstorming style exploration; brainstorming is about solving problems in interesting ways, i.e., there is reasoning behind it, not slightly more random word salad.

nequo 2y ago

Setting the temperature to a higher value would emulate this, wouldn't it? Then the model would be more willing to deviate from the most likely next token.

blackle 2y ago
Increasing temperature just makes more uncommon tokens more likely to appear. This can include both uncommon ideas and uncommon spelling and grammatical mistakes. It also won't make ideas that the LLM isn't capable of thinking appear, unless you crank the temperature way up and get lucky (like monkeys on typewriters lucky)
- ein0p 2y ago
  Not just “uncommon tokens”. In the case when generation does weighted random sampling, it autoregressively takes its own past tokens into account. It generates whole different sequences. So if it’s diversity of output you want, you will get it.
- lanstin 2y ago
  That is also true for creative people. Generating interesting ideas is one thing then filtering out impractical or unaesthetic ideas is also needed.
  - dragonwriter 2y ago
    Right, so in rough outline an LLM-based brainstorming framework might generate multiple responses to a question with multiple different combinations of temperature, top_p, top_k, etc., to get a mix of dull-but-baseline (you don’t want to miss those options) and more off-the-beaten path responses, then take each one and send it back to the LLM with a prompt to evaluate “does this provide a coherent answer – however unconventional or bad of an idea – to the question asked?” to filter out the cases where it was asked for summer vacation ideas and responded with something like “Banana banana banana banana” (this request probably with default tuning.)

XorNot 2y ago

> and have unbounded error when extrapolating outside their training examples I would say, try reading a conspiracy theory forum - or simply wait for one to get posted on HN, and then revisit this particular conclusion. I don't think this is a unique failure of LLMs, except insofar as LLMs have less of a physical grounding in reality then humans do: but we have people out there who have convinced themselves they can project forcefields, and take that all the way through to getting punched in the face by a martial artist when it turns out that, no, they can't.

password_ 2y ago

>people do the same thing how do you know?

jhbadger 2y ago
Nothing is really original. That's why you can make finite lists of the possible plots of books and movies. It's why TVtropes is a thing. Everything is just a mashup of things before.
- hobobaggins 2y ago
  Ah, yes, Everything that can be invented has been invented. - apocryphally attributed to Charles H. Duell, Commissioner of US Patent Office way back in 1889. This is a myth. Just because you don't really see blinding insight every day doesn't mean it doesn't exist (and it's more spectacular when you do see it!) Another reason that this seems to happen is that innovation and invention tend to happen in more esoteric and technical realms today, and so they're only really fully understood by experts in that field.
  - galdosdi 2y ago
    Even Charles Duell wasn't speaking originally here: "What has been will be again, what has been done will be done again; there is nothing new under the sun." - Ecclesiastes 1:9 (circa 250 BC)
  - jhbadger 2y ago
    I happen to actually be a a computational biologist whose work could be considered "esoteric and technical" and whose papers are only fully understood by others in my field. But the fact is every paper of mine (and everybody else's) has 50 to 100 references. Because research is pretty much just reusing existing knowledge in the papers cited. The main source of "innovation", if it can be called that, is taking an idea from another field and applying it to your own. The idea of a scientist or inventor who comes up with an idea completely out the blue with no context is common in fiction, but that isn't how it works in reality.
- 2y ago

hobobaggins 2y ago

> people do the same thing You're right, most people do that, but that's because they haven't trained themselves to be inventors. This is a skill that definitely needs to be developed more, but finding truly creative (and sometimes backwards-seeming) solutions can sometimes mean thinking about problems in a completely different way. However, all LLM's do this. That's one of the points of the article.

refulgentis 2y ago
There was an actual peer-reviewed article from an institution yesterday showing, numerically, robustly, human evaluated, that LLMs are better at creative thinking than humans. I don't think handwaving about exclusively outputting training data and random punditry via article really holds up here. It's wrong for so many reasons, from the training data having more perspectives, to the "Fat Tony" test, empirically, we can go make it output novel combinations right now, to the CS test, we can have it emulate arbitrary Turing machines.
- wrycoder 2y ago
  Might you possibly have a link?
  - refulgentis 2y ago
    My bad, looked it up and meant to edit it in, dunno what happened: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10858891/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10858891/
    - barfbagginus 2y ago
      The methodology of this paper is seriously suspect. First they have a very small sample size of around n = 150. Second, they don't represent average husband, they use people working on mechanical Turk for $9 an hour. Finally, they don't have any standard measures of divergent thought - they just use 4 different types of question answering problems, and those are supposed to measure creativity. Whether they do or don't would require us to evaluate the questions. It's important to be clear that this is saying that LLMs might be more creative than low paid workers on mechanical Turk, given no training time. That's not a very good representation of the average human, or the fact that humans need training. So for example, an average doctor or scientist might still be more creative than LLMs. Tldr; this is not a good paper to throw around if you want to convince people. It's truly awful in its design, and the way it's presented is not always honest.

pixelsort 2y ago

I came here to say pretty much the same. It is so tiring to hear opinions like these from technically-minded people who clearly lack "better-than-average" creativity and lateral thinking skills. LLMs are garbage-in garbage-out in terms of brainstorming. If you aren't good at brainstorming without LLMs then they won't be much use. So many people are just seeing their own limits reflected back at them. When I'm brainstorming math or cryptography or philosophy with an LLM it is only because of the connections and leaps that I introduce that leads to novel and amusing discoveries. You have to develop successful patterns. Coaxing and condensing. Drilling into numbered lists. Asking for tables with progressively more columns and altered sorting. You have to know the spatial reasoning limits of the model. Asking for enumeration and disabling of social niceties. The number of skills needed is probably beyond most people's imaginations.