4 ms·

> Despite impressive capabilities, large language models have yet to produce a genuine breakthrough. The puzzle is why. I don't see why this is remotely surpri

> Despite impressive capabilities, large language models have yet to produce a genuine breakthrough. The puzzle is why. I don't see why this is remotely surprising. Despite all the hoopla, LLMs are not AGI or artifical brains - they are predict-next-word language models. By design they are not built for creativity, but rather quite the opposite, they are designed to continue the input in the way best suggested by the training data - they are essentially built for recall, not creativity. For an AI to be creative it needs to have innate human/brain-like features such as novelty (prediction failure) driven curiosity, boredom, as well as ability to learn continuously. IOW if you want the AI to be creative it needs to be able to learn for itself, not just regurgitate the output of others, and have these innate mechanisms that will cause it to pursue discovery.

NetSurferSu 1y ago
[dead]
karmakaze 1y ago
Yes LLMs choose probable sequences because they recognize similarity. Because of that, it can diverge from similarity to be creative: increase the temperature. What LLMs don't have is (good) taste—we need to build an artificial tongue and feed it as a prerequisite.
- grey-area 1y ago
  Well they also don't have understanding, a model of the world, and the ability to reason (no chain-of-thought created by AI companies is not reasoning), as well as having no taste. So there is quite a lot missing.
- HarHarVeryFunny 1y ago
  It depends on what you mean by "creative" - they can recombine fragments of training data (i.e. apply generative rules) in any order - generate the deductive closure of the training set, but that is it. Without moving beyond LLMs to a more brain-like cognitive architecture, all you can do is squeeze the juice out of the training data, by using RL/etc to bias the generative process (according to reasoning data, good taste or whatever), but you can't move beyond the training data to be truly creative.
  - vonneumannstan 1y ago
    >It depends on what you mean by "creative" - they can recombine fragments of training data (i.e. apply generative rules) in any order - generate the deductive closure of the training set, but that is it. Without moving beyond LLMs to a more brain-like cognitive architecture, all you can do is squeeze the juice out of the training data, but using RL/etc to bias the generative process (according to reasoning data, good taste or whatever), but you can't move beyond the training data to be truly creative. It's clear these models can actually reason on unseen problems and if you don't believe that you aren't actually following the field.
    - HarHarVeryFunny 1y ago
      Sure - but only if the unseen problem can be solved via the deductive/generative closure of the training data. And of course this type of "reasoning" is only as good as the RL pre-training it is based on - working well for closed domains like math where verification is easy, and not so well in the more general case.
    - js8 1y ago
      Both can be true (and that's why I downvoted you in the other comment, for presenting this as a dichotomy), LLMs can reason and yet "stochastically parrot" the training data. For example, LLM might learn a rule that sentences that are similar to "A is given. From A follows B.", are followed by statement "Therefore, B". This is modus ponens. LLM can apply this rule to wide variety of A and B, producing novel statements. Yet, these statements are still the statistically probable ones. I think the problem is, when people say "AI should produce something novel" (or "are producing", depending whether they advocate or dismiss), they are not very clear what the "novel" actually means. Mathematically, it's very easy to produce a never-before-seen theorem; but is it interesting? Probably not.
  - awongh 1y ago
    By volume how much of human speech / writing is pattern matching and how much of it is truly original cognition that would pass your bar of creativity? It is probably 90% rote pattern matching. I don't think LLMs are AGI, but in most senses I don't think people give enough credit to their capabilities. It's just ironic how human-like the flaws of the system are. (Hallucinations that are asserting untrue facts, just because they are plausible from a pattern matching POV)
vonneumannstan 1y ago
> Despite all the hoopla, LLMs are not AGI or artifical brains - they are predict-next-word language models. By design they are not built for creativity, but rather quite the opposite, they are designed to continue the input in the way best suggested by the training data - they are essentially built for recall, not creativity. This is just a completely base level of understanding of LLMs. How do you predict the next token with superhuman accuracy? Really think about how that is possible. If you think it's just stochastic parroting you are ngmi. >large language models have yet to produce a genuine breakthrough. The puzzle is why. I think you should really update on the fact that world class researchers are surprised by this. They understand something you don't and that is that it's clear these models build robust world models and that text prompts act as probes into those world models. The surprising part is that despite these sophisticated world models we can't seem to get unique insights out which almost surely already exist in those models. Even if all the model is capable of is memorizing text then just the sheer volume it has memorized should yield unique insights, no human can ever hope to hold this much text in their memory and then make connections between it. It's possible we just lack the prompt creativity to get these insights out but nevertheless there is something strange happening here.
- HarHarVeryFunny 1y ago
  > This is just a completely base level of understanding of LLMs. How do you predict the next token with superhuman accuracy? Really think about how that is possible. If you think it's just stochastic parroting you are ngmi. Yes, thank-you, I do understand how LLMs work. They learn a lot of generative rules from the training data, and will apply them in flexible fashion according to the context patterns they have learnt. You said stochastic parroting, not me. However, we're not discussing whether LLMs can be superhuman at tasks where they had the necessary training - we're discussing whether they are capable of creativity (and presumably not just the trivially obvious case of being able to apply their generative rules in any order - deductive closure, not stochastic parroting in the dumbest sense of that expression).
  - vonneumannstan 1y ago
    >However, we're not discussing whether LLMs can be superhuman at tasks where they had the necessary training - we're discussing whether they are capable of creativity "Even if all the model is capable of is memorizing text then just the sheer volume it has memorized should yield unique insights, no human can ever hope to hold this much text in their memory and then make connections between it." Unless you think humans have magic meat then all we are really doing with "creativity" is connecting previously unconnected facts.
fragmede 1y ago
Define creativity. Three things LLMs can do is write song lyrics, poems, and jokes, all of which require some level of what we think of as human creativity. Of course detractors will say LLM versions of those three aren't very good, and they may even be right, but a twelve year old child coming up with the same would be seen as creative, even if they didn't get significant recognition for it.
- HarHarVeryFunny 1y ago
  Sure, but the author of TFA is well versed in LLMs and so is addressing something different. Novelty isn't the same as creativity, especially when limited to generating based on a fixed repertoire of moves. The term "deductive closure" has been used to describe what LLMs are capable of, and therefore what they are not capable of. They can generate novelty (e.g. new poem) by applying the rules they have learnt in novel ways, but are ultimately restricted by their fixed weights and what was present in the training data, as well as being biased to predict rather than learn (which they anyways can't!) and explore. An LLM may do a superhuman job of applying what it "knows" to create solutions to novel goals (be that a math olympiad problem, or some type of "creative" output that has been requested, such as a poem), but is unlikely to create a whole new field of math that wasn't hinted at in the training data because it is biased to predict, and anyways doesn't have the ability to learn that would allow it to build a new theory from the ground up one step at a time. Note (for anyone who might claim otherwise) that "in-context learning" is really a misnomer - it's not about learning but rather about using data that is only present in-context rather than having been in the training set.
tmaly 1y ago
I think we will see more breakthroughs with an AI/Human hybrid approach. Tobias Rees had some interesting thoughts https://www.noemamag.com/why-ai-is-a-philosophical-rupture/ https://www.noemamag.com/why-ai-is-a-philosophical-rupture/ where he poses this idea that AI and humans together can think new types of thoughts that humans alone cannot think.

NetSurferSu 1y ago

[dead]

karmakaze 1y ago

Yes LLMs choose probable sequences because they recognize similarity. Because of that, it can diverge from similarity to be creative: increase the temperature. What LLMs don't have is (good) taste—we need to build an artificial tongue and feed it as a prerequisite.

grey-area 1y ago
Well they also don't have understanding, a model of the world, and the ability to reason (no chain-of-thought created by AI companies is not reasoning), as well as having no taste. So there is quite a lot missing.
HarHarVeryFunny 1y ago
It depends on what you mean by "creative" - they can recombine fragments of training data (i.e. apply generative rules) in any order - generate the deductive closure of the training set, but that is it. Without moving beyond LLMs to a more brain-like cognitive architecture, all you can do is squeeze the juice out of the training data, by using RL/etc to bias the generative process (according to reasoning data, good taste or whatever), but you can't move beyond the training data to be truly creative.
- vonneumannstan 1y ago
  >It depends on what you mean by "creative" - they can recombine fragments of training data (i.e. apply generative rules) in any order - generate the deductive closure of the training set, but that is it. Without moving beyond LLMs to a more brain-like cognitive architecture, all you can do is squeeze the juice out of the training data, but using RL/etc to bias the generative process (according to reasoning data, good taste or whatever), but you can't move beyond the training data to be truly creative. It's clear these models can actually reason on unseen problems and if you don't believe that you aren't actually following the field.
  - HarHarVeryFunny 1y ago
    Sure - but only if the unseen problem can be solved via the deductive/generative closure of the training data. And of course this type of "reasoning" is only as good as the RL pre-training it is based on - working well for closed domains like math where verification is easy, and not so well in the more general case.
  - js8 1y ago
    Both can be true (and that's why I downvoted you in the other comment, for presenting this as a dichotomy), LLMs can reason and yet "stochastically parrot" the training data. For example, LLM might learn a rule that sentences that are similar to "A is given. From A follows B.", are followed by statement "Therefore, B". This is modus ponens. LLM can apply this rule to wide variety of A and B, producing novel statements. Yet, these statements are still the statistically probable ones. I think the problem is, when people say "AI should produce something novel" (or "are producing", depending whether they advocate or dismiss), they are not very clear what the "novel" actually means. Mathematically, it's very easy to produce a never-before-seen theorem; but is it interesting? Probably not.
- awongh 1y ago
  By volume how much of human speech / writing is pattern matching and how much of it is truly original cognition that would pass your bar of creativity? It is probably 90% rote pattern matching. I don't think LLMs are AGI, but in most senses I don't think people give enough credit to their capabilities. It's just ironic how human-like the flaws of the system are. (Hallucinations that are asserting untrue facts, just because they are plausible from a pattern matching POV)

vonneumannstan 1y ago

> Despite all the hoopla, LLMs are not AGI or artifical brains - they are predict-next-word language models. By design they are not built for creativity, but rather quite the opposite, they are designed to continue the input in the way best suggested by the training data - they are essentially built for recall, not creativity. This is just a completely base level of understanding of LLMs. How do you predict the next token with superhuman accuracy? Really think about how that is possible. If you think it's just stochastic parroting you are ngmi. >large language models have yet to produce a genuine breakthrough. The puzzle is why. I think you should really update on the fact that world class researchers are surprised by this. They understand something you don't and that is that it's clear these models build robust world models and that text prompts act as probes into those world models. The surprising part is that despite these sophisticated world models we can't seem to get unique insights out which almost surely already exist in those models. Even if all the model is capable of is memorizing text then just the sheer volume it has memorized should yield unique insights, no human can ever hope to hold this much text in their memory and then make connections between it. It's possible we just lack the prompt creativity to get these insights out but nevertheless there is something strange happening here.

HarHarVeryFunny 1y ago
> This is just a completely base level of understanding of LLMs. How do you predict the next token with superhuman accuracy? Really think about how that is possible. If you think it's just stochastic parroting you are ngmi. Yes, thank-you, I do understand how LLMs work. They learn a lot of generative rules from the training data, and will apply them in flexible fashion according to the context patterns they have learnt. You said stochastic parroting, not me. However, we're not discussing whether LLMs can be superhuman at tasks where they had the necessary training - we're discussing whether they are capable of creativity (and presumably not just the trivially obvious case of being able to apply their generative rules in any order - deductive closure, not stochastic parroting in the dumbest sense of that expression).
- vonneumannstan 1y ago
  >However, we're not discussing whether LLMs can be superhuman at tasks where they had the necessary training - we're discussing whether they are capable of creativity "Even if all the model is capable of is memorizing text then just the sheer volume it has memorized should yield unique insights, no human can ever hope to hold this much text in their memory and then make connections between it." Unless you think humans have magic meat then all we are really doing with "creativity" is connecting previously unconnected facts.

fragmede 1y ago

Define creativity. Three things LLMs can do is write song lyrics, poems, and jokes, all of which require some level of what we think of as human creativity. Of course detractors will say LLM versions of those three aren't very good, and they may even be right, but a twelve year old child coming up with the same would be seen as creative, even if they didn't get significant recognition for it.

HarHarVeryFunny 1y ago
Sure, but the author of TFA is well versed in LLMs and so is addressing something different. Novelty isn't the same as creativity, especially when limited to generating based on a fixed repertoire of moves. The term "deductive closure" has been used to describe what LLMs are capable of, and therefore what they are not capable of. They can generate novelty (e.g. new poem) by applying the rules they have learnt in novel ways, but are ultimately restricted by their fixed weights and what was present in the training data, as well as being biased to predict rather than learn (which they anyways can't!) and explore. An LLM may do a superhuman job of applying what it "knows" to create solutions to novel goals (be that a math olympiad problem, or some type of "creative" output that has been requested, such as a poem), but is unlikely to create a whole new field of math that wasn't hinted at in the training data because it is biased to predict, and anyways doesn't have the ability to learn that would allow it to build a new theory from the ground up one step at a time. Note (for anyone who might claim otherwise) that "in-context learning" is really a misnomer - it's not about learning but rather about using data that is only present in-context rather than having been in the training set.

tmaly 1y ago

I think we will see more breakthroughs with an AI/Human hybrid approach. Tobias Rees had some interesting thoughts https://www.noemamag.com/why-ai-is-a-philosophical-rupture/ https://www.noemamag.com/why-ai-is-a-philosophical-rupture/ where he poses this idea that AI and humans together can think new types of thoughts that humans alone cannot think.