5 ms·

Here's how I see it, but I'm not sure how valid my mental model is. Imagine a source corpus that consists of: Cows are big. Big animals are happy. Some other

by koliber 11mo ago

Here's how I see it, but I'm not sure how valid my mental model is. Imagine a source corpus that consists of: Cows are big. Big animals are happy. Some other big animals include pigs, horses, and whales. A Markov chain can only return verbatim combinations. So it might return "Cows are big animals" or "Are big animals happy". An LLM can get a sense of meaning in these words and can return ideas expressed in the input corpus. So in this case it might say "Pigs and horses are happy". It's not limited to responding with verbatim sequences. It can be seen as a bit more creative. However, LLMs will not be able to represent ideas that it has not encountered before. It won't be able to come up with truly novel concepts, or even ask questions about them. Humans (some at least) have that unbounded creativity that LLMs do not.

marcellus23 11mo ago
> A Markov chain can only return verbatim combinations. So it might return "Cows are big animals" or "Are big animals happy". Just for my own edification, do you mean "Are big animals are happy"? "animals happy" never shows up in the source text so "happy" would not be a possible successor to "animals", correct?
- koliber 11mo ago
  Please forgive me. I am not a Markov chain.
vidarh 11mo ago
> However, LLMs will not be able to represent ideas that it has not encountered before. It won't be able to come up with truly novel concepts, or even ask questions about them. Humans (some at least) have that unbounded creativity that LLMs do not. There's absolutely no evidence to support this claim. It'd require humans to exceed the Turing computable, and we have no evidence that is possible.
- koliber 11mo ago
  If you tell me that trees are big, and trees are made of hard wood, I as a human am capable of asking whether trees feel pain. I don't think what you said is false and I am not familiar with computational theory to be able to debate it. People occasionally have novel creative insights that do not derive from past experience or knowledge, and that is what I think of when I think of creativity. Humans created novel concepts like writing literally out of thin air. I like how the book "Guns, Steels, and Germs" describes that novel creative process and contrasts it via a disseminative derivation process.
  - vidarh 11mo ago
    > People occasionally have novel creative insights that do not derive from past experience or knowledge, and that is what I think of when I think of creativity. If they are not derived from past experience or knowledge, then unless humans exceed the Turing computable, they would need to be the result of randomness in one form or other. There's absolutely no reason why an LLM can not do that. The only reason a far "dumber" pure random number generator based string generator "can't" do that is because it would take too long to chance on something coherent, but it most certainly would keep spitting out novel things. The only difference is how coherent the novel things are.
    - Jensson 11mo ago
      > If they are not derived from past experience or knowledge Every animal is born with intuition, you missed that part.
      - vidarh 11mo ago
        So knowledge encoded in the physical structure of the brain. You're missing the part where unless there is unknown physics going on in the brain that breaks maths as me know it, there is no mechanism for a brain to exceed the Turing computable, in which case any Turing complete system is comptationally equivalent to it.
        deleted 11mo ago
- somenameforme 11mo ago
  Turing computability is tangential to his claim, as LLMs are obviously not carrying out the breadth of all computable concepts. His claim can be trivially proven by considering the history of humanity. We went from a starting point of having literally no language whatsoever, and technology that would not have expanded much beyond an understanding of 'poke him with the pointy side'. And from there we would go on to discover the secrets of the atom, put a man on the Moon, and more. To say nothing of inventing language itself. An LLM trained on this starting state of humanity is never going to do anything except remix basically nothing. It's never going to discover the secrets of the atom, or how to put a man on the Moon. Now whether any artificial device could achieve what humans did is where the question of computability comes into play, and that's a much more interesting one. But if we limit ourselves to LLMs, then this is very straight forward to answer.
  - vidarh 11mo ago
    > Turing computability is tangential to his claim, as LLMs are obviously not carrying out the breadth of all computable concepts They don't need to. To be Turing complete a system including an LLM need to be able to simulate a 2-state 3-symbol Turing machine (or the inverse). Any LLM with a loop can satisfy that. If you think Turing computability is tangential to this claim, you don't understand the implications of Turing computability. > His claim can be trivially proven by considering the history of humanity. Then show me a single example where humans demonstrably exceeding the Turing computable. We don't even know any way for that to be possible.
    - somenameforme 11mo ago
      This is akin to claiming that a tic-tac-toe game is turing complete since after all we could simply just modify it to make it not a tic tac toe game. It's not exactly a clever argument. And again there are endless things that seem to reasonably defy turing computability except when you assume your own conclusion. Going from nothing, not even language, to richly communicating, inventing things with no logical basis for such, and so is difficult to even conceive as a computable process unless again you simply assume that it must be computable. For a more common example that rapidly enters into the domain of philosophy - there is the nature of consciousness. It's impossible to prove that such is Turing computable because you can't even prove consciousness exists. The only way I know it exists is because I'm most certainly conscious, and I assume you are too, but you can never prove that to me, anymore than I could ever prove I'm conscious to you. And so now we enter into the domain of trying to computationally imagine something which you can't even prove exists, it's all just a complete nonstarter. ----- I'd also add here that I think the current consensus among those in AI is implicit agreement with this issue. If we genuinely wanted AGI it would make vastly more sense to start from as little as possible because it'd ostensibly reduce computational and other requirements by many orders of magnitude, and we could likely also help create a more controllable and less biased model by starting from a bare minimum of first principles. And there's potentially trillions of dollars for anybody that could achieve this. Instead, we get everything dumped into token prediction algorithms which are inherently limited in potential.
- Fargren 11mo ago
  You are making a big assumption here, which is that LLMs are the main "algorithm" that the human brain uses. The human brain can easily be a Turing machine, that's "running" something that's not an LLM. If that's the case, we can say that the fact that humans can come up with novel concept does not imply that LLMs can do the same.
  - vidarh 11mo ago
    No, I am not assuming anything about the structure of the human brain. The point of talking about Turing completeness is that any universal Turing machine can emulate any other (Turing equivalence). This is fundamental to the theory of computation. And since we can easily show that both can be rigged up in ways that makes the system Turing complete, for humans to be "special", we would need to be able to be more than Turing complete. There is no evidence to suggest we are, and no evidence to suggest that is even possible.
    - Fargren 11mo ago
      An LLM is not a universal Turing machine, though. It's a specific family of algorithms. You can't build an LLM that will factorize arbitrarily large numbers, even in infinite time. But a Turing machine can.
      - vidarh 11mo ago
        To make a universal Turing machine out of an LLM only requires a loop and the ability to make a model that will look up a 2x3 matrix of operations based on context and output operations to the context on the basis of them (the smallest Turing machine has 2 states and 3 symbols or the inverse). So, yes, you can. Once you have a (2,3) Turing machine, you can from that build a model that models any larger Turing machine - it's just a question of allowing it enough computation and enough layers. It is not guaranteed that any specific architecture can do it efficiently, but that is entirely besides the point.
        johnisgood 11mo ago
fragmede 11mo ago
> However, LLMs will not be able to represent ideas that it has not encountered before. Sure they do. We call them hallucinations and complain that they're not true, however.
- koliber 11mo ago
  Hmmm. Didn't think about that. In people there is a difference between unconscious hallucinations vs. intentional creativity. However, there might be situations where they're not distinguishable. In LLMs, it's hard to talk about intentionality. I love where you took this.
  - gishh 11mo ago
    A hallucination isn’t a creative new idea, it’s blatantly wrong information, provably. If an LLM had actual intellectual ability it could tell “us” how we can improve models. They can’t. They’re literally defined by their token count and they use statistics to generate token chains. They’re as creative as the most statistically relevant token chains they’ve been trained on by _people_ who actually used intelligence to type words on a keyboard.
- anonzzzies 11mo ago
  Or, 10000000s times a day while coding all over the world and it hallucinating something it never saw before which turned out to be the thing needed.
- johnisgood 11mo ago
  Hallucinations are not novel ideas. They are novel combinations of tokens constrained by learned probability distributions. I have mentioned Hume before, and will do so again. You can combine "golden" and "mountain" without seeing a golden mountain, but you cannot conjure "golden" without having encountered something that gave you the concept. LLMs may generate strings they have not seen, but those strings are still composed entirely from training-derived representations. The model can output "quantum telepathic blockchain" but each token's semantic content comes from training data. It is recombination, not creation. The model has not built representations of concepts it never encountered in training; it is just sampling poorly constrained combinations. Can you distinguish between a false hallucination and a genuinely novel conceptual representation?

by koliber 11mo ago

marcellus23 11mo ago

> A Markov chain can only return verbatim combinations. So it might return "Cows are big animals" or "Are big animals happy". Just for my own edification, do you mean "Are big animals are happy"? "animals happy" never shows up in the source text so "happy" would not be a possible successor to "animals", correct?

koliber 11mo ago
Please forgive me. I am not a Markov chain.

vidarh 11mo ago

> However, LLMs will not be able to represent ideas that it has not encountered before. It won't be able to come up with truly novel concepts, or even ask questions about them. Humans (some at least) have that unbounded creativity that LLMs do not. There's absolutely no evidence to support this claim. It'd require humans to exceed the Turing computable, and we have no evidence that is possible.

koliber 11mo ago
If you tell me that trees are big, and trees are made of hard wood, I as a human am capable of asking whether trees feel pain. I don't think what you said is false and I am not familiar with computational theory to be able to debate it. People occasionally have novel creative insights that do not derive from past experience or knowledge, and that is what I think of when I think of creativity. Humans created novel concepts like writing literally out of thin air. I like how the book "Guns, Steels, and Germs" describes that novel creative process and contrasts it via a disseminative derivation process.
- vidarh 11mo ago
  > People occasionally have novel creative insights that do not derive from past experience or knowledge, and that is what I think of when I think of creativity. If they are not derived from past experience or knowledge, then unless humans exceed the Turing computable, they would need to be the result of randomness in one form or other. There's absolutely no reason why an LLM can not do that. The only reason a far "dumber" pure random number generator based string generator "can't" do that is because it would take too long to chance on something coherent, but it most certainly would keep spitting out novel things. The only difference is how coherent the novel things are.
  - Jensson 11mo ago
    > If they are not derived from past experience or knowledge Every animal is born with intuition, you missed that part.
    - vidarh 11mo ago
      So knowledge encoded in the physical structure of the brain. You're missing the part where unless there is unknown physics going on in the brain that breaks maths as me know it, there is no mechanism for a brain to exceed the Turing computable, in which case any Turing complete system is comptationally equivalent to it.
      - deleted 11mo ago
somenameforme 11mo ago
Turing computability is tangential to his claim, as LLMs are obviously not carrying out the breadth of all computable concepts. His claim can be trivially proven by considering the history of humanity. We went from a starting point of having literally no language whatsoever, and technology that would not have expanded much beyond an understanding of 'poke him with the pointy side'. And from there we would go on to discover the secrets of the atom, put a man on the Moon, and more. To say nothing of inventing language itself. An LLM trained on this starting state of humanity is never going to do anything except remix basically nothing. It's never going to discover the secrets of the atom, or how to put a man on the Moon. Now whether any artificial device could achieve what humans did is where the question of computability comes into play, and that's a much more interesting one. But if we limit ourselves to LLMs, then this is very straight forward to answer.
- vidarh 11mo ago
  > Turing computability is tangential to his claim, as LLMs are obviously not carrying out the breadth of all computable concepts They don't need to. To be Turing complete a system including an LLM need to be able to simulate a 2-state 3-symbol Turing machine (or the inverse). Any LLM with a loop can satisfy that. If you think Turing computability is tangential to this claim, you don't understand the implications of Turing computability. > His claim can be trivially proven by considering the history of humanity. Then show me a single example where humans demonstrably exceeding the Turing computable. We don't even know any way for that to be possible.
  - somenameforme 11mo ago
    This is akin to claiming that a tic-tac-toe game is turing complete since after all we could simply just modify it to make it not a tic tac toe game. It's not exactly a clever argument. And again there are endless things that seem to reasonably defy turing computability except when you assume your own conclusion. Going from nothing, not even language, to richly communicating, inventing things with no logical basis for such, and so is difficult to even conceive as a computable process unless again you simply assume that it must be computable. For a more common example that rapidly enters into the domain of philosophy - there is the nature of consciousness. It's impossible to prove that such is Turing computable because you can't even prove consciousness exists. The only way I know it exists is because I'm most certainly conscious, and I assume you are too, but you can never prove that to me, anymore than I could ever prove I'm conscious to you. And so now we enter into the domain of trying to computationally imagine something which you can't even prove exists, it's all just a complete nonstarter. ----- I'd also add here that I think the current consensus among those in AI is implicit agreement with this issue. If we genuinely wanted AGI it would make vastly more sense to start from as little as possible because it'd ostensibly reduce computational and other requirements by many orders of magnitude, and we could likely also help create a more controllable and less biased model by starting from a bare minimum of first principles. And there's potentially trillions of dollars for anybody that could achieve this. Instead, we get everything dumped into token prediction algorithms which are inherently limited in potential.
Fargren 11mo ago
You are making a big assumption here, which is that LLMs are the main "algorithm" that the human brain uses. The human brain can easily be a Turing machine, that's "running" something that's not an LLM. If that's the case, we can say that the fact that humans can come up with novel concept does not imply that LLMs can do the same.
- vidarh 11mo ago
  No, I am not assuming anything about the structure of the human brain. The point of talking about Turing completeness is that any universal Turing machine can emulate any other (Turing equivalence). This is fundamental to the theory of computation. And since we can easily show that both can be rigged up in ways that makes the system Turing complete, for humans to be "special", we would need to be able to be more than Turing complete. There is no evidence to suggest we are, and no evidence to suggest that is even possible.
  - Fargren 11mo ago
    An LLM is not a universal Turing machine, though. It's a specific family of algorithms. You can't build an LLM that will factorize arbitrarily large numbers, even in infinite time. But a Turing machine can.
    - vidarh 11mo ago
      To make a universal Turing machine out of an LLM only requires a loop and the ability to make a model that will look up a 2x3 matrix of operations based on context and output operations to the context on the basis of them (the smallest Turing machine has 2 states and 3 symbols or the inverse). So, yes, you can. Once you have a (2,3) Turing machine, you can from that build a model that models any larger Turing machine - it's just a question of allowing it enough computation and enough layers. It is not guaranteed that any specific architecture can do it efficiently, but that is entirely besides the point.
      - johnisgood 11mo ago

fragmede 11mo ago

> However, LLMs will not be able to represent ideas that it has not encountered before. Sure they do. We call them hallucinations and complain that they're not true, however.

koliber 11mo ago
Hmmm. Didn't think about that. In people there is a difference between unconscious hallucinations vs. intentional creativity. However, there might be situations where they're not distinguishable. In LLMs, it's hard to talk about intentionality. I love where you took this.
- gishh 11mo ago
  A hallucination isn’t a creative new idea, it’s blatantly wrong information, provably. If an LLM had actual intellectual ability it could tell “us” how we can improve models. They can’t. They’re literally defined by their token count and they use statistics to generate token chains. They’re as creative as the most statistically relevant token chains they’ve been trained on by _people_ who actually used intelligence to type words on a keyboard.
anonzzzies 11mo ago
Or, 10000000s times a day while coding all over the world and it hallucinating something it never saw before which turned out to be the thing needed.
johnisgood 11mo ago
Hallucinations are not novel ideas. They are novel combinations of tokens constrained by learned probability distributions. I have mentioned Hume before, and will do so again. You can combine "golden" and "mountain" without seeing a golden mountain, but you cannot conjure "golden" without having encountered something that gave you the concept. LLMs may generate strings they have not seen, but those strings are still composed entirely from training-derived representations. The model can output "quantum telepathic blockchain" but each token's semantic content comes from training data. It is recombination, not creation. The model has not built representations of concepts it never encountered in training; it is just sampling poorly constrained combinations. Can you distinguish between a false hallucination and a genuinely novel conceptual representation?