5 ms·

> We think, that is a fact. It likely is a fact, but we don't really know what we mean by "think". LLMs have illuminated this point from a relatively new dire

> We think, that is a fact. It likely is a fact, but we don't really know what we mean by "think". LLMs have illuminated this point from a relatively new direction: we do not know if their mechanism(s) for language generation are similar to our own, or not. We don't really understand the relationship between "reasoning" and "thinking". We don't really understand the difference between Kahneman's "fast" and "slow" thinking. Something happens, probably in our brains, that we experience and that seems causally prior to some of our behavior. We call it thinking, but we don't know much about what it actually is.

jay_kyburz 2y ago
I don't think its useful or even interesting to talk about AI in relation to how humans think, or whether or not they will be "conscious" whatever that might mean. AIs are not going to be like humans because they will have perfect recall of a massive database of facts, and be able to do math well beyond any human brain. The interesting question to me is, when will we be able to give AI very large tasks, and when will it to be able to break the tasks down into smaller and smaller tasks and complete them. When will it be able to set its own goals, and know when it has achieved them? When will it be able to recognize that it doesn't know something and do the work to fill in the blanks. I get the impression that LLMs don't really know what they are saying at the moment, so don't have any way to test what they are saying is true or not.
- unparagoned 2y ago
  There was a study where they trained a model to lie. When they looked at what was happening internally they could see it knew the truth but just switched things at the outer nodes to lie.
agarren 2y ago
I think we have a pretty good idea that we are not stochastic parrots - sophisticated or not. Anyone suggesting that we’re running billion parameter models in order to bang out a snarky comment is probably trying to sell you something (and crypto’s likely involved.) I think you’re right, LLMs have demonstrated that relatively sophisticated mathematics involving billions of params and an internet full of training data is capable of some truly, truly, remarkable things. But as Penrose is saying, there are provable limits to computation. If we’re going to assume that intelligence as we experience it is computable, then Gödel’s theorem (and, frankly, the field of mathematics) seems to present a problem.
- PaulDavisThe1st 2y ago
  I've never had any time for Penrose. Gödel’s theorem "merely" asserts that in any system capable of a specific form of expression there are statements which are true but not provable. What this has to do with (a) limits to computation or (b) human intelligence has never been clear to me, despite four decades or more of interest in the topic. There's no reason I can see why we should think that humans are somehow without computational limits. Whether our limits correspond to Gödel’s theorem or not is mildly interesting, but not really foundational from my perspective.
  - Marazan 2y ago
    At the end of the day Penrose's arguments is just Dualism. Humans have a special thingy that makes the consciousness Computers do not have the special thingy Therefore Computers cannot be consciousness. But Dualism gets you laughed at these days so Dualists have to code their arguments and pretend they aren't into that there Dualism. Penrose's arguments against AI has always felt to me like special pleading that humans (or to stretch a bit further, carbon based lifeforms) are unique.
    - tsimionescu 2y ago
      While I don't like Penrose's argument and I think it stands on very shaky ground, I very much disagree it's a form of dualism. His argument is simply that human thinking is not reducible to a Turing machine, that it is a form of hyper-computation. If this were to be true, it would follow that computers as we build them today would fundamentally not be able to match human problem-solving. But it would not follow, in any way, that it would be impossible to build "hyper computers" that do. It just means you wouldn't have any chance of getting there with current technology. Now, I don't think Penrose's arguments for why he thinks this is the case are very strong. But they're definitely not mystical dualistic arguments, they're completely materialistic mathematical arguments. I think he leans towards an idea that quantum mechanics has a way of making more-than-Turing computation happen (note that this is not about what we call quantum computers, which are fully Turing-equivalent systems, just more efficient for certain problems), and that this is how our brains actually function.

jay_kyburz 2y ago

I don't think its useful or even interesting to talk about AI in relation to how humans think, or whether or not they will be "conscious" whatever that might mean. AIs are not going to be like humans because they will have perfect recall of a massive database of facts, and be able to do math well beyond any human brain. The interesting question to me is, when will we be able to give AI very large tasks, and when will it to be able to break the tasks down into smaller and smaller tasks and complete them. When will it be able to set its own goals, and know when it has achieved them? When will it be able to recognize that it doesn't know something and do the work to fill in the blanks. I get the impression that LLMs don't really know what they are saying at the moment, so don't have any way to test what they are saying is true or not.

unparagoned 2y ago
There was a study where they trained a model to lie. When they looked at what was happening internally they could see it knew the truth but just switched things at the outer nodes to lie.

agarren 2y ago

I think we have a pretty good idea that we are not stochastic parrots - sophisticated or not. Anyone suggesting that we’re running billion parameter models in order to bang out a snarky comment is probably trying to sell you something (and crypto’s likely involved.) I think you’re right, LLMs have demonstrated that relatively sophisticated mathematics involving billions of params and an internet full of training data is capable of some truly, truly, remarkable things. But as Penrose is saying, there are provable limits to computation. If we’re going to assume that intelligence as we experience it is computable, then Gödel’s theorem (and, frankly, the field of mathematics) seems to present a problem.

PaulDavisThe1st 2y ago
I've never had any time for Penrose. Gödel’s theorem "merely" asserts that in any system capable of a specific form of expression there are statements which are true but not provable. What this has to do with (a) limits to computation or (b) human intelligence has never been clear to me, despite four decades or more of interest in the topic. There's no reason I can see why we should think that humans are somehow without computational limits. Whether our limits correspond to Gödel’s theorem or not is mildly interesting, but not really foundational from my perspective.
- Marazan 2y ago
  At the end of the day Penrose's arguments is just Dualism. Humans have a special thingy that makes the consciousness Computers do not have the special thingy Therefore Computers cannot be consciousness. But Dualism gets you laughed at these days so Dualists have to code their arguments and pretend they aren't into that there Dualism. Penrose's arguments against AI has always felt to me like special pleading that humans (or to stretch a bit further, carbon based lifeforms) are unique.
  - tsimionescu 2y ago
    While I don't like Penrose's argument and I think it stands on very shaky ground, I very much disagree it's a form of dualism. His argument is simply that human thinking is not reducible to a Turing machine, that it is a form of hyper-computation. If this were to be true, it would follow that computers as we build them today would fundamentally not be able to match human problem-solving. But it would not follow, in any way, that it would be impossible to build "hyper computers" that do. It just means you wouldn't have any chance of getting there with current technology. Now, I don't think Penrose's arguments for why he thinks this is the case are very strong. But they're definitely not mystical dualistic arguments, they're completely materialistic mathematical arguments. I think he leans towards an idea that quantum mechanics has a way of making more-than-Turing computation happen (note that this is not about what we call quantum computers, which are fully Turing-equivalent systems, just more efficient for certain problems), and that this is how our brains actually function.