5 ms·

I think it's a mistake to conflate consciousness with AGI and vice-versa. It's entirely possible that you can have AGI without anyone being "home" and conversel

I think it's a mistake to conflate consciousness with AGI and vice-versa. It's entirely possible that you can have AGI without anyone being "home" and conversely it's also possible that there is a subjective experience going in much simpler systems. We just don't know, and maybe we never can. But regardless, I think it's easy to imagine that there could be a machine that is generally intelligent, or super-generally-intelligent. but doesn't experience.

bena 2y ago
It's equally easy to imagine a machine that can't be generally intelligent because it can't experience. In other words, just because you can imagine it, doesn't mean you get to treat your assumptions as fact. What we know right now is that we don't have AGI. Therefore, we currently don't know how to achieve AGI. While it is possible that scaling current methods and models will get us there, it is also possible that it never gets us there. And when you consider that we see examples of general intelligence everywhere, in other people and animals, operating at far less power than it takes to train these models, it's not out of hand to say that we are possibly barking up the wrong tree. Then there is the matter of actually defining general intelligence. It may also be the definition of consciousness, or at least require it. But currently, there is no mutually agreed upon definition of "general intelligence". Some people try to define it as "has a lot of knowledge". In which case, a hard drive can be considered intelligent. Or even a book. But people generally accept neither as intelligent. So "knowledgeable" is not a fitting definition. Something with intelligence must be able to act on its knowledge. And I think that's where consciousness would come into play, if it does. It could be that consciousness is needed to direct knowledge. And one's ability to direct that knowledge is what we would consider intelligence. Which could be why LLMs plateau. They have no internal direction. They are pure knowledge without direction. We supply the direction. And we must navigate the nodes to find the knowledge we can use. LLMs can't really tell truth from falsehoods, we do that. Just like we do internally. We discard untrue things, or things that aren't quite what we were thinking, or some other filter. LLMs just expose that process because now part of our knowledge is contained in the LLM. So, just because it's easy to imagine, it doesn't mean that it's possible.
- elevaet 2y ago
  > It's equally easy to imagine a machine that can't be generally intelligent because it can't experience. I agree with this. I was just pointing out that the parent comment: >I post this because I think the current neural-net hegemony of AI research underappreciates the importance of consciousness and implicitly assumes that it can just scale its way to AGI assumes that you need conciousness for AGI, but we actually don't know if that's true.
  - bena 2y ago
    It's a better bet that it's a requirement however. The only known examples we have of general intelligence comes with consciousness. Not to mention, he's positing that AI researchers are assuming that consciousness is unnecessary and he's saying he disagrees with that position. So saying that he could be wrong is really just circling back to what AI researchers are assuming. Or basically, disagreeing with his disagreement. I would expect either some sort of exposition on why or a third option being presented. I also was not a fan of the position of "I can imagine this therefore it is a valid option". I don't accept the logic that something is necessarily a possibility simply because we can imagine that it can happen.
    - elevaet 2y ago
      It's also true that the only known examples of general intelligence are embodied in meat machines. Is this a prerequisite for AGI? Again, we don't know. I think probably not, but some people think it is, and the debate is unresolved. Similarly, my argument is that it's premature to assume that consciousness is a prerequisite for AGI. Finally, I don't think there's anything invalid about disagreeing with someone's disagreement, and then stating the reason why. In fact you also did this in response to my comment!
      - bena 2y ago
        Until you can show me a counterexample, the null hypothesis is that intelligence requires consciousness. The two sides are not equally weighted. You need to come with something. In your original response, you stated a reason why you disagree. And I pointed out why that reason is not good. It's so low as to not even be a threshold. Other than that, all you've done is reiterate the disagreement. I've given a counter-hypothetical to point out why your reasoning is flawed. I've illustrated reasons why the discussion is complicated. I'm not disagreeing with your disagreement, I'm pointing out not only why I disagree with your premise, but where I believe your premise doesn't hold. So far, the only thing you've offered in response is "Well, maybe it's not required". Why do you believe that? Beyond, "We don't know, and I can imagine it". And even then, I'd treat your imagination with a little skepticism. Just because you can construct the sentence "AGI does not require consciousness" does not mean you can actually conceptualize what that means. Mostly because it would require defining both AGI and consciousness in a mutually agreed upon way. And defining it in a way that would definitely include everything we accept as conscious and exclude everything we accept as lacking conscious.

bena 2y ago

It's equally easy to imagine a machine that can't be generally intelligent because it can't experience. In other words, just because you can imagine it, doesn't mean you get to treat your assumptions as fact. What we know right now is that we don't have AGI. Therefore, we currently don't know how to achieve AGI. While it is possible that scaling current methods and models will get us there, it is also possible that it never gets us there. And when you consider that we see examples of general intelligence everywhere, in other people and animals, operating at far less power than it takes to train these models, it's not out of hand to say that we are possibly barking up the wrong tree. Then there is the matter of actually defining general intelligence. It may also be the definition of consciousness, or at least require it. But currently, there is no mutually agreed upon definition of "general intelligence". Some people try to define it as "has a lot of knowledge". In which case, a hard drive can be considered intelligent. Or even a book. But people generally accept neither as intelligent. So "knowledgeable" is not a fitting definition. Something with intelligence must be able to act on its knowledge. And I think that's where consciousness would come into play, if it does. It could be that consciousness is needed to direct knowledge. And one's ability to direct that knowledge is what we would consider intelligence. Which could be why LLMs plateau. They have no internal direction. They are pure knowledge without direction. We supply the direction. And we must navigate the nodes to find the knowledge we can use. LLMs can't really tell truth from falsehoods, we do that. Just like we do internally. We discard untrue things, or things that aren't quite what we were thinking, or some other filter. LLMs just expose that process because now part of our knowledge is contained in the LLM. So, just because it's easy to imagine, it doesn't mean that it's possible.

elevaet 2y ago
> It's equally easy to imagine a machine that can't be generally intelligent because it can't experience. I agree with this. I was just pointing out that the parent comment: >I post this because I think the current neural-net hegemony of AI research underappreciates the importance of consciousness and implicitly assumes that it can just scale its way to AGI assumes that you need conciousness for AGI, but we actually don't know if that's true.
- bena 2y ago
  It's a better bet that it's a requirement however. The only known examples we have of general intelligence comes with consciousness. Not to mention, he's positing that AI researchers are assuming that consciousness is unnecessary and he's saying he disagrees with that position. So saying that he could be wrong is really just circling back to what AI researchers are assuming. Or basically, disagreeing with his disagreement. I would expect either some sort of exposition on why or a third option being presented. I also was not a fan of the position of "I can imagine this therefore it is a valid option". I don't accept the logic that something is necessarily a possibility simply because we can imagine that it can happen.
  - elevaet 2y ago
    It's also true that the only known examples of general intelligence are embodied in meat machines. Is this a prerequisite for AGI? Again, we don't know. I think probably not, but some people think it is, and the debate is unresolved. Similarly, my argument is that it's premature to assume that consciousness is a prerequisite for AGI. Finally, I don't think there's anything invalid about disagreeing with someone's disagreement, and then stating the reason why. In fact you also did this in response to my comment!
    - bena 2y ago
      Until you can show me a counterexample, the null hypothesis is that intelligence requires consciousness. The two sides are not equally weighted. You need to come with something. In your original response, you stated a reason why you disagree. And I pointed out why that reason is not good. It's so low as to not even be a threshold. Other than that, all you've done is reiterate the disagreement. I've given a counter-hypothetical to point out why your reasoning is flawed. I've illustrated reasons why the discussion is complicated. I'm not disagreeing with your disagreement, I'm pointing out not only why I disagree with your premise, but where I believe your premise doesn't hold. So far, the only thing you've offered in response is "Well, maybe it's not required". Why do you believe that? Beyond, "We don't know, and I can imagine it". And even then, I'd treat your imagination with a little skepticism. Just because you can construct the sentence "AGI does not require consciousness" does not mean you can actually conceptualize what that means. Mostly because it would require defining both AGI and consciousness in a mutually agreed upon way. And defining it in a way that would definitely include everything we accept as conscious and exclude everything we accept as lacking conscious.