4 ms·

I post this because I think the current neural-net hegemony of AI research underappreciates the importance of consciousness and implicitly assumes that it can j

I post this because I think the current neural-net hegemony of AI research underappreciates the importance of consciousness and implicitly assumes that it can just scale its way to AGI

lostmsu 2y ago
At this point even if it turned out the brain operates on subatomic mechanical gears so small we cannot actually see, it would not in any way affect the reasoning that the current approach is going to scale to AGI. The reasoning is not based on closeness of LLMs to the operation mechanics of the human brain, but on the observed capabilities.
- chrsw 2y ago
  But how far do you have to scale LLMs to get to a conscious system? Would it even be practical?
  - exe34 2y ago
    a conscious system is aware of itself. chat gpt is aware of itself, that's the first thing they tell it in the system prompt.
    - whtsthmttrmn 2y ago
      On an Etch-a-Sketch I can tell it that it's aware of itself...doesn't mean it actually is.
      - exe34 2y ago
        you probably claim to be aware too.
    - slowmovintarget 2y ago
      Chat GPT is not aware. It has no self. It produces token strings that are statistically likely to be a match for any given prompt based on text it's been fed. But the reason it gets things wrong is not the same reason we get things wrong. It gets things wrong because the probabilities matched but the actual meaning did not. It sounds great that RAG stands for Recursively Assembled Grammar (in a discussion about LLMs), only it doesn't. The LLM generated text that even explained RAG as a recursive grammar applied in the context of LLM usage, but RAG stands for Request-Augmented Generation. When I pointed this out the system output "Oh, of course, that is also a thing and it means..." But the system had no awareness. Not of the meaning of the text it was producing, and not of any sort of self.
  - lostmsu 2y ago
    > But how far do you have to scale LLMs to get to a conscious system? Well, that's relatively simple to see. Run a Turing test on models of different sizes and extrapolate. That will give you some approximation of the "distance". > Would it even be practical? You can make a descent educated guess by doing the above.
chrsw 2y ago
Beyond underappreciated, it's essentially being ignored. As far as what is publicly available anyway. I think the explanation for this partially or wholly lies in the goals and values of the people that research these systems vs the goals and values of natural brains. Generally speaking, AI researchers are trying to leverage existing hardware and tooling to demonstrate performance increases on a set of benchmarks. AI founders are trying to apply the research into products to attract investors. Established tech companies are trying to make money. None of these goals and values are anything like what natural brains are trying to do: survive. If AI researchers, founders and tech companies thought artificial consciousness was the fastest path to any of those goals, they'd put resources into that. But they probably think it's an intractable problem, so it gets passed over for things that are likely to have a real payoff in the short term.
- mistrial9 2y ago
  > what natural brains are trying to do and so much more! "survive" does not acknowledge the inspirational, right?
  - almostnormal 2y ago
    Surviving is impossible in the long term. If anything, it's reproduction?
elevaet 2y ago
I think it's a mistake to conflate consciousness with AGI and vice-versa. It's entirely possible that you can have AGI without anyone being "home" and conversely it's also possible that there is a subjective experience going in much simpler systems. We just don't know, and maybe we never can. But regardless, I think it's easy to imagine that there could be a machine that is generally intelligent, or super-generally-intelligent. but doesn't experience.
- bena 2y ago
  It's equally easy to imagine a machine that can't be generally intelligent because it can't experience. In other words, just because you can imagine it, doesn't mean you get to treat your assumptions as fact. What we know right now is that we don't have AGI. Therefore, we currently don't know how to achieve AGI. While it is possible that scaling current methods and models will get us there, it is also possible that it never gets us there. And when you consider that we see examples of general intelligence everywhere, in other people and animals, operating at far less power than it takes to train these models, it's not out of hand to say that we are possibly barking up the wrong tree. Then there is the matter of actually defining general intelligence. It may also be the definition of consciousness, or at least require it. But currently, there is no mutually agreed upon definition of "general intelligence". Some people try to define it as "has a lot of knowledge". In which case, a hard drive can be considered intelligent. Or even a book. But people generally accept neither as intelligent. So "knowledgeable" is not a fitting definition. Something with intelligence must be able to act on its knowledge. And I think that's where consciousness would come into play, if it does. It could be that consciousness is needed to direct knowledge. And one's ability to direct that knowledge is what we would consider intelligence. Which could be why LLMs plateau. They have no internal direction. They are pure knowledge without direction. We supply the direction. And we must navigate the nodes to find the knowledge we can use. LLMs can't really tell truth from falsehoods, we do that. Just like we do internally. We discard untrue things, or things that aren't quite what we were thinking, or some other filter. LLMs just expose that process because now part of our knowledge is contained in the LLM. So, just because it's easy to imagine, it doesn't mean that it's possible.
  - 2y ago
evilduck 2y ago
As far as I'm aware there's no scientific consensus on what consciousness is, how to measure it, what exact mechanisms are in play, or if it even exists at all. Before bothering with the topic of consciousness in AI and AGI, we really need to nail down that definition for humans before squabbling over its presence or absence in something else.

I post this because I think the current neural-net hegemony of AI research underappreciates the importance of consciousness and implicitly assumes that it can just scale its way to AGI

lostmsu 2y ago

At this point even if it turned out the brain operates on subatomic mechanical gears so small we cannot actually see, it would not in any way affect the reasoning that the current approach is going to scale to AGI. The reasoning is not based on closeness of LLMs to the operation mechanics of the human brain, but on the observed capabilities.

chrsw 2y ago
But how far do you have to scale LLMs to get to a conscious system? Would it even be practical?
- exe34 2y ago
  a conscious system is aware of itself. chat gpt is aware of itself, that's the first thing they tell it in the system prompt.
  - whtsthmttrmn 2y ago
    On an Etch-a-Sketch I can tell it that it's aware of itself...doesn't mean it actually is.
    - exe34 2y ago
      you probably claim to be aware too.
  - slowmovintarget 2y ago
    Chat GPT is not aware. It has no self. It produces token strings that are statistically likely to be a match for any given prompt based on text it's been fed. But the reason it gets things wrong is not the same reason we get things wrong. It gets things wrong because the probabilities matched but the actual meaning did not. It sounds great that RAG stands for Recursively Assembled Grammar (in a discussion about LLMs), only it doesn't. The LLM generated text that even explained RAG as a recursive grammar applied in the context of LLM usage, but RAG stands for Request-Augmented Generation. When I pointed this out the system output "Oh, of course, that is also a thing and it means..." But the system had no awareness. Not of the meaning of the text it was producing, and not of any sort of self.
- lostmsu 2y ago
  > But how far do you have to scale LLMs to get to a conscious system? Well, that's relatively simple to see. Run a Turing test on models of different sizes and extrapolate. That will give you some approximation of the "distance". > Would it even be practical? You can make a descent educated guess by doing the above.

chrsw 2y ago

Beyond underappreciated, it's essentially being ignored. As far as what is publicly available anyway. I think the explanation for this partially or wholly lies in the goals and values of the people that research these systems vs the goals and values of natural brains. Generally speaking, AI researchers are trying to leverage existing hardware and tooling to demonstrate performance increases on a set of benchmarks. AI founders are trying to apply the research into products to attract investors. Established tech companies are trying to make money. None of these goals and values are anything like what natural brains are trying to do: survive. If AI researchers, founders and tech companies thought artificial consciousness was the fastest path to any of those goals, they'd put resources into that. But they probably think it's an intractable problem, so it gets passed over for things that are likely to have a real payoff in the short term.

mistrial9 2y ago
> what natural brains are trying to do and so much more! "survive" does not acknowledge the inspirational, right?
- almostnormal 2y ago
  Surviving is impossible in the long term. If anything, it's reproduction?

elevaet 2y ago

I think it's a mistake to conflate consciousness with AGI and vice-versa. It's entirely possible that you can have AGI without anyone being "home" and conversely it's also possible that there is a subjective experience going in much simpler systems. We just don't know, and maybe we never can. But regardless, I think it's easy to imagine that there could be a machine that is generally intelligent, or super-generally-intelligent. but doesn't experience.

bena 2y ago
It's equally easy to imagine a machine that can't be generally intelligent because it can't experience. In other words, just because you can imagine it, doesn't mean you get to treat your assumptions as fact. What we know right now is that we don't have AGI. Therefore, we currently don't know how to achieve AGI. While it is possible that scaling current methods and models will get us there, it is also possible that it never gets us there. And when you consider that we see examples of general intelligence everywhere, in other people and animals, operating at far less power than it takes to train these models, it's not out of hand to say that we are possibly barking up the wrong tree. Then there is the matter of actually defining general intelligence. It may also be the definition of consciousness, or at least require it. But currently, there is no mutually agreed upon definition of "general intelligence". Some people try to define it as "has a lot of knowledge". In which case, a hard drive can be considered intelligent. Or even a book. But people generally accept neither as intelligent. So "knowledgeable" is not a fitting definition. Something with intelligence must be able to act on its knowledge. And I think that's where consciousness would come into play, if it does. It could be that consciousness is needed to direct knowledge. And one's ability to direct that knowledge is what we would consider intelligence. Which could be why LLMs plateau. They have no internal direction. They are pure knowledge without direction. We supply the direction. And we must navigate the nodes to find the knowledge we can use. LLMs can't really tell truth from falsehoods, we do that. Just like we do internally. We discard untrue things, or things that aren't quite what we were thinking, or some other filter. LLMs just expose that process because now part of our knowledge is contained in the LLM. So, just because it's easy to imagine, it doesn't mean that it's possible.
- 2y ago

evilduck 2y ago

As far as I'm aware there's no scientific consensus on what consciousness is, how to measure it, what exact mechanisms are in play, or if it even exists at all. Before bothering with the topic of consciousness in AI and AGI, we really need to nail down that definition for humans before squabbling over its presence or absence in something else.