3 ms·

Why does AGI necessitate having feelings or consciousness, or the ability to suffer? It seems a bit far to be giving future ultra-advanced calculators legal per

Why does AGI necessitate having feelings or consciousness, or the ability to suffer? It seems a bit far to be giving future ultra-advanced calculators legal personhood?

Retric 2y ago
The general part of general intelligence. If they don’t think in those terms there’s an inherent limitation. Now, something that’s arbitrarily close to AGI but doesn’t care about endlessly working on drudgery etc seems possible, but also a more difficult problem you’d need to be able to build AGI to create.
- AstroBen 2y ago
  Artificial general intelligence (AGI) refers to the hypothetical intelligence of a machine that possesses the ability to understand or learn any intellectual task that a human being can. Generalization ability and Common Sense Knowledge [1] If we go by this definition then there's no caring, or a noticing of drudgery? It's simply defined by its ability to generalize solving problems across domains. The narrow AI that we currently have certainly doesn't care about anything. It does what its programmed to do So one day we figure out how to generalize the problem solving, and enable it to work on a million times harder things.. and suddenly there is sentience and suffering? I don't see it. It's still just a calculator 1- https://cloud.google.com/discover/what-is-artificial-general-intelligence https://cloud.google.com/discover/what-is-artificial-general...
  - Retric 2y ago
    “ability to understand” Isn’t just the ability to preform a task. One of the issues with current AI training is it’s really terrible at discovering which aspects of the training data are false and should be ignored. That requires all kinds of mental tasks to be constantly active including evaluating emotional context to figure out if someone is being deceptive etc.
    - AstroBen 2y ago
      > Isn’t just the ability to preform a task. Right. In this case I'd say it's the ability to interpret data and use it to succeed at whatever goals it has Evaluating emotional context would be similar to a chess engine calculating its next move. There's nothing there that implies a conscience, sentience, morals, feelings, suffering or anything 'human'. It's just a necessary intermediate function to achieve its goal Rob miles has some really good videos on AI safety research which touches on how AGI would think. Thats shaped a lot of how I think about it https://www.youtube.com/watch?v=hEUO6pjwFOo https://www.youtube.com/watch?v=hEUO6pjwFOo
      - Retric 2y ago
        > Evaluating emotional context would be similar to a chess engine calculating its next move. There's nothing there that implies a conscience, sentience, morals, feelings, suffering or anything 'human'. It's just a necessary intermediate function to achieve its goal If it’s limited to achieving goals it’s not AGI. Real time personal goal setting based on human equivalent emotions is an “intellectual task.” One of many requirements for AGI therefore is to A understand the world in real time and B emotionally respond to it. Aka AGI would by definition “necessitate having feelings.” There’s philosophical arguments that there’s something inherently unique about humans here, but without some testable definition you could make the same argument that some arbitrary group of humans don’t have those qualities “gingers have no souls.” Or perhaps “dancing people have no consciousness” which seems like gibberish not because it’s a less defensible argument, but because you haven’t been exposed to it before.
        AstroBen 1y ago
        I mean we just fundamentally have different definitions of AGI. Mine's based on outcomes and what it can do, so purely goal based. Not the processes that mimic humans or animals I think this is the most likely first step of what would happen seeing as we're pushing for it to be created to solve real world problems
        1y ago
  - krupan 2y ago
    It's really hard to picture general intelligence that's useful that doesn't have any intrinsic motivation or initiative. My biggest complaint about LLMs right now is that they lack those things. They don't care even if they give you correct information or not and you have to prompt them for everything! That's not anything close to AGI. I don't know how you get to AGI without it developing preferences, self-motivation and initiative, and I don't know how you then get it to effectively do tasks that it doesn't like, tasks that don't line up with whatever motivates it.
    - deleted 2y ago
      [deleted]
  - imtringued 1y ago
    Exactly. It's called artificial general intelligence, not human general intelligence.
    - Retric 1y ago
      Something can’t “Operate this cat Android, pretending to be a cat.” if it can’t do what I described. A single general intelligence needs to be able to fly an aircraft, get a degree, run a business, and raise a baby to adulthood just like a person or it’s not general.
      - 9rx 1y ago
        So AGI is really about the hardware?
        Retric 1y ago
        We’ve built hardware capable of those things if remotely controlled. It’s the thinking bits that are hard.
        9rx 1y ago
        Only to the extent of having specialized bespoke solutions. We have hardware to fly a plane, but that same hardware isn't able to throw a mortarboard in the air after receiving its degree, and the hardware that can do that isn't able to lactate for a young child. General intelligence is easy compared to general physicality. And, of course, if you keep the hardware specialized to make its creation more tractable, what do you need general intelligence for? Special intelligence that matches the special hardware will work just as well.
        Retric 1y ago
        Flying an aircraft requires talking to air traffic control which existing systems can’t do. Though obviously not a huge issue when the aircraft already has radios, except all those FAA regulations apply to every single aircraft you’ve retrofitting. The advantage of general intelligence is using a small set of hardware now lets you tackle a huge range of tasks or in the above aircraft types. We can mix speakers, eyes, and hands to do a vast array of tasks. Needing new hardware and software for every task very quickly becomes prohibitive.
Workaccount2 2y ago
>Why does AGI necessitate having feelings or consciousness No one knows if it does or not. We don't know why we are conscious and we have no test whatsoever to measure consciousness. In fact the only reason we know that current AI has no consciousness is because "obviously it's not conscious."
- quonn 2y ago
  Excel and Powerpoint are not conscious and so there is not reason to expect any other computation inside a digital computer to be different. You may say something similar for matter and human minds, but we have a very limited and incomplete understanding of the brain and possibly even of the universe. Furthermore we do have a subjective experience of consciousness. On the other hand we have a complete understanding of how LLM inference ultimately maps to matrix multiplications which map to discrete instructions and how those execute on hardware.
  - Filligree 2y ago
    I know I have a subjective experience of consciousness. I’m less sure about you. Simply claiming you do isn’t hard evidence of the fact; after all, LLMs do the same.
    - quonn 1y ago
      If there were evidence that one LLM would be conscious I would also accept it for others. However this is not the case. But we know at least one human is conscious. That‘s convincing me.

Why does AGI necessitate having feelings or consciousness, or the ability to suffer? It seems a bit far to be giving future ultra-advanced calculators legal personhood?

Retric 2y ago

The general part of general intelligence. If they don’t think in those terms there’s an inherent limitation. Now, something that’s arbitrarily close to AGI but doesn’t care about endlessly working on drudgery etc seems possible, but also a more difficult problem you’d need to be able to build AGI to create.

AstroBen 2y ago
Artificial general intelligence (AGI) refers to the hypothetical intelligence of a machine that possesses the ability to understand or learn any intellectual task that a human being can. Generalization ability and Common Sense Knowledge [1] If we go by this definition then there's no caring, or a noticing of drudgery? It's simply defined by its ability to generalize solving problems across domains. The narrow AI that we currently have certainly doesn't care about anything. It does what its programmed to do So one day we figure out how to generalize the problem solving, and enable it to work on a million times harder things.. and suddenly there is sentience and suffering? I don't see it. It's still just a calculator 1- https://cloud.google.com/discover/what-is-artificial-general-intelligence https://cloud.google.com/discover/what-is-artificial-general...
- Retric 2y ago
  “ability to understand” Isn’t just the ability to preform a task. One of the issues with current AI training is it’s really terrible at discovering which aspects of the training data are false and should be ignored. That requires all kinds of mental tasks to be constantly active including evaluating emotional context to figure out if someone is being deceptive etc.
  - AstroBen 2y ago
    > Isn’t just the ability to preform a task. Right. In this case I'd say it's the ability to interpret data and use it to succeed at whatever goals it has Evaluating emotional context would be similar to a chess engine calculating its next move. There's nothing there that implies a conscience, sentience, morals, feelings, suffering or anything 'human'. It's just a necessary intermediate function to achieve its goal Rob miles has some really good videos on AI safety research which touches on how AGI would think. Thats shaped a lot of how I think about it https://www.youtube.com/watch?v=hEUO6pjwFOo https://www.youtube.com/watch?v=hEUO6pjwFOo
    - Retric 2y ago
      > Evaluating emotional context would be similar to a chess engine calculating its next move. There's nothing there that implies a conscience, sentience, morals, feelings, suffering or anything 'human'. It's just a necessary intermediate function to achieve its goal If it’s limited to achieving goals it’s not AGI. Real time personal goal setting based on human equivalent emotions is an “intellectual task.” One of many requirements for AGI therefore is to A understand the world in real time and B emotionally respond to it. Aka AGI would by definition “necessitate having feelings.” There’s philosophical arguments that there’s something inherently unique about humans here, but without some testable definition you could make the same argument that some arbitrary group of humans don’t have those qualities “gingers have no souls.” Or perhaps “dancing people have no consciousness” which seems like gibberish not because it’s a less defensible argument, but because you haven’t been exposed to it before.
      - AstroBen 1y ago
        I mean we just fundamentally have different definitions of AGI. Mine's based on outcomes and what it can do, so purely goal based. Not the processes that mimic humans or animals I think this is the most likely first step of what would happen seeing as we're pushing for it to be created to solve real world problems
        1y ago
- krupan 2y ago
  It's really hard to picture general intelligence that's useful that doesn't have any intrinsic motivation or initiative. My biggest complaint about LLMs right now is that they lack those things. They don't care even if they give you correct information or not and you have to prompt them for everything! That's not anything close to AGI. I don't know how you get to AGI without it developing preferences, self-motivation and initiative, and I don't know how you then get it to effectively do tasks that it doesn't like, tasks that don't line up with whatever motivates it.
  - deleted 2y ago
    [deleted]
- imtringued 1y ago
  Exactly. It's called artificial general intelligence, not human general intelligence.
  - Retric 1y ago
    Something can’t “Operate this cat Android, pretending to be a cat.” if it can’t do what I described. A single general intelligence needs to be able to fly an aircraft, get a degree, run a business, and raise a baby to adulthood just like a person or it’s not general.
    - 9rx 1y ago
      So AGI is really about the hardware?
      - Retric 1y ago
        We’ve built hardware capable of those things if remotely controlled. It’s the thinking bits that are hard.
        9rx 1y ago
        Only to the extent of having specialized bespoke solutions. We have hardware to fly a plane, but that same hardware isn't able to throw a mortarboard in the air after receiving its degree, and the hardware that can do that isn't able to lactate for a young child. General intelligence is easy compared to general physicality. And, of course, if you keep the hardware specialized to make its creation more tractable, what do you need general intelligence for? Special intelligence that matches the special hardware will work just as well.
        Retric 1y ago
        Flying an aircraft requires talking to air traffic control which existing systems can’t do. Though obviously not a huge issue when the aircraft already has radios, except all those FAA regulations apply to every single aircraft you’ve retrofitting. The advantage of general intelligence is using a small set of hardware now lets you tackle a huge range of tasks or in the above aircraft types. We can mix speakers, eyes, and hands to do a vast array of tasks. Needing new hardware and software for every task very quickly becomes prohibitive.

Workaccount2 2y ago

>Why does AGI necessitate having feelings or consciousness No one knows if it does or not. We don't know why we are conscious and we have no test whatsoever to measure consciousness. In fact the only reason we know that current AI has no consciousness is because "obviously it's not conscious."

quonn 2y ago
Excel and Powerpoint are not conscious and so there is not reason to expect any other computation inside a digital computer to be different. You may say something similar for matter and human minds, but we have a very limited and incomplete understanding of the brain and possibly even of the universe. Furthermore we do have a subjective experience of consciousness. On the other hand we have a complete understanding of how LLM inference ultimately maps to matrix multiplications which map to discrete instructions and how those execute on hardware.
- Filligree 2y ago
  I know I have a subjective experience of consciousness. I’m less sure about you. Simply claiming you do isn’t hard evidence of the fact; after all, LLMs do the same.
  - quonn 1y ago
    If there were evidence that one LLM would be conscious I would also accept it for others. However this is not the case. But we know at least one human is conscious. That‘s convincing me.