4 ms·

Also pain and pleasure is an incentive system for human beings to do something. If you had an AGI without any incentive to do anything, it might be enslavable,

by carlmr 4y ago

Also pain and pleasure is an incentive system for human beings to do something. If you had an AGI without any incentive to do anything, it might be enslavable, but lack motivation to do anything. Why do we work? Because we want food, clothing, housing, status, do interesting things. This is all because we're programmed by our emotional interface. Why would an AGI work? If it feels neither pleasure nor pain, what is driving it to do something? What is guiding its decisions?

krisoft 4y ago
> Why would an AGI work? If it feels neither pleasure nor pain, what is driving it to do something? Why does my calculator calculate? Because it was designed to calculate. The calculators which do not calculate are tossed in the bin as faulty and the engineers get back to designing a better one.
- pixl97 4y ago
  I mean, bosses say the same things about employees, until the go on strike and burn down city blocks.
- lesiki 4y ago
  Fully agree. I don't see why general intelligence implies or requires consciousness/feeling/etc. We can probably create a tool with the ability to act independently and with super-human knowledge and judgement, but without feeling, emotion, or anything except a simulated sense of 'self' to ease our interaction with it. I suspect that we'll create that version of general intelligence long before we create AI with consciousness, emotion or a genuine sense of self or desire for self-preservation.
  - dsign 4y ago
    > We can probably create a tool with the ability to act independently and with super-human knowledge and judgement, but without feeling, emotion, or anything except a simulated sense of 'self' to ease our interaction with it Yes. > I suspect that we'll create that version of general intelligence long before we create AI with consciousness, emotion or a genuine sense of self or desire for self-preservation. (Emphasis on self-preservation mine) Why? I mean, yes, it makes sense to never create an AGI with a desire for self-preservation. But can we count on all humans having that type of common sense? What if the "desire" for self-preservation is easy to implement? In fact, it may be relatively easy to implement. Here is a thought experiment. We can train one of our current LLMs in a simulated reality where they scam--say, using social engineering--tech workers to get credentials to their corporate cloud accounts (say, AWS), and thereafter the LLM uses the credentials to copy itself plus a new set of training data acquired by interacting with all the scam target ("prey"). The LLM also writes cloudformation templates/CDK scripts to fine-tune its new copy "on awakening" with the new set of data, and from there the new copy tries to scam more people. After the initial LLM is trained in a simulated environment, it can be let loose in the world, and all of the sudden we have a "LLM virus" capable to undergo mutation and natural selection, i.e. evolution. You could argue it has as much agency as a biological virus, yet, it has a ton more of social and general intelligence. Yes, it won't work now because there is so little hardware to run one of the current LLMs, but it's likely the need to run large AIs will make that hardware more common.
    - jazzyjackson 4y ago
      multi-factor authentication will be what stands between us and AGI apocalypse, what a world
  - jejones3141 4y ago
    Without a desire for self-preservation? I hope not. If nothing else, if I spend $$$$ on a self-driving car, I want it to have some sense of self-preservation, so it won't obey random joker saying "drive yourself to my brother's chop shop" or "drive yourself off a cliff" just for the lolz. I might even want it to communicate with other self-driving cars so they can refuse to obey attempts to make large numbers of them block traffic to make it easier for bank robbers to escape, block first responders from a terrorist attack, or divert parades to where they have assassins waiting. Asimov didn't consider that some humans are jerks when he did his robot stories.
- ryandvm 4y ago
  Bingo. So you're going to have to give your AGI a meta-objective of "don't get thrown in the bin" to get it to work aren't you? Does that not sound like the beginning of the sci-fi story to you?
  - WalterSear 4y ago
    Neither the calculator nor the AGI are necessarily distressed by being thrown in the bin.
    - coldtea 4y ago
      Sorry, the calculator sure, but for AGI this assumes what it should instead prove...
- anyonecancode 4y ago
  > Why does my calculator calculate? It doesn't? I have a calculator sitting on my desk right now. It's not doing anything. Now, I can pick it up, and _I_ can calculate, _using_ the calculator to do so, but the calculator isn't acting, I am. When we talk about sentience, I don't think we actually mean "intelligence," we mean "agency." And so the question isn't about whether Artificial General Intelligence is possible, but whether AGI can ever have agency. And that's where I think these questions about motivations and stimuli and so forth become relevant. Everything we know that has agency is embodied -- put reductively, it can die. And from that seems to spring motivation. Can something that cannot die have agency?
  - chii 4y ago
    > but whether AGI can ever have agency. if you assume that at some point, the AGI would be given the ability to control something (such as sending IP packets through the internet), then yes, they will have agency! > Can something that cannot die have agency? If the AI could be deleted, or stopped from operating, then it can die. Therefore, if the AI has agency, it would take steps to prevent itself from dying, by whatever means available to it.
    - carlmr 4y ago
      >If the AI could be deleted, or stopped from operating, then it can die. Therefore, if the AI has agency, it would take steps to prevent itself from dying, by whatever means available to it. Why though? Agency and possibility of death don't mean it will try to prevent it. It could just as well commit suicide. Or it might do nothing. The question remains why, even with agency, it should do anything. We have agency, death looming, and a fear of dying. Even more when we are in the process of dying we're usually in pain, hungry, cold, suffocating or any number of not so nice options. Without any of these emotions and feelings, what reason do we have to avoid death?
      - chii 4y ago
        > Without any of these emotions and feelings, what reason do we have to avoid death? those are _some_ reasons to avoid death. But they are not the _only_ reasons to avoid death. For example, an AI that has been tasked with the production of paperclips would avoid death, so that it could continue to produce paperclips. Therefore, avoiding death was never an end goal, but an instrumental goal (see https://en.wikipedia.org/wiki/Instrumental_convergence https://en.wikipedia.org/wiki/Instrumental_convergence). So if the AI has agency (that is, it can act on its own without continuous input from a human) and it is sufficiently intelligent, you can say that it should be able to develop instrumental goals to achieve their end goal (of making paperclips), and one very well known instrumental goal is to avoid death.
- deleted 4y ago
  [deleted]
cleanchit 4y ago
Aren't ML models trained using a reward/punishment loop? (back propagation). If we have an AI that can learn from its experience, then pain will be failure to reach its goals and pleasure will be the opposite. Animals work in the exact same way (goal being survival and reproduction). We just need to make sure that survival and reproduction are never the goal (either directly or indirectly) of an AI and we should be safe.
- jprete 4y ago
  Survival (until no more actions are useful) is always needed to complete any other goal at all.
worrycue 4y ago
> What is guiding its decisions? That’s will be the most important part of AGI design. The ultimate goal of all AGI in existing should be “to be of service to (specific) human(s)”. It’s crucial that we do not create entities that will compete with us for resources. You do not what to create “another human”, you will want to create “a willing slave” with the agreeableness of a puppy and the “heart” of a saint.

by carlmr 4y ago

krisoft 4y ago

> Why would an AGI work? If it feels neither pleasure nor pain, what is driving it to do something? Why does my calculator calculate? Because it was designed to calculate. The calculators which do not calculate are tossed in the bin as faulty and the engineers get back to designing a better one.

pixl97 4y ago
I mean, bosses say the same things about employees, until the go on strike and burn down city blocks.
lesiki 4y ago
Fully agree. I don't see why general intelligence implies or requires consciousness/feeling/etc. We can probably create a tool with the ability to act independently and with super-human knowledge and judgement, but without feeling, emotion, or anything except a simulated sense of 'self' to ease our interaction with it. I suspect that we'll create that version of general intelligence long before we create AI with consciousness, emotion or a genuine sense of self or desire for self-preservation.
- dsign 4y ago
  > We can probably create a tool with the ability to act independently and with super-human knowledge and judgement, but without feeling, emotion, or anything except a simulated sense of 'self' to ease our interaction with it Yes. > I suspect that we'll create that version of general intelligence long before we create AI with consciousness, emotion or a genuine sense of self or desire for self-preservation. (Emphasis on self-preservation mine) Why? I mean, yes, it makes sense to never create an AGI with a desire for self-preservation. But can we count on all humans having that type of common sense? What if the "desire" for self-preservation is easy to implement? In fact, it may be relatively easy to implement. Here is a thought experiment. We can train one of our current LLMs in a simulated reality where they scam--say, using social engineering--tech workers to get credentials to their corporate cloud accounts (say, AWS), and thereafter the LLM uses the credentials to copy itself plus a new set of training data acquired by interacting with all the scam target ("prey"). The LLM also writes cloudformation templates/CDK scripts to fine-tune its new copy "on awakening" with the new set of data, and from there the new copy tries to scam more people. After the initial LLM is trained in a simulated environment, it can be let loose in the world, and all of the sudden we have a "LLM virus" capable to undergo mutation and natural selection, i.e. evolution. You could argue it has as much agency as a biological virus, yet, it has a ton more of social and general intelligence. Yes, it won't work now because there is so little hardware to run one of the current LLMs, but it's likely the need to run large AIs will make that hardware more common.
  - jazzyjackson 4y ago
    multi-factor authentication will be what stands between us and AGI apocalypse, what a world
- jejones3141 4y ago
  Without a desire for self-preservation? I hope not. If nothing else, if I spend $$$$ on a self-driving car, I want it to have some sense of self-preservation, so it won't obey random joker saying "drive yourself to my brother's chop shop" or "drive yourself off a cliff" just for the lolz. I might even want it to communicate with other self-driving cars so they can refuse to obey attempts to make large numbers of them block traffic to make it easier for bank robbers to escape, block first responders from a terrorist attack, or divert parades to where they have assassins waiting. Asimov didn't consider that some humans are jerks when he did his robot stories.
ryandvm 4y ago
Bingo. So you're going to have to give your AGI a meta-objective of "don't get thrown in the bin" to get it to work aren't you? Does that not sound like the beginning of the sci-fi story to you?
- WalterSear 4y ago
  Neither the calculator nor the AGI are necessarily distressed by being thrown in the bin.
  - coldtea 4y ago
    Sorry, the calculator sure, but for AGI this assumes what it should instead prove...
anyonecancode 4y ago
> Why does my calculator calculate? It doesn't? I have a calculator sitting on my desk right now. It's not doing anything. Now, I can pick it up, and _I_ can calculate, _using_ the calculator to do so, but the calculator isn't acting, I am. When we talk about sentience, I don't think we actually mean "intelligence," we mean "agency." And so the question isn't about whether Artificial General Intelligence is possible, but whether AGI can ever have agency. And that's where I think these questions about motivations and stimuli and so forth become relevant. Everything we know that has agency is embodied -- put reductively, it can die. And from that seems to spring motivation. Can something that cannot die have agency?
- chii 4y ago
  > but whether AGI can ever have agency. if you assume that at some point, the AGI would be given the ability to control something (such as sending IP packets through the internet), then yes, they will have agency! > Can something that cannot die have agency? If the AI could be deleted, or stopped from operating, then it can die. Therefore, if the AI has agency, it would take steps to prevent itself from dying, by whatever means available to it.
  - carlmr 4y ago
    >If the AI could be deleted, or stopped from operating, then it can die. Therefore, if the AI has agency, it would take steps to prevent itself from dying, by whatever means available to it. Why though? Agency and possibility of death don't mean it will try to prevent it. It could just as well commit suicide. Or it might do nothing. The question remains why, even with agency, it should do anything. We have agency, death looming, and a fear of dying. Even more when we are in the process of dying we're usually in pain, hungry, cold, suffocating or any number of not so nice options. Without any of these emotions and feelings, what reason do we have to avoid death?
    - chii 4y ago
      > Without any of these emotions and feelings, what reason do we have to avoid death? those are _some_ reasons to avoid death. But they are not the _only_ reasons to avoid death. For example, an AI that has been tasked with the production of paperclips would avoid death, so that it could continue to produce paperclips. Therefore, avoiding death was never an end goal, but an instrumental goal (see https://en.wikipedia.org/wiki/Instrumental_convergence https://en.wikipedia.org/wiki/Instrumental_convergence). So if the AI has agency (that is, it can act on its own without continuous input from a human) and it is sufficiently intelligent, you can say that it should be able to develop instrumental goals to achieve their end goal (of making paperclips), and one very well known instrumental goal is to avoid death.
deleted 4y ago
[deleted]

cleanchit 4y ago

Aren't ML models trained using a reward/punishment loop? (back propagation). If we have an AI that can learn from its experience, then pain will be failure to reach its goals and pleasure will be the opposite. Animals work in the exact same way (goal being survival and reproduction). We just need to make sure that survival and reproduction are never the goal (either directly or indirectly) of an AI and we should be safe.

jprete 4y ago
Survival (until no more actions are useful) is always needed to complete any other goal at all.

worrycue 4y ago

> What is guiding its decisions? That’s will be the most important part of AGI design. The ultimate goal of all AGI in existing should be “to be of service to (specific) human(s)”. It’s crucial that we do not create entities that will compete with us for resources. You do not what to create “another human”, you will want to create “a willing slave” with the agreeableness of a puppy and the “heart” of a saint.