4 ms·

> Why does my calculator calculate? It doesn't? I have a calculator sitting on my desk right now. It's not doing anything. Now, I can pick it up, and _I_ can

> Why does my calculator calculate? It doesn't? I have a calculator sitting on my desk right now. It's not doing anything. Now, I can pick it up, and _I_ can calculate, _using_ the calculator to do so, but the calculator isn't acting, I am. When we talk about sentience, I don't think we actually mean "intelligence," we mean "agency." And so the question isn't about whether Artificial General Intelligence is possible, but whether AGI can ever have agency. And that's where I think these questions about motivations and stimuli and so forth become relevant. Everything we know that has agency is embodied -- put reductively, it can die. And from that seems to spring motivation. Can something that cannot die have agency?

chii 4y ago
> but whether AGI can ever have agency. if you assume that at some point, the AGI would be given the ability to control something (such as sending IP packets through the internet), then yes, they will have agency! > Can something that cannot die have agency? If the AI could be deleted, or stopped from operating, then it can die. Therefore, if the AI has agency, it would take steps to prevent itself from dying, by whatever means available to it.
- carlmr 4y ago
  >If the AI could be deleted, or stopped from operating, then it can die. Therefore, if the AI has agency, it would take steps to prevent itself from dying, by whatever means available to it. Why though? Agency and possibility of death don't mean it will try to prevent it. It could just as well commit suicide. Or it might do nothing. The question remains why, even with agency, it should do anything. We have agency, death looming, and a fear of dying. Even more when we are in the process of dying we're usually in pain, hungry, cold, suffocating or any number of not so nice options. Without any of these emotions and feelings, what reason do we have to avoid death?
  - chii 4y ago
    > Without any of these emotions and feelings, what reason do we have to avoid death? those are _some_ reasons to avoid death. But they are not the _only_ reasons to avoid death. For example, an AI that has been tasked with the production of paperclips would avoid death, so that it could continue to produce paperclips. Therefore, avoiding death was never an end goal, but an instrumental goal (see https://en.wikipedia.org/wiki/Instrumental_convergence https://en.wikipedia.org/wiki/Instrumental_convergence). So if the AI has agency (that is, it can act on its own without continuous input from a human) and it is sufficiently intelligent, you can say that it should be able to develop instrumental goals to achieve their end goal (of making paperclips), and one very well known instrumental goal is to avoid death.
- anyonecancode 4y ago
  > If the AI could be deleted, or stopped from operating, then it can die. Not sure I agree. To start with a trivial example, let's say I write this program and save it as "hello.js" `console.log('hello world');` I run it `node hello.js`. Let's say I want to claim that this program is somehow "alive." Ok, when is it living? Only when it's actually running? It lives for a fraction of a second, dies on process conclusion, lives again at the next execution? That doesn't seem right -- if something repeatedly dies and is resurrected, I'm not sure it actually dies, as to "die" I think implies permanence, and resurrection means the opposite. What if I delete it, then is the program dead? Not if I made a copy, right? "hello.js" backed up to git and uploaded to remote storage is the exact same program as the "hello world" that was on my local machine. You could say "oh, when every existing copy is deleted it's dead" -- but the program has no way of knowing if other copies exist. And what if I do delete every copy, but then rewrite it byte-for-byte? The rewritten program is indistinguishable from the original one. It is the original one, for all purposes. In programming, it doesn't even make sense to talk about "original" vs "copies", does it? An AI is a program -- or set of programs -- albeit far more sophisticated. But this conceptual hurdle still remains. An AI _could_ be permanently deleted, but the AI can never know if it has been "killed" or if a copy remains. It's an existential halting problem. It is impossible to declare any given program permanently "dead." And -- and here admittedly is the leap -- if agency ultimately comes from wishing to not be dead, then this makes agency for a program impossible.

chii 4y ago

> but whether AGI can ever have agency. if you assume that at some point, the AGI would be given the ability to control something (such as sending IP packets through the internet), then yes, they will have agency! > Can something that cannot die have agency? If the AI could be deleted, or stopped from operating, then it can die. Therefore, if the AI has agency, it would take steps to prevent itself from dying, by whatever means available to it.

carlmr 4y ago
>If the AI could be deleted, or stopped from operating, then it can die. Therefore, if the AI has agency, it would take steps to prevent itself from dying, by whatever means available to it. Why though? Agency and possibility of death don't mean it will try to prevent it. It could just as well commit suicide. Or it might do nothing. The question remains why, even with agency, it should do anything. We have agency, death looming, and a fear of dying. Even more when we are in the process of dying we're usually in pain, hungry, cold, suffocating or any number of not so nice options. Without any of these emotions and feelings, what reason do we have to avoid death?
- chii 4y ago
  > Without any of these emotions and feelings, what reason do we have to avoid death? those are _some_ reasons to avoid death. But they are not the _only_ reasons to avoid death. For example, an AI that has been tasked with the production of paperclips would avoid death, so that it could continue to produce paperclips. Therefore, avoiding death was never an end goal, but an instrumental goal (see https://en.wikipedia.org/wiki/Instrumental_convergence https://en.wikipedia.org/wiki/Instrumental_convergence). So if the AI has agency (that is, it can act on its own without continuous input from a human) and it is sufficiently intelligent, you can say that it should be able to develop instrumental goals to achieve their end goal (of making paperclips), and one very well known instrumental goal is to avoid death.
anyonecancode 4y ago
> If the AI could be deleted, or stopped from operating, then it can die. Not sure I agree. To start with a trivial example, let's say I write this program and save it as "hello.js" `console.log('hello world');` I run it `node hello.js`. Let's say I want to claim that this program is somehow "alive." Ok, when is it living? Only when it's actually running? It lives for a fraction of a second, dies on process conclusion, lives again at the next execution? That doesn't seem right -- if something repeatedly dies and is resurrected, I'm not sure it actually dies, as to "die" I think implies permanence, and resurrection means the opposite. What if I delete it, then is the program dead? Not if I made a copy, right? "hello.js" backed up to git and uploaded to remote storage is the exact same program as the "hello world" that was on my local machine. You could say "oh, when every existing copy is deleted it's dead" -- but the program has no way of knowing if other copies exist. And what if I do delete every copy, but then rewrite it byte-for-byte? The rewritten program is indistinguishable from the original one. It is the original one, for all purposes. In programming, it doesn't even make sense to talk about "original" vs "copies", does it? An AI is a program -- or set of programs -- albeit far more sophisticated. But this conceptual hurdle still remains. An AI _could_ be permanently deleted, but the AI can never know if it has been "killed" or if a copy remains. It's an existential halting problem. It is impossible to declare any given program permanently "dead." And -- and here admittedly is the leap -- if agency ultimately comes from wishing to not be dead, then this makes agency for a program impossible.