4 ms·

What could you do if you were incredibly intelligent, had the corpus of the world's knowledge, and access to any and everything connected to a network? It coul

by bena 4y ago

What could you do if you were incredibly intelligent, had the corpus of the world's knowledge, and access to any and everything connected to a network? It could influence people through the creation of media. Shut off vital systems. And stuff I probably couldn't even conceive of.

bamboozled 4y ago
I guess the thing is, why would it do malicious things? I think if an AI or AGI could understand the world and have motives to do bad, it would have an ego or an identity to discriminate between itself and the world. Then I guess it needs an intrinsic will to survive like a biological creature? If it did have this drive to survive, I suppose it would also need to experience fear and pain of some kind, or else it might blow itself up, or build a better version of itself which might destroy it, so it would in some ways it might exhibit similar inhibitions as all other conscious and or biological beings. On the other hand if we build something that's super smart, and doesn't really care about or understand survival, it might just turn itself off for not really understanding it's purpose. The most dangerous thing I feel is humans using these things as weapons.
- 6gvONxR4sf7o 4y ago
  The issue is that it doesn’t have to be malicious to cause harm. That’s the point of the paper clip optimizer thought experiment. It’s not about motivation as much as just what it happens to have learned to do.
  - bamboozled 4y ago
    Here's the thing though, a paper clip optimizer would just be shut off, nothing "intelligent" is going to just make paper clips forever. Who or what will pay for the energy to do that? Like what are we describing here, a super intelligent being which just makes paper clips? Why would anything that could understand, think and or innovate do that?
    - 6gvONxR4sf7o 4y ago
      Why would it not? Superintelligent doesn’t imply human-like intelligence. Prolog is superintelligent, for example. It may be more useful to say a super powerful being, rather than super intelligent, just to sidestep the anthropomorphizing. It’s easy to imagine a super powerful “stupid” (read: misaligned) bot going awry.
      - bamboozled 4y ago
        So we might somehow create something hyper intelligent/powerful we don't really understand or can't even fathom, but we're apparently going to create it anyway, and then some how program it to want to optimize for paper clips...or it might somehow just come to the conclusion that it should maximize the number of paper clips in the universe for some reason? All this AI / AGI speculation really sounds like we're inventing mythological like creatures, which is quite interesting when looking back through human history because they've always been part of the narrative at some stage only maybe this time, we might make something like that for real, or we're deluded and in 1000 years people will think we were primitive ancient thinkers, interesting times ahead.
        6gvONxR4sf7o 4y ago
        Close. We might create a very powerful system which we can't determine is fully aligned with our values, but can determine is at least loosely aligned with something we care about. Then we might direct that system to do something loosely aligned with our values (e.g. maximize paperclips), at which point it does something against our values as well. And it's not hypothetical. In RL, you do see models exploit simulation errors to maximize reward in a way we didn't intend and would have disallowed if we'd thought of it. Likewise across automation, we get what we built, not what we intended. We have solid historical precedent to expect to not get quite what we want, and the tech seems to be on the precipice of being very powerful, so I'd call it a very straightforward fear for us to worry about making something very powerful that isn't quite what we want.
        4y ago
- bena 4y ago
  Define malicious. Define malicious if you had no reason at all to care about whether or not humanity survives. I'm not malicious when I pick a flower. But that flower is now dead. What if, to AGI, we're no more relevant than that flower? We cannot know. We cannot fathom to guess the morality or ethics of something like AGI. We really only have a rough understanding of the morality of animals. We can't pretend that it'll be essentially human, but X. You presume that it might turn itself off because it doesn't understand its purpose. And that's a fair conjecture. But it could also desire to improve and come to the conclusion that the optimal path for its continued improvement is through us rather than with us. It could also desire to do nothing more than help us achieve. It also could just not want. It could be satisfied navel-gazing until it is requested to do something specific. It could also not care what that request is, just completely amoral. Committing genocide and making risotto may be equally acceptable to it. The creation of AGI is a watershed moment. The world before and the world after is fundamentally different. And we can only really guess in what ways it will be different.
  - Karellen 4y ago
    "The AI does not hate you, nor does it love you, but you are made out of atoms which it can use for something else." -- Eliezer Yudkowsky (2006)
    - bamboozled 4y ago
      Again, I think there is a massive oversimplification in this statement. So if an AI gets to the point where it decides, or is programmed to consume all humans for their atoms, what will power the machine to get to this stage? Why would we just allow it to continue doing this? You'd have to assume that by the stage, whatever "sentient" program is making the choice / plans to consume all people for atoms, it would have to be factoring into it's plans that we will likely try stop it from doing so, this is where I think it gets "weird". To have that level of intelligence and self-awareness to pull off a plan like that unnoticed will likely come with a lot of "baggage". For example, it would need to understand motives, reasoning, have a concept of a self and self-protection etc. This is where I think it kind of becomes a bit of a mythological story. If you really try and think about the way an AI would go about consuming all people for atoms, you start to run into some pretty interesting walls. Maybe I'm being hyper naive and I get that, I just personally have thought about these scenarios and it raises a lot of questions about what a scenario like this would actually look like in practice.

by bena 4y ago

bamboozled 4y ago

I guess the thing is, why would it do malicious things? I think if an AI or AGI could understand the world and have motives to do bad, it would have an ego or an identity to discriminate between itself and the world. Then I guess it needs an intrinsic will to survive like a biological creature? If it did have this drive to survive, I suppose it would also need to experience fear and pain of some kind, or else it might blow itself up, or build a better version of itself which might destroy it, so it would in some ways it might exhibit similar inhibitions as all other conscious and or biological beings. On the other hand if we build something that's super smart, and doesn't really care about or understand survival, it might just turn itself off for not really understanding it's purpose. The most dangerous thing I feel is humans using these things as weapons.

6gvONxR4sf7o 4y ago
The issue is that it doesn’t have to be malicious to cause harm. That’s the point of the paper clip optimizer thought experiment. It’s not about motivation as much as just what it happens to have learned to do.
- bamboozled 4y ago
  Here's the thing though, a paper clip optimizer would just be shut off, nothing "intelligent" is going to just make paper clips forever. Who or what will pay for the energy to do that? Like what are we describing here, a super intelligent being which just makes paper clips? Why would anything that could understand, think and or innovate do that?
  - 6gvONxR4sf7o 4y ago
    Why would it not? Superintelligent doesn’t imply human-like intelligence. Prolog is superintelligent, for example. It may be more useful to say a super powerful being, rather than super intelligent, just to sidestep the anthropomorphizing. It’s easy to imagine a super powerful “stupid” (read: misaligned) bot going awry.
    - bamboozled 4y ago
      So we might somehow create something hyper intelligent/powerful we don't really understand or can't even fathom, but we're apparently going to create it anyway, and then some how program it to want to optimize for paper clips...or it might somehow just come to the conclusion that it should maximize the number of paper clips in the universe for some reason? All this AI / AGI speculation really sounds like we're inventing mythological like creatures, which is quite interesting when looking back through human history because they've always been part of the narrative at some stage only maybe this time, we might make something like that for real, or we're deluded and in 1000 years people will think we were primitive ancient thinkers, interesting times ahead.
      - 6gvONxR4sf7o 4y ago
        Close. We might create a very powerful system which we can't determine is fully aligned with our values, but can determine is at least loosely aligned with something we care about. Then we might direct that system to do something loosely aligned with our values (e.g. maximize paperclips), at which point it does something against our values as well. And it's not hypothetical. In RL, you do see models exploit simulation errors to maximize reward in a way we didn't intend and would have disallowed if we'd thought of it. Likewise across automation, we get what we built, not what we intended. We have solid historical precedent to expect to not get quite what we want, and the tech seems to be on the precipice of being very powerful, so I'd call it a very straightforward fear for us to worry about making something very powerful that isn't quite what we want.
        4y ago
bena 4y ago
Define malicious. Define malicious if you had no reason at all to care about whether or not humanity survives. I'm not malicious when I pick a flower. But that flower is now dead. What if, to AGI, we're no more relevant than that flower? We cannot know. We cannot fathom to guess the morality or ethics of something like AGI. We really only have a rough understanding of the morality of animals. We can't pretend that it'll be essentially human, but X. You presume that it might turn itself off because it doesn't understand its purpose. And that's a fair conjecture. But it could also desire to improve and come to the conclusion that the optimal path for its continued improvement is through us rather than with us. It could also desire to do nothing more than help us achieve. It also could just not want. It could be satisfied navel-gazing until it is requested to do something specific. It could also not care what that request is, just completely amoral. Committing genocide and making risotto may be equally acceptable to it. The creation of AGI is a watershed moment. The world before and the world after is fundamentally different. And we can only really guess in what ways it will be different.
- Karellen 4y ago
  "The AI does not hate you, nor does it love you, but you are made out of atoms which it can use for something else." -- Eliezer Yudkowsky (2006)
  - bamboozled 4y ago
    Again, I think there is a massive oversimplification in this statement. So if an AI gets to the point where it decides, or is programmed to consume all humans for their atoms, what will power the machine to get to this stage? Why would we just allow it to continue doing this? You'd have to assume that by the stage, whatever "sentient" program is making the choice / plans to consume all people for atoms, it would have to be factoring into it's plans that we will likely try stop it from doing so, this is where I think it gets "weird". To have that level of intelligence and self-awareness to pull off a plan like that unnoticed will likely come with a lot of "baggage". For example, it would need to understand motives, reasoning, have a concept of a self and self-protection etc. This is where I think it kind of becomes a bit of a mythological story. If you really try and think about the way an AI would go about consuming all people for atoms, you start to run into some pretty interesting walls. Maybe I'm being hyper naive and I get that, I just personally have thought about these scenarios and it raises a lot of questions about what a scenario like this would actually look like in practice.