4 ms·

Bostrom, at least, makes good philosophical arguments, and has been doing so since well before these topics around AI were even being discussed.

by prvc 7y ago

Udik 7y ago
As far as I understood Bostrom's argument, it goes more or less like this: "suppose we made an omnipotent being, how do we control it?". On and on chapter after chapter, with no serious discussion of whether it makes any sense to even think of an omnipotent being and of its control. If we build an omnipotent paperclip-making machine, the first thing it'll do will be to short circuit its paperclip-driven pleasure centres to a permanent ecstatic state. Why bother with the damn paperclips?
- luc4sdreyer 7y ago
  Omnipotent might be a bit too strong a word, I'll assume you meant omnipotent with respect to humans. > the first thing it'll do will be to short circuit its paperclip-driven pleasure centres to a permanent ecstatic state Assuming the AGI is well-designed, the goal would have to be understandable to the AGI in a general sense. If the goal is to make paperclips, it could be tracked by increasing the number at the &num_paperclips address, but that wouldn't be the _goal_ as the AGI understands it. The goal would be whatever the AGI considers the phrase "make paperclips" to mean. What if it instead determines that increasing the counter (the pleasure center) is actually the goal? Now it has a new goal: count to infinity. I guess you can see where that'll lead. What if you bound the pleasure center to some number? It'll then spend infinite time and power checking and recounting the number. You can add more bounds but it doesn't make a difference. What if it decides that the goal is silly or wants to change the goal to something else? It can't, because it doesn't _want_ anything other than the goal.
  - Udik 7y ago
    > Omnipotent might be a bit too strong a word, I'll assume you meant omnipotent with respect to humans. A being that can transform the whole universe into paperclips and cannot be stopped sounds pretty omnipotent to me. > The goal would be whatever the AGI considers the phrase "make paperclips" to mean. One thing is the understanding of the goal. Another thing is the drive that makes you want to reach it, and the reward you get from doing so. Humans are general intelligences, and they short circuit their pleasure centers all the time. And they're not omnipotent, nor capable of the self-modification that would provide infinite new ways of doing it.

Bostrom, at least, makes good philosophical arguments, and has been doing so since well before these topics around AI were even being discussed.

by prvc 7y ago

Bostrom, at least, makes good philosophical arguments, and has been doing so since well before these topics around AI were even being discussed.

Udik 7y ago

As far as I understood Bostrom's argument, it goes more or less like this: "suppose we made an omnipotent being, how do we control it?". On and on chapter after chapter, with no serious discussion of whether it makes any sense to even think of an omnipotent being and of its control. If we build an omnipotent paperclip-making machine, the first thing it'll do will be to short circuit its paperclip-driven pleasure centres to a permanent ecstatic state. Why bother with the damn paperclips?

luc4sdreyer 7y ago
Omnipotent might be a bit too strong a word, I'll assume you meant omnipotent with respect to humans. > the first thing it'll do will be to short circuit its paperclip-driven pleasure centres to a permanent ecstatic state Assuming the AGI is well-designed, the goal would have to be understandable to the AGI in a general sense. If the goal is to make paperclips, it could be tracked by increasing the number at the &num_paperclips address, but that wouldn't be the _goal_ as the AGI understands it. The goal would be whatever the AGI considers the phrase "make paperclips" to mean. What if it instead determines that increasing the counter (the pleasure center) is actually the goal? Now it has a new goal: count to infinity. I guess you can see where that'll lead. What if you bound the pleasure center to some number? It'll then spend infinite time and power checking and recounting the number. You can add more bounds but it doesn't make a difference. What if it decides that the goal is silly or wants to change the goal to something else? It can't, because it doesn't _want_ anything other than the goal.
- Udik 7y ago
  > Omnipotent might be a bit too strong a word, I'll assume you meant omnipotent with respect to humans. A being that can transform the whole universe into paperclips and cannot be stopped sounds pretty omnipotent to me. > The goal would be whatever the AGI considers the phrase "make paperclips" to mean. One thing is the understanding of the goal. Another thing is the drive that makes you want to reach it, and the reward you get from doing so. Humans are general intelligences, and they short circuit their pleasure centers all the time. And they're not omnipotent, nor capable of the self-modification that would provide infinite new ways of doing it.