7 ms·

Sure you can. When people working on the Manhattan project calculated how far away they could stand from an atomic bomb detonation--that was an appreciable prec

Sure you can. When people working on the Manhattan project calculated how far away they could stand from an atomic bomb detonation--that was an appreciable precaution. There are AGI risks that are well-reasoned and have detailed descriptions. If you aren't engaging with those arguments, proving them invalid or providing a well-reasoned protection, you aren't representing a coherent point of view.

hueving 10y ago
>If you aren't engaging with those arguments, proving them invalid or providing a well-reasoned protection, you aren't representing a coherent point of view. Since when does refusing to engage with fear mongering make you incoherent?
- rojobuffalo 10y ago
  I already described the difference between fear mongering and pushing for caution, so don't conflate those terms. It's incoherent to not engage the arguments--take the control problem into consideration. Are you familiar with the reasoning for the expectancy that an AGI will be built that is smarter than humans at any reasoning task? Are you willing to go along with that? If so, then what follows is that the AGI improves the design of its own software and hardware better than any human engineer. At this point, there is a recursive improvement process termed an "intelligence explosion". From that point forward, we have no idea how to ensure that the goals of the AGI remain in alignment with human values. And the AGI can out-maneuver any attempt to turn it off. It's not guaranteed that it will want to avoid being turned off; but you could easily see how avoiding being turned off would improve the probability that it achieves its optimization parameters. Where do you check out? Edit: Nick Bostrom is the thought leader on this subject and he spells it out in this talk: https://goo.gl/DzNnk2 https://goo.gl/DzNnk2. His book, Superintelligence, is really good. Edit 2: And here is the audio for a Sam Harris talk also on this subject: https://goo.gl/6wdG43 https://goo.gl/6wdG43

hueving 10y ago

>If you aren't engaging with those arguments, proving them invalid or providing a well-reasoned protection, you aren't representing a coherent point of view. Since when does refusing to engage with fear mongering make you incoherent?

rojobuffalo 10y ago
I already described the difference between fear mongering and pushing for caution, so don't conflate those terms. It's incoherent to not engage the arguments--take the control problem into consideration. Are you familiar with the reasoning for the expectancy that an AGI will be built that is smarter than humans at any reasoning task? Are you willing to go along with that? If so, then what follows is that the AGI improves the design of its own software and hardware better than any human engineer. At this point, there is a recursive improvement process termed an "intelligence explosion". From that point forward, we have no idea how to ensure that the goals of the AGI remain in alignment with human values. And the AGI can out-maneuver any attempt to turn it off. It's not guaranteed that it will want to avoid being turned off; but you could easily see how avoiding being turned off would improve the probability that it achieves its optimization parameters. Where do you check out? Edit: Nick Bostrom is the thought leader on this subject and he spells it out in this talk: https://goo.gl/DzNnk2 https://goo.gl/DzNnk2. His book, Superintelligence, is really good. Edit 2: And here is the audio for a Sam Harris talk also on this subject: https://goo.gl/6wdG43 https://goo.gl/6wdG43