4 ms·

It's only fear mongering if it's deliberate for some ulterior end. If spreading concern is only motivated by protecting the human race from destroying itself--t

It's only fear mongering if it's deliberate for some ulterior end. If spreading concern is only motivated by protecting the human race from destroying itself--that would be a reasonable thing to talk about. Who's benefiting from urging caution about potential bad outcomes for AGI?

hueving 10y ago
>Who's benefiting from urging caution about potential bad outcomes for AGI? Every single person working in a position on the cusp of automation. The same thing happens with any new technology that threatens massive industries. (automobiles, airplanes, electricity, heating oil, etc)
- treehau5 10y ago
  Except automation unlike all those others you reference in that they all created new industries. AGI to cut the jobs and save labor costs for these large corporations.
- rojobuffalo 10y ago
  AGI and heating oil? There's a troubling gap in reasoning there. Narrow AI and automation replacing jobs has its own set of problems. But a machine intelligence explosion is a completely different thing, and a legitimate existential risk. It's really important to understand the difference and the specific arguments.
  - hartator 10y ago
    Still, you can't find anytime in history where we ended up saying "thanks we were careful" related to new technoloies. It's usually the reverse, governments delaying change for not smart reasons.
    - rojobuffalo 10y ago
      Sure you can. When people working on the Manhattan project calculated how far away they could stand from an atomic bomb detonation--that was an appreciable precaution. There are AGI risks that are well-reasoned and have detailed descriptions. If you aren't engaging with those arguments, proving them invalid or providing a well-reasoned protection, you aren't representing a coherent point of view.
      - hueving 10y ago
        >If you aren't engaging with those arguments, proving them invalid or providing a well-reasoned protection, you aren't representing a coherent point of view. Since when does refusing to engage with fear mongering make you incoherent?
        rojobuffalo 10y ago
        I already described the difference between fear mongering and pushing for caution, so don't conflate those terms. It's incoherent to not engage the arguments--take the control problem into consideration. Are you familiar with the reasoning for the expectancy that an AGI will be built that is smarter than humans at any reasoning task? Are you willing to go along with that? If so, then what follows is that the AGI improves the design of its own software and hardware better than any human engineer. At this point, there is a recursive improvement process termed an "intelligence explosion". From that point forward, we have no idea how to ensure that the goals of the AGI remain in alignment with human values. And the AGI can out-maneuver any attempt to turn it off. It's not guaranteed that it will want to avoid being turned off; but you could easily see how avoiding being turned off would improve the probability that it achieves its optimization parameters. Where do you check out? Edit: Nick Bostrom is the thought leader on this subject and he spells it out in this talk: https://goo.gl/DzNnk2 https://goo.gl/DzNnk2. His book, Superintelligence, is really good. Edit 2: And here is the audio for a Sam Harris talk also on this subject: https://goo.gl/6wdG43 https://goo.gl/6wdG43
    - yakult 10y ago
      Anthropic principle. You are not observing the counterfactual universes where people weren't careful enough around certain dangerous technologies, because in those universes all the observers are dead.

hueving 10y ago

>Who's benefiting from urging caution about potential bad outcomes for AGI? Every single person working in a position on the cusp of automation. The same thing happens with any new technology that threatens massive industries. (automobiles, airplanes, electricity, heating oil, etc)

treehau5 10y ago
Except automation unlike all those others you reference in that they all created new industries. AGI to cut the jobs and save labor costs for these large corporations.
rojobuffalo 10y ago
AGI and heating oil? There's a troubling gap in reasoning there. Narrow AI and automation replacing jobs has its own set of problems. But a machine intelligence explosion is a completely different thing, and a legitimate existential risk. It's really important to understand the difference and the specific arguments.
- hartator 10y ago
  Still, you can't find anytime in history where we ended up saying "thanks we were careful" related to new technoloies. It's usually the reverse, governments delaying change for not smart reasons.
  - rojobuffalo 10y ago
    Sure you can. When people working on the Manhattan project calculated how far away they could stand from an atomic bomb detonation--that was an appreciable precaution. There are AGI risks that are well-reasoned and have detailed descriptions. If you aren't engaging with those arguments, proving them invalid or providing a well-reasoned protection, you aren't representing a coherent point of view.
    - hueving 10y ago
      >If you aren't engaging with those arguments, proving them invalid or providing a well-reasoned protection, you aren't representing a coherent point of view. Since when does refusing to engage with fear mongering make you incoherent?
      - rojobuffalo 10y ago
        I already described the difference between fear mongering and pushing for caution, so don't conflate those terms. It's incoherent to not engage the arguments--take the control problem into consideration. Are you familiar with the reasoning for the expectancy that an AGI will be built that is smarter than humans at any reasoning task? Are you willing to go along with that? If so, then what follows is that the AGI improves the design of its own software and hardware better than any human engineer. At this point, there is a recursive improvement process termed an "intelligence explosion". From that point forward, we have no idea how to ensure that the goals of the AGI remain in alignment with human values. And the AGI can out-maneuver any attempt to turn it off. It's not guaranteed that it will want to avoid being turned off; but you could easily see how avoiding being turned off would improve the probability that it achieves its optimization parameters. Where do you check out? Edit: Nick Bostrom is the thought leader on this subject and he spells it out in this talk: https://goo.gl/DzNnk2 https://goo.gl/DzNnk2. His book, Superintelligence, is really good. Edit 2: And here is the audio for a Sam Harris talk also on this subject: https://goo.gl/6wdG43 https://goo.gl/6wdG43
  - yakult 10y ago
    Anthropic principle. You are not observing the counterfactual universes where people weren't careful enough around certain dangerous technologies, because in those universes all the observers are dead.