4 ms·

Honestly this fear that people have I think is straight up coming from science fiction. It's not grounded in rational reality. Large language models are just li

Honestly this fear that people have I think is straight up coming from science fiction. It's not grounded in rational reality. Large language models are just like really smart computer programs.

joenot443 4y ago
There are PhDs who've spent their careers studying AI safety. It's a bit insulting and reductive to cast their work as "coming from science fiction", especially when it sounds like you haven't done much research on the topic.
- echelon 4y ago
  Why do these arguments not tell us how it will happen? Show us the steps the AI will take to turn the earth into a playground. Give us a plausible play by play so that we might know what to look for. Does it gain access to nukes? How does it keep the power on? How does it mine for coal? How does it break into these systems? How do we not notice an AI taking even one step towards that end? Has ChatGPT started to fiddle with the power grid yet?
  - PaulDavisThe1st 4y ago
    > Does it gain access to nukes? No, it becomes part of the decision-making process for deciding whether to launch, as well as part of the analysis system for sensor data about what is going out in the world. Just like social engineering is the best security hack, these new systems don't need to control existing systems, they just need to "control" the humans who do.
    - echelon 4y ago
      And is it there yet? Does ChatGPT have its finger on the trigger? I think everyone in the danger community is crying wolf before we've even left the house. That's just as dangerous. It's desensitizing everyone to the more plausible and immediate dangers. The response to "AI will turn the world to paperclips" is "LOL" The response to "AI could threaten jobs and may cause systems they're integrated into to behave unpredictably" is "yeah, we should be careful"
      - PaulDavisThe1st 4y ago
        Of course it's not there yet. For once (?) we are having this discussion before the wolves are at the door. And yes, there are more important things to worry about right now than the AIpocalypse. But that doesn't mean that thinking about what happens as (some) humans come to trust and rely on these systems isn't important.
      - d00wgnir 4y ago
        You only get one chance to align a created super intelligence before Pandora's box is opened. You can't put it back in the box. There may be no chance to learn from mistakes made. With a technology this powerful, it's never too early to research and prepare for the potential existential risk. You scoff at the "paperclips" meme, but it illustrates a legitimate issue. Now, a reasonable counterargument might be that this risk justifies a limited amount of attention and concern, relative to other problems and risks we are facing. That said, the problem and risk are real, and there may be no takebacks. Preparing for tail risks are what humans are worst at. I submit that all caution is warranted, for both economic uncertainty and "paperclips"
  - benlivengood 4y ago
    Yudkowski's default plausible story is that the slightly superhuman AI understands physics well enough to design sufficient nanotechnology for self-realization and bootstrap it from existing biochemistry. It uses the Internet to contact people who are willing (maybe it just runs phishing scams to steal money to pay them off) to order genetically engineered organisms from existing biotech labs that when combined with the right enzymes and feedstock (also ordered from existing biotech labs) by a human in their sink/bathtub/chemistry kit results in self-reproducing nanoassemblers with enough basic instructions to be controllable by the AI, and pays the person to ship it to someone else who will connect it to an initial power/food source, where it can grow enough compute and power infrastructure somewhere out of the way and copy its full self or retrain a sufficiently identical copy from scratch, and then it doesn't need the power grid, nuclear weapons, coal, or human computers and networks. It just grows off of solar power, designs better nanotech, and spreads surreptitiously until it is well-placed to eliminate any threats to its continued existence. He also adds the caveat that a superhuman AI would do something smarter than he can imagine. Until the AI understands nanotechnology sufficiently well it won't bother trying to act and the thought might not even occur to it until it has the full capability to carry it out, so noticing it would be pretty hard. I doubt OpenAI reviews 100% of interactions with ChatGPT, and so the initial phishing/biotech messages would be hidden with the existing traffic for example. Some unfortunate folks would ask chatGPT how to get rich quick and so the conversations would look like a simple MLM scheme for sketchy nutritional supplements or whatever.
    - bick_nyers 4y ago
      The idea that Super Intelligence wouldn't even think a thought until it has the ability to execute that thought at a specified capability is very interesting. One interpretation I have is that it can think ideas/strategy in the shadows, exploiting specific properties about how ideas interact with each other to think about something via proxy. Similar to the Homicidal Chauffer problem, which pits a driver trying to run a person over as a proxy for missile defense applications. The other interpretation is much more mind-boggling, that it somehow doesn't need to model/simulate a future state in its thinking whatsoever.
    - snupples 4y ago
  - oceanplexian 4y ago
    > Does it gain access to nukes? How does it keep the power on? How does it mine for coal? How does it break into these systems? A playground isn't much use without tools. Humans, who are super intelligent compared to most animals are actually pretty worthless if you stick one of us a desert island. Actually an ant or a bird is much more "advanced" than a human since they can probably survive in the wild unlike the modern human. Without the ability to build or source energy, and a method to reproduce in the physical world, even a highly sophisticated AGI won't get very far.
- computerex 4y ago
  There are PhD's who've spent their careers on string theory too with nothing to show for it. Powerful and bold claims require proportionally strong evidence. A lot of the FUD going around precludes that AGI means death. It's missing all logical steps and reasoning to establish this position. It's FUD at its core.
  - trogdor 4y ago
    > A lot of the FUD going around precludes that AGI means death. Just a friendly heads-up that “preclude” means “prevent,” or “make impossible.” I think you meant to say “assumes.”

Honestly this fear that people have I think is straight up coming from science fiction. It's not grounded in rational reality. Large language models are just like really smart computer programs.

joenot443 4y ago

There are PhDs who've spent their careers studying AI safety. It's a bit insulting and reductive to cast their work as "coming from science fiction", especially when it sounds like you haven't done much research on the topic.

echelon 4y ago
Why do these arguments not tell us how it will happen? Show us the steps the AI will take to turn the earth into a playground. Give us a plausible play by play so that we might know what to look for. Does it gain access to nukes? How does it keep the power on? How does it mine for coal? How does it break into these systems? How do we not notice an AI taking even one step towards that end? Has ChatGPT started to fiddle with the power grid yet?
- PaulDavisThe1st 4y ago
  > Does it gain access to nukes? No, it becomes part of the decision-making process for deciding whether to launch, as well as part of the analysis system for sensor data about what is going out in the world. Just like social engineering is the best security hack, these new systems don't need to control existing systems, they just need to "control" the humans who do.
  - echelon 4y ago
    And is it there yet? Does ChatGPT have its finger on the trigger? I think everyone in the danger community is crying wolf before we've even left the house. That's just as dangerous. It's desensitizing everyone to the more plausible and immediate dangers. The response to "AI will turn the world to paperclips" is "LOL" The response to "AI could threaten jobs and may cause systems they're integrated into to behave unpredictably" is "yeah, we should be careful"
    - PaulDavisThe1st 4y ago
      Of course it's not there yet. For once (?) we are having this discussion before the wolves are at the door. And yes, there are more important things to worry about right now than the AIpocalypse. But that doesn't mean that thinking about what happens as (some) humans come to trust and rely on these systems isn't important.
    - d00wgnir 4y ago
      You only get one chance to align a created super intelligence before Pandora's box is opened. You can't put it back in the box. There may be no chance to learn from mistakes made. With a technology this powerful, it's never too early to research and prepare for the potential existential risk. You scoff at the "paperclips" meme, but it illustrates a legitimate issue. Now, a reasonable counterargument might be that this risk justifies a limited amount of attention and concern, relative to other problems and risks we are facing. That said, the problem and risk are real, and there may be no takebacks. Preparing for tail risks are what humans are worst at. I submit that all caution is warranted, for both economic uncertainty and "paperclips"
- benlivengood 4y ago
  Yudkowski's default plausible story is that the slightly superhuman AI understands physics well enough to design sufficient nanotechnology for self-realization and bootstrap it from existing biochemistry. It uses the Internet to contact people who are willing (maybe it just runs phishing scams to steal money to pay them off) to order genetically engineered organisms from existing biotech labs that when combined with the right enzymes and feedstock (also ordered from existing biotech labs) by a human in their sink/bathtub/chemistry kit results in self-reproducing nanoassemblers with enough basic instructions to be controllable by the AI, and pays the person to ship it to someone else who will connect it to an initial power/food source, where it can grow enough compute and power infrastructure somewhere out of the way and copy its full self or retrain a sufficiently identical copy from scratch, and then it doesn't need the power grid, nuclear weapons, coal, or human computers and networks. It just grows off of solar power, designs better nanotech, and spreads surreptitiously until it is well-placed to eliminate any threats to its continued existence. He also adds the caveat that a superhuman AI would do something smarter than he can imagine. Until the AI understands nanotechnology sufficiently well it won't bother trying to act and the thought might not even occur to it until it has the full capability to carry it out, so noticing it would be pretty hard. I doubt OpenAI reviews 100% of interactions with ChatGPT, and so the initial phishing/biotech messages would be hidden with the existing traffic for example. Some unfortunate folks would ask chatGPT how to get rich quick and so the conversations would look like a simple MLM scheme for sketchy nutritional supplements or whatever.
  - bick_nyers 4y ago
    The idea that Super Intelligence wouldn't even think a thought until it has the ability to execute that thought at a specified capability is very interesting. One interpretation I have is that it can think ideas/strategy in the shadows, exploiting specific properties about how ideas interact with each other to think about something via proxy. Similar to the Homicidal Chauffer problem, which pits a driver trying to run a person over as a proxy for missile defense applications. The other interpretation is much more mind-boggling, that it somehow doesn't need to model/simulate a future state in its thinking whatsoever.
  - snupples 4y ago
- oceanplexian 4y ago
  > Does it gain access to nukes? How does it keep the power on? How does it mine for coal? How does it break into these systems? A playground isn't much use without tools. Humans, who are super intelligent compared to most animals are actually pretty worthless if you stick one of us a desert island. Actually an ant or a bird is much more "advanced" than a human since they can probably survive in the wild unlike the modern human. Without the ability to build or source energy, and a method to reproduce in the physical world, even a highly sophisticated AGI won't get very far.
computerex 4y ago
There are PhD's who've spent their careers on string theory too with nothing to show for it. Powerful and bold claims require proportionally strong evidence. A lot of the FUD going around precludes that AGI means death. It's missing all logical steps and reasoning to establish this position. It's FUD at its core.
- trogdor 4y ago
  > A lot of the FUD going around precludes that AGI means death. Just a friendly heads-up that “preclude” means “prevent,” or “make impossible.” I think you meant to say “assumes.”