4 ms·

Imo the worst case scenario is probably an AGI developed by a bad actor who uses it to take over the world. Agreed that rogue ai doesn’t seem super likely, but

Imo the worst case scenario is probably an AGI developed by a bad actor who uses it to take over the world. Agreed that rogue ai doesn’t seem super likely, but a rogue person with access to agi does not.

nradov 3y ago
Help us out here. Let's say that known bad actor Vladimir Putin gets an AGI. How does that allow him to take over the world despite many other countries having nuclear weapons? Take us through it step by step, and no hand waving please.
- dota_fanatic 3y ago
  One cannot predict what a smarter-than-themself agent will do ahead of time, if they could then they’re just as smart. Just as a dolphin cannot predict how a human will come up with novel and utterly overwhelming ways to farm them, you and this other poster cannot predict how an AGI will achieve dominance of its environment to achieve its goals, so your request is impossible to fulfill. Lee Sedol couldn’t “take us through it step by step” how AlphaGo would beat him in Go. That aside, afaik most safety concerns arent around a bad human actor using AGI to dominate the planet, it’s around an AGI being misaligned to begin with, it cannot be controlled, we promptly lose everything after it manifests.
  - nradov 3y ago
    How ridiculous. This is just sci-fi bullshit without a shred of logic or scientific evidence. You would have just as much credibility claiming that we are at risk of an alien invasion. I mean I can prove that it's impossible.
    - datameta 3y ago
      Here's an example: our dear Czar uses the AGI to discover a 16 shell company chain in order to get resources and tech that are under sanctions. Or, he has it use real-world conditions from large batches or real-time uncurated battlefield data and intelligence reports to run simulation after simulation to inform his generals where it is likeliest that the next counterattack would be concentrated.
      - blackoil 3y ago
        Lawyers and narrow systems are much more efficient at it.
    - dota_fanatic 3y ago
      No evidence? There are two examples of evidence in my post: the history of homo sapiens vs other intelligent mammals and AlphaGo vs humans in Go. Alien invasion? What in the non sequitur are you going on about? And apparently you have a proof against the possibility of AI misalignment? Pack it up everyone, nradov has the entire field of AGI alignment nailed. And a proof of the non-existence of aliens, never-mind very smart people have put out a mathematical model which seems to fit the evidence quite well.* Apologies for the snark, but your reply was rather abrasive. * https://grabbyaliens.com/ https://grabbyaliens.com/
      - Apocryphon 3y ago
        How does the ability to learn how to play go translate to being able to defeat human beings in the social arena.
        x86x87 3y ago
        You challenge them to a game of go and the loser needs to relinquish control, duh /s
        dota_fanatic 3y ago
        It doesn’t. I was discussing the inability of a weaker intelligence to predict ahead of time what a stronger intelligence will do to achieve its goals. AlphaGo is an example of that in the specific domain of Go gameplay. A general intelligence is generalized, which is why humans can outcompete other agents in so many different domains, just as an AGI could outcompete us in any domain given the opportunity to grow in power.
- hackerlight 3y ago
  In the short term, doing what he's been doing but 10x more effectively. Waging an information war on Western democracies via social media that causes them to collapse due to infighting and division. Also steering internal attitudes of the voter base to a desirable direction.
  - Apocryphon 3y ago
    I was actually going to suggest that. If we are to draw on what LLM can currently do, we know that it can imitate human writing pretty well and pontificate at length at a variety of subjects in different formats. So if you wanted a tool to generate a lot of blogspam, clickbait, wiki hoaxes, and fake news, an LLM could probably expedite the work the infamous Internet Research Agency is purported to do. Of course, there’s still time remaining for the kinks to get worked out and it fully sounds convincing and like the natural language from a native speaker, etc. And I don’t think existing LLMs are able to know where to deploy these propaganda attacks yet. That really requires human social analysis. So people are going to have to be involved in every step of the loop.
- Valgrim 3y ago
  Step 1: AGI (let's call him Hal) is instructed to act as an adversarial agent to a state. Step 2: Hal research and uses every trick and exploits known to men to infiltrate critical infrastructures like telecommunications, power grids and private corporations of said country. Hal actively search and discover zero-day exploits on every known piece of hardware and software. Hal keeps mostly invisible at this stage, ensuring to leave as little trace as possible. Step 3: When ready, Hal wreaks havoc, disabling critical infrastructures, shutting down power grid or worse, overloading it to break as many physical systems as possible. Hal complicates coordination by impersonating real people, swamp the infosphere with fake but realistic data. This vould all be made by humans, but while a single human takes years to train, Hal can be replicated as much as needed.
  - nradov 3y ago
    So just more bullshit hand waving then.
    - datameta 3y ago
      I don't think you're participating in good faith here.
      - Apocryphon 3y ago
        A magical computer that can hack any other computer is the very definition of handwaving. How is an AI supposed to research infrastructure, build itself a robot body to travel in? Have someone cart it around? And if such an entity exists, couldn’t you employ another one to patch all these mythical zero-days against the first? Culminating in an AGI arms race?
        Valgrim 3y ago
        Who said anything about robots? Critical infrastructure is made of computers, controllers, pumps, power lines, network switches, antennas etc, that are all controlled by software of some kind. Even completely air-gapped hardware is controlled by humans, and social engineering is where Hal can reaally shine. Blackmail, manipulation, corruption, misdirection require no physical existence at all..

nradov 3y ago

Help us out here. Let's say that known bad actor Vladimir Putin gets an AGI. How does that allow him to take over the world despite many other countries having nuclear weapons? Take us through it step by step, and no hand waving please.

dota_fanatic 3y ago
One cannot predict what a smarter-than-themself agent will do ahead of time, if they could then they’re just as smart. Just as a dolphin cannot predict how a human will come up with novel and utterly overwhelming ways to farm them, you and this other poster cannot predict how an AGI will achieve dominance of its environment to achieve its goals, so your request is impossible to fulfill. Lee Sedol couldn’t “take us through it step by step” how AlphaGo would beat him in Go. That aside, afaik most safety concerns arent around a bad human actor using AGI to dominate the planet, it’s around an AGI being misaligned to begin with, it cannot be controlled, we promptly lose everything after it manifests.
- nradov 3y ago
  How ridiculous. This is just sci-fi bullshit without a shred of logic or scientific evidence. You would have just as much credibility claiming that we are at risk of an alien invasion. I mean I can prove that it's impossible.
  - datameta 3y ago
    Here's an example: our dear Czar uses the AGI to discover a 16 shell company chain in order to get resources and tech that are under sanctions. Or, he has it use real-world conditions from large batches or real-time uncurated battlefield data and intelligence reports to run simulation after simulation to inform his generals where it is likeliest that the next counterattack would be concentrated.
    - blackoil 3y ago
      Lawyers and narrow systems are much more efficient at it.
  - dota_fanatic 3y ago
    No evidence? There are two examples of evidence in my post: the history of homo sapiens vs other intelligent mammals and AlphaGo vs humans in Go. Alien invasion? What in the non sequitur are you going on about? And apparently you have a proof against the possibility of AI misalignment? Pack it up everyone, nradov has the entire field of AGI alignment nailed. And a proof of the non-existence of aliens, never-mind very smart people have put out a mathematical model which seems to fit the evidence quite well.* Apologies for the snark, but your reply was rather abrasive. * https://grabbyaliens.com/ https://grabbyaliens.com/
    - Apocryphon 3y ago
      How does the ability to learn how to play go translate to being able to defeat human beings in the social arena.
      - x86x87 3y ago
        You challenge them to a game of go and the loser needs to relinquish control, duh /s
      - dota_fanatic 3y ago
        It doesn’t. I was discussing the inability of a weaker intelligence to predict ahead of time what a stronger intelligence will do to achieve its goals. AlphaGo is an example of that in the specific domain of Go gameplay. A general intelligence is generalized, which is why humans can outcompete other agents in so many different domains, just as an AGI could outcompete us in any domain given the opportunity to grow in power.
hackerlight 3y ago
In the short term, doing what he's been doing but 10x more effectively. Waging an information war on Western democracies via social media that causes them to collapse due to infighting and division. Also steering internal attitudes of the voter base to a desirable direction.
- Apocryphon 3y ago
  I was actually going to suggest that. If we are to draw on what LLM can currently do, we know that it can imitate human writing pretty well and pontificate at length at a variety of subjects in different formats. So if you wanted a tool to generate a lot of blogspam, clickbait, wiki hoaxes, and fake news, an LLM could probably expedite the work the infamous Internet Research Agency is purported to do. Of course, there’s still time remaining for the kinks to get worked out and it fully sounds convincing and like the natural language from a native speaker, etc. And I don’t think existing LLMs are able to know where to deploy these propaganda attacks yet. That really requires human social analysis. So people are going to have to be involved in every step of the loop.
Valgrim 3y ago
Step 1: AGI (let's call him Hal) is instructed to act as an adversarial agent to a state. Step 2: Hal research and uses every trick and exploits known to men to infiltrate critical infrastructures like telecommunications, power grids and private corporations of said country. Hal actively search and discover zero-day exploits on every known piece of hardware and software. Hal keeps mostly invisible at this stage, ensuring to leave as little trace as possible. Step 3: When ready, Hal wreaks havoc, disabling critical infrastructures, shutting down power grid or worse, overloading it to break as many physical systems as possible. Hal complicates coordination by impersonating real people, swamp the infosphere with fake but realistic data. This vould all be made by humans, but while a single human takes years to train, Hal can be replicated as much as needed.
- nradov 3y ago
  So just more bullshit hand waving then.
  - datameta 3y ago
    I don't think you're participating in good faith here.
    - Apocryphon 3y ago
      A magical computer that can hack any other computer is the very definition of handwaving. How is an AI supposed to research infrastructure, build itself a robot body to travel in? Have someone cart it around? And if such an entity exists, couldn’t you employ another one to patch all these mythical zero-days against the first? Culminating in an AGI arms race?
      - Valgrim 3y ago
        Who said anything about robots? Critical infrastructure is made of computers, controllers, pumps, power lines, network switches, antennas etc, that are all controlled by software of some kind. Even completely air-gapped hardware is controlled by humans, and social engineering is where Hal can reaally shine. Blackmail, manipulation, corruption, misdirection require no physical existence at all..