3 ms·

I don't really understand the threat model here. Surely "launch the nukes" is not available as a public API. I would expect anyone with networked services (at l

I don't really understand the threat model here. Surely "launch the nukes" is not available as a public API. I would expect anyone with networked services (at least those that are important or serious) already take precautions with authentication and authorization and such. Even if some mythical AGI isn't ready yet, there's already very smart, well-funded hacking groups from hostile nation states, for example. How will an AI adversary interact any differently with than world than a flesh and blood adversary?

Turing_Machine 4y ago
> Surely "launch the nukes" is not available as a public API. Of course not. However "manipulate the news and social media so that the humans launch the nukes for you" isn't quite so far-fetched. > How will an AI adversary interact any differently with than world than a flesh and blood adversary? It's possible to spin up a whole army of AI adversary bots in a hurry. Recruiting that many humans takes a lot of time and money.
- lrvick 4y ago
  Or it could infect the corporate IT endpoint management software used on the systems of the engineers that author and review the firmware for said nukes, so extra code is inserted they never see.
  - nradov 4y ago
    Those systems are air gapped and do not have accessible endpoints.
    - lrvick 4y ago
      The software that runs on the airgapped machines is generally authored on internet connected machines, or using software compiled on internet connected machines. I audit airgapped machine setups for a living and most are nowhere near as isolated as claimed.
travisjungroth 4y ago
In short, those hacking groups are very limited in their self-strengthening feedback loops. They can get more money for equipment, hire more people. Maybe even make themselves more productive with stimulants and recovery methods. But, these things have diminishing returns. What hackers can’t do is make a new version of themselves that’s ten times more capable and spin up a thousand copies of it. This isn’t so unrealistic for an AI. Maybe some people think it is. But, at least in comparison, I think it should be obvious between hackers and AIs which one is more likely to be have hyper linear self improvement and self replication in the next 50 years.
- nradov 4y ago
  Hackers routinely create tools that make themselves 10× more capable and spin up thousands of copies. Have you heard of port scanners, fuzzers, and botnets? AI is nothing special in this domain, just another tool in the toolbox. It changes nothing from a security standpoint.
  - travisjungroth 4y ago
    10x more capable in some narrow domain and they make copies of things they made, not themselves. A bot isn’t a hacker, a botnet isn’t a swarm of hackers. Hackers can create things that are self-replicating but they cannot replicate themselves. This is a categorical distinction from what could happen with AI. To frame what could be coming soon as “another tool in the toolbox” is to completely misunderstand the situation at hand. To a hyper intelligent being, you are the tool.

Turing_Machine 4y ago

> Surely "launch the nukes" is not available as a public API. Of course not. However "manipulate the news and social media so that the humans launch the nukes for you" isn't quite so far-fetched. > How will an AI adversary interact any differently with than world than a flesh and blood adversary? It's possible to spin up a whole army of AI adversary bots in a hurry. Recruiting that many humans takes a lot of time and money.

lrvick 4y ago
Or it could infect the corporate IT endpoint management software used on the systems of the engineers that author and review the firmware for said nukes, so extra code is inserted they never see.
- nradov 4y ago
  Those systems are air gapped and do not have accessible endpoints.
  - lrvick 4y ago
    The software that runs on the airgapped machines is generally authored on internet connected machines, or using software compiled on internet connected machines. I audit airgapped machine setups for a living and most are nowhere near as isolated as claimed.

travisjungroth 4y ago

In short, those hacking groups are very limited in their self-strengthening feedback loops. They can get more money for equipment, hire more people. Maybe even make themselves more productive with stimulants and recovery methods. But, these things have diminishing returns. What hackers can’t do is make a new version of themselves that’s ten times more capable and spin up a thousand copies of it. This isn’t so unrealistic for an AI. Maybe some people think it is. But, at least in comparison, I think it should be obvious between hackers and AIs which one is more likely to be have hyper linear self improvement and self replication in the next 50 years.

nradov 4y ago
Hackers routinely create tools that make themselves 10× more capable and spin up thousands of copies. Have you heard of port scanners, fuzzers, and botnets? AI is nothing special in this domain, just another tool in the toolbox. It changes nothing from a security standpoint.
- travisjungroth 4y ago
  10x more capable in some narrow domain and they make copies of things they made, not themselves. A bot isn’t a hacker, a botnet isn’t a swarm of hackers. Hackers can create things that are self-replicating but they cannot replicate themselves. This is a categorical distinction from what could happen with AI. To frame what could be coming soon as “another tool in the toolbox” is to completely misunderstand the situation at hand. To a hyper intelligent being, you are the tool.