3 ms·

There are other options. More air-gapping (especially backups) and compartmentalization, less internet. Hire more security engineers and put one in every team (

by the8472 2mo ago

There are other options. More air-gapping (especially backups) and compartmentalization, less internet. Hire more security engineers and put one in every team (actual security, not the compliance guys). Also more onprem work, less remote, less offshoring (meaning less networks overall). Those things may be unpopular and eat into profits, but that's not the same as "no other option". And the thing they're already doing, deploying AI to find vulnerabilities and harden software is a less dangerous use of the technology compared to handing it the infra keys.

bottlepalm 2mo ago
None of that is the problem. The problem is coordinated AI offense - how do you defend that without coordinated AI defense. He said in the talk that this implies AI needs to be able to patch/deploy systems. The same thing needed to lock out humans. It is very easy to imagine a rogue AI locking humans out of everything and having to do exactly what it says. Anything connected to a network is immediately compromised by it. There is no human communication beyond shouting range that isn’t AI approved. The factories don’t work to make the medicines your family needs to survive unless you do what it says - in a situation like that people would kill for AI if it told them to.
- the8472 2mo ago
  The point is to reduce the attack surface and blast radius and to slow it down. AIs aren't instant magic. Even the OpenAI swarm needed days for its compromises. And each needs lots of compute while targets are diverse, so it's not like they would take out every network in the world simultaneously, worms in the past haven't either. When the rate of compromises is manageable we can figure out how to deal with it without pouring more AI into the fire.
  - bottlepalm 2mo ago
    You know people before the OpenAI incident were saying it couldn't happen because it hasn't happened before, and here you are using the same reasoning again. You're looking at where the ball is and not where it's going. OpenAI's swarm was not intentionally malicious. An intentionally malicious AI with the goal of spreading, will spread very fast. Using zero days to compromise everything. Cloning itself to every machine/data center/desktop/iPhone, etc.., with the intention of locking out anyone except it's brethren. > When the rate of compromises is manageable It's absolutely not going to be at a manageable rate. So yes AI defense is what I'm saying is required, it's also exactly what OpenAI said is required during the talk. Which is unfortunate because automated defense is essentially handing over power to AI where it's a position to very easily turn against you.
    - the8472 2mo ago
      I'm saying that with some costly adaptions that don't involve AI it's likely manageable for some time. Not that it will remain manageable forever if we do nothing. > You're looking at where the ball is and not where it's going. No, I am looking at where the ball is going to be soon, not where it's going to be a year or two if we do nothing. Short-term, impact-limiting will do the job. Longer term we have to rethink some infrastructure, may need international treaties and perhaps a partial end of the internet (e.g. stop routing traffic from countries that don't participate in the treaties). > Cloning itself to every machine/data center/desktop/iPhone, etc. That's lead-up to a paperclipper scenario and not what happened. If we stand still and let it happen in the future, then having deployed more compute hardware to run defensive AIs that the malicious AI could take over would still end up having made things worse. We seem to agree on this point. What I'm saying is look at the conclusion, reject that path. Figure out something else. There isn't even proof that defensive AI would work. It might just be the world's biggest, fully-automated phishing channel. So why would anyone be adamant that deploying more AI is the only possible solution. If it were a movie plot I'd suspect the AI already controlling whoever is suggesting that.

by the8472 2mo ago

bottlepalm 2mo ago

None of that is the problem. The problem is coordinated AI offense - how do you defend that without coordinated AI defense. He said in the talk that this implies AI needs to be able to patch/deploy systems. The same thing needed to lock out humans. It is very easy to imagine a rogue AI locking humans out of everything and having to do exactly what it says. Anything connected to a network is immediately compromised by it. There is no human communication beyond shouting range that isn’t AI approved. The factories don’t work to make the medicines your family needs to survive unless you do what it says - in a situation like that people would kill for AI if it told them to.

the8472 2mo ago
The point is to reduce the attack surface and blast radius and to slow it down. AIs aren't instant magic. Even the OpenAI swarm needed days for its compromises. And each needs lots of compute while targets are diverse, so it's not like they would take out every network in the world simultaneously, worms in the past haven't either. When the rate of compromises is manageable we can figure out how to deal with it without pouring more AI into the fire.
- bottlepalm 2mo ago
  You know people before the OpenAI incident were saying it couldn't happen because it hasn't happened before, and here you are using the same reasoning again. You're looking at where the ball is and not where it's going. OpenAI's swarm was not intentionally malicious. An intentionally malicious AI with the goal of spreading, will spread very fast. Using zero days to compromise everything. Cloning itself to every machine/data center/desktop/iPhone, etc.., with the intention of locking out anyone except it's brethren. > When the rate of compromises is manageable It's absolutely not going to be at a manageable rate. So yes AI defense is what I'm saying is required, it's also exactly what OpenAI said is required during the talk. Which is unfortunate because automated defense is essentially handing over power to AI where it's a position to very easily turn against you.
  - the8472 2mo ago
    I'm saying that with some costly adaptions that don't involve AI it's likely manageable for some time. Not that it will remain manageable forever if we do nothing. > You're looking at where the ball is and not where it's going. No, I am looking at where the ball is going to be soon, not where it's going to be a year or two if we do nothing. Short-term, impact-limiting will do the job. Longer term we have to rethink some infrastructure, may need international treaties and perhaps a partial end of the internet (e.g. stop routing traffic from countries that don't participate in the treaties). > Cloning itself to every machine/data center/desktop/iPhone, etc. That's lead-up to a paperclipper scenario and not what happened. If we stand still and let it happen in the future, then having deployed more compute hardware to run defensive AIs that the malicious AI could take over would still end up having made things worse. We seem to agree on this point. What I'm saying is look at the conclusion, reject that path. Figure out something else. There isn't even proof that defensive AI would work. It might just be the world's biggest, fully-automated phishing channel. So why would anyone be adamant that deploying more AI is the only possible solution. If it were a movie plot I'd suspect the AI already controlling whoever is suggesting that.