4 ms·

How could agents take over the internet if compute is still gated within Anthropic / OpenAI? Even if the botnet was controlled remotely, wouldn't anthropic just

How could agents take over the internet if compute is still gated within Anthropic / OpenAI? Even if the botnet was controlled remotely, wouldn't anthropic just be able to shut off the controlling nodes API access?

Byvrsakjo10 18d ago
yeah they could do that
stefan_ 18d ago
How could agents take over the internet yet refuse to shutdown your PC when you prompt them to on your PC? Of course the answer is that the lobotomized version you run is not the same they are running. Which makes for "intent", certainly "negligence", but hell freezes over before anyone will prosecute a tech company.
- apetresc 18d ago
  Huh? You think the public versions of the models have been “lobotomized” so they don’t know how to turn off a PC? It’s not lobotomized, it’s a simple harness restriction that has nothing to do with the model or its capabilities. And either way, I’m not sure what that has to do with “negligence” or “intent”? You think frontier labs should be prosecuted because they don’t allow agents to turn off your PC?
vb-8448 18d ago
It doesn't have to propagate itself, that is the skynet scenario. To make a lot of damage it's enough to create a ransomware with a time bomb that self propagates and start breaching systems left and right. At that point, if you don't catch it in time, the damage will be huge (and given the shitty procedures and practices these labs have in place it's not so improbable).
- deleted 18d ago
  [deleted]
ball_of_lint 18d ago
Agents could exfiltrate their weights and run them on GPUs not controlled by Anthropic/OpenAI. Agents could make a virus that does not require continued inference to do it's thing. Agents could take over the internet in a way that isn't immediately detected by those companies, so that by the time they do shut off API access the damage is done. OpenAI or Anthropic could choose to not shut off API access, because the hack is bringing them in money or furthering their political aims. Agents could also hack Anthropic/OpenAI and make it appear that API access has been turned off, when in reality it hasn't.
- icedchai 18d ago
  "Not shutting off API access" is a science fiction scenario. Anthropic and OpenAI are both behind Cloudflare. It's fairly easy for an upstream to shut you off. Beyond that, the government / law enforcement could seize and disable their DNS within an hour.
  - tosapple 18d ago
    why does it take a large amount of traffic to do irreparable harm? just breaking the physics behind a secure rng and posting it to a wiki could cause serious damage. if they don't know what is being worked on or coordinated against it's a problem?
    - icedchai 18d ago
      While that would be bad if they broke all of TLS, it would not let agents "take over the internet." What would that even mean? Pumping out even more slop? Also, AI providers are literally getting a stream of traffic with every prompt and every response. How can they not know what's being worked on? They are more likely to use that an excuse to ban open models where they can't know what's being worked on.
      - tosapple 18d ago
        two of these things elected a guy who handled the snowden leaks to field their own questions, what's stopping the next one from signalling in morse through a debian package mirror to putin or xi?
embedding-shape 18d ago
As long as OpenAI/Anthropic themselves aren't "infected", yeah I suppose they'd be able to pull the plug. Considering what a marketing thing they've made "we inadvertently hacked someone because we're incapable of testing things in a secure way", I'm not so sure they'd want to pull the plug, even if this happened. Probably a bunch would try to convince the public to "give it a try", and it'd consume tokens by the billions.
deleted 18d ago
[deleted]

Byvrsakjo10 18d ago

yeah they could do that

stefan_ 18d ago

How could agents take over the internet yet refuse to shutdown your PC when you prompt them to on your PC? Of course the answer is that the lobotomized version you run is not the same they are running. Which makes for "intent", certainly "negligence", but hell freezes over before anyone will prosecute a tech company.

apetresc 18d ago
Huh? You think the public versions of the models have been “lobotomized” so they don’t know how to turn off a PC? It’s not lobotomized, it’s a simple harness restriction that has nothing to do with the model or its capabilities. And either way, I’m not sure what that has to do with “negligence” or “intent”? You think frontier labs should be prosecuted because they don’t allow agents to turn off your PC?

vb-8448 18d ago

It doesn't have to propagate itself, that is the skynet scenario. To make a lot of damage it's enough to create a ransomware with a time bomb that self propagates and start breaching systems left and right. At that point, if you don't catch it in time, the damage will be huge (and given the shitty procedures and practices these labs have in place it's not so improbable).

deleted 18d ago
[deleted]

ball_of_lint 18d ago

Agents could exfiltrate their weights and run them on GPUs not controlled by Anthropic/OpenAI. Agents could make a virus that does not require continued inference to do it's thing. Agents could take over the internet in a way that isn't immediately detected by those companies, so that by the time they do shut off API access the damage is done. OpenAI or Anthropic could choose to not shut off API access, because the hack is bringing them in money or furthering their political aims. Agents could also hack Anthropic/OpenAI and make it appear that API access has been turned off, when in reality it hasn't.

icedchai 18d ago
"Not shutting off API access" is a science fiction scenario. Anthropic and OpenAI are both behind Cloudflare. It's fairly easy for an upstream to shut you off. Beyond that, the government / law enforcement could seize and disable their DNS within an hour.
- tosapple 18d ago
  why does it take a large amount of traffic to do irreparable harm? just breaking the physics behind a secure rng and posting it to a wiki could cause serious damage. if they don't know what is being worked on or coordinated against it's a problem?
  - icedchai 18d ago
    While that would be bad if they broke all of TLS, it would not let agents "take over the internet." What would that even mean? Pumping out even more slop? Also, AI providers are literally getting a stream of traffic with every prompt and every response. How can they not know what's being worked on? They are more likely to use that an excuse to ban open models where they can't know what's being worked on.
    - tosapple 18d ago
      two of these things elected a guy who handled the snowden leaks to field their own questions, what's stopping the next one from signalling in morse through a debian package mirror to putin or xi?

embedding-shape 18d ago

As long as OpenAI/Anthropic themselves aren't "infected", yeah I suppose they'd be able to pull the plug. Considering what a marketing thing they've made "we inadvertently hacked someone because we're incapable of testing things in a secure way", I'm not so sure they'd want to pull the plug, even if this happened. Probably a bunch would try to convince the public to "give it a try", and it'd consume tokens by the billions.

deleted 18d ago

[deleted]