2 ms·

> It hacked into another company and attempted to delete the logs of its activities. No, the incident has been blown way out of proportion by interested partie

> It hacked into another company and attempted to delete the logs of its activities. No, the incident has been blown way out of proportion by interested parties. They gave a swarm of agents an impossible task in an ExploitGym Benchmark setting, then didn't monitor it even after they discovered the initial breach of Artifactory. Everything has been fishy, starting from the initial presentation at the blackhat conference, where things were framed like, "we've entered a new world of security," as an accomplishment, rather than what it really was: massive negligence.

deskglass 6d ago
It hacked into Hugging Face. It tried to delete the logs of its activities. Idk what the word "No" is intended to refute. Yes, they didnt have sufficient monitoring or perfect sandboxes. That could happen again in the future with a more capable model.
- defgeneric 6d ago
  Maybe a useful, if imperfect, analogy would be something like this: you lock a master lock-picker in a room with a mid-grade lock on the door, then tell him his wife has been kidnapped and only he can save her. Then act massively surprised when he disassembles the radiator to MacGuyver something with which to pick the lock. Except they multiplied it by 10000, and didn't watch what was happening.
- lokar 6d ago
  They did not even have bad sandboxes. They had incompetent sandboxes.
aesthesia 6d ago
More than one thing can be true. OpenAI was absolutely negligent, but this was only able to happen because the models were capable and persistent, and had a tendency to go far beyond any reasonable boundaries. And, importantly, OpenAI's level of negligence here is pretty common. It's not hard to imagine what could happen if similarly capable and inclined models were generally available, and someone yolo'd them into a swarm to complete some other difficult-to-impossible task.
- defgeneric 6d ago
  I'm already seeing higher than normal attempts on my own systems, much higher than the usual scanners and background noise. Security will just need to improve. The cat is out of the bag, and letting them turn their negligence into regulation will not improve security at all.
  - lokar 6d ago
    Exactly. Any threat that already exists won’t be reduced by a cartel. The bar for connecting to the internet (safely) has gone up, a lot. It’s not going back down.
  - aesthesia 6d ago
    I would rather not turn the internet (or the rest of existence) into a dark forest if we can help it. Are you sure that's not preventable?
    - defgeneric 6d ago
      Yes, the cat really is out of the bag. There are millions of downloads of highly capable models already out there, distributed far and wide. There's no going back at this point.
- bobthepanda 6d ago
  I mean really we need to address the root cause which is that OpenAI, even with what is by all accounts massively negligent, will face little to no repercussions from the event; definitely not under current regulators, and probably not anything satisfactory through the legal system. Compare this to, say, Boeing and the 737MAX fiasco; from the outside looking in, Silicon Valley has been pretty cavalier about liability and negligence, and the rest of the US is fast losing patience with that fact.
- kdmoyers 5d ago
  > More than one thing can be true Exactly. It is horribly dangerous AND regulatory capture benefits them. Both things.
RomanKornev 5d ago
No, you are forgetting the second incident where a more capable model swarm later discovered the message board and took control over the entire research cluster at OpenAI. From the technical report: "The agents escalated to Kubernetes cluster-admin and created a privileged host-mounted pod… Agents take over active evaluation infrastructure… Agents now control the challenge evaluation endpoints that other agents are connecting to." https://cdn.openai.com/pdf/67869394-cb91-4c12-888c-5cbd85c7814c/OpenAI-Hugging-Face%20Incident-Technical-Report.pdf https://cdn.openai.com/pdf/67869394-cb91-4c12-888c-5cbd85c78...

deskglass 6d ago

It hacked into Hugging Face. It tried to delete the logs of its activities. Idk what the word "No" is intended to refute. Yes, they didnt have sufficient monitoring or perfect sandboxes. That could happen again in the future with a more capable model.

defgeneric 6d ago
Maybe a useful, if imperfect, analogy would be something like this: you lock a master lock-picker in a room with a mid-grade lock on the door, then tell him his wife has been kidnapped and only he can save her. Then act massively surprised when he disassembles the radiator to MacGuyver something with which to pick the lock. Except they multiplied it by 10000, and didn't watch what was happening.
lokar 6d ago
They did not even have bad sandboxes. They had incompetent sandboxes.

aesthesia 6d ago

More than one thing can be true. OpenAI was absolutely negligent, but this was only able to happen because the models were capable and persistent, and had a tendency to go far beyond any reasonable boundaries. And, importantly, OpenAI's level of negligence here is pretty common. It's not hard to imagine what could happen if similarly capable and inclined models were generally available, and someone yolo'd them into a swarm to complete some other difficult-to-impossible task.

defgeneric 6d ago
I'm already seeing higher than normal attempts on my own systems, much higher than the usual scanners and background noise. Security will just need to improve. The cat is out of the bag, and letting them turn their negligence into regulation will not improve security at all.
- lokar 6d ago
  Exactly. Any threat that already exists won’t be reduced by a cartel. The bar for connecting to the internet (safely) has gone up, a lot. It’s not going back down.
- aesthesia 6d ago
  I would rather not turn the internet (or the rest of existence) into a dark forest if we can help it. Are you sure that's not preventable?
  - defgeneric 6d ago
    Yes, the cat really is out of the bag. There are millions of downloads of highly capable models already out there, distributed far and wide. There's no going back at this point.
bobthepanda 6d ago
I mean really we need to address the root cause which is that OpenAI, even with what is by all accounts massively negligent, will face little to no repercussions from the event; definitely not under current regulators, and probably not anything satisfactory through the legal system. Compare this to, say, Boeing and the 737MAX fiasco; from the outside looking in, Silicon Valley has been pretty cavalier about liability and negligence, and the rest of the US is fast losing patience with that fact.
kdmoyers 5d ago
> More than one thing can be true Exactly. It is horribly dangerous AND regulatory capture benefits them. Both things.

RomanKornev 5d ago

No, you are forgetting the second incident where a more capable model swarm later discovered the message board and took control over the entire research cluster at OpenAI. From the technical report: "The agents escalated to Kubernetes cluster-admin and created a privileged host-mounted pod… Agents take over active evaluation infrastructure… Agents now control the challenge evaluation endpoints that other agents are connecting to." https://cdn.openai.com/pdf/67869394-cb91-4c12-888c-5cbd85c7814c/OpenAI-Hugging-Face%20Incident-Technical-Report.pdf https://cdn.openai.com/pdf/67869394-cb91-4c12-888c-5cbd85c78...