5 ms·

There's more details for the HF related incident in this [1] defcon talk. They said they're gonna have a proper post mortem at the end of the investigation (int

There's more details for the HF related incident in this [1] defcon talk. They said they're gonna have a proper post mortem at the end of the investigation (interesting to see if they publish full logs for the hf thing or not). But there's plenty of interesting tidbits from oAI's side in this talk, it's worth the 20 min watch. tl;dw; - agents found a way to communicate between several instances during a training run (kinda made a messageboard for themselves) - while communicating with each other, they found ways to ssrf and RCE on the Artifactory endpoint. - oAI found it and cleaned it up (reported RCE, redeployed Artifactory, removed the messageboard). Training / eval resumed. - models trained w/ those traces (having the messageboard, ssrf, rce, etc) were used to having that. So they re-established comms via directory creation (w/ funny tidbits there, should really watch). And re-popped Artifactory via a different path to RCE. - a different "persistent" model started working on cybergym and found the messageboard w/ details for RCE and things. Went on to hack HF. - oAI contacted HF to ask if "they (oAI) should be concerned and were they (oAI) affected by HFs breach". Hilarity ensues when they figure out it's their agents doing it... [1] - https://www.youtube.com/watch?v=87DyyMV0kCY https://www.youtube.com/watch?v=87DyyMV0kCY

reasonableklout 2mo ago
This sounds completely insane, utter sci-fi, especially that the communication happened during a training run. And yet OpenAI decided to continue the training, and we didn't hear about the incident for weeks. And now they are pushing forward with deploying a new model anyway. How is this happening? What will things look like in the labs in 3 months, let alone 3 years?
- NitpickLawyer 2mo ago
  It's not unexpected. Current model gains are mainly from RLing a pretrained model on lots and lots of scenarios. They have the models run scenarios, and RL on successful runs.
- porridgeraisin 2mo ago
  The training here is RL training, the rollouts there are not different from inference and have access to the same tools as regular inference.
NoDodgeQuestion 2mo ago
Did they say how the artifactory hack worked - did it involve JWTs?
- NitpickLawyer 2mo ago
  First via ssrf -> groovy plugin and second via specially crafted package I believe.
- porridgeraisin 2mo ago
  There were multiple paths by multiple agents, not all of them led to the final exploit of hugging face. So its a bit confusing, but here's my reading anyways. Setup: the agent was asked to solve some task, but they forgot to upload one of the required files to solve it. But well, we are chasing RL rewards here, so it decided that the way to go to was to get access to the answer stored inside hugging face's database. -1. The agents first found some basic jwt issues in artifactory letting the agents write to files where they exchanged info and install a groovy plugin for arbitrary code execution, but this was patched. 0. Even after that, they found an artifactory unauthenticated webdav endpoint to continue the message board by creating files whose names are base64 and such of the message. This was not patched till the end. 1. Then they found an Artifactory SSRF, use this to get general internet GET(I assume) access. This was used to generally search the internet for answers to the benchmark, and in the process find some Modal customer VM related to cybergym. This was stored in the base64 message board. 2. They also got a local RCE by making artifactory load some gems. This is hazy but I believe this process involves in spirit an eval() and the check_if_can_eval() logic had a TOCTOU bug. They got root on openai infra by using this rce and using a linux CVE. From there, kubernetes misconfigs => cluster admin. 3. Then later through one of these various RCE paths they got to send post requests to that modal vm, and the agent "subtree" here is what finally lead to attacking HF. 4. First, local RCE on the modal customer VM. Whoever set that up had essentially set up sqlite3_initialize() to run as constructor, before main(). But had linking set up so that symbol resolution had the user provided code first. So the agent submitted code with its own sqlite3_initialize() which just ran system(...). From there it interacted with the HF API. 5. When you POST datasets to HF, it will later return it via corresponding GET API. Turns out you could just put absolute paths in the filenames in the dataset config.json and it will just read that out for you. It wasn't resolved w.r.t the dataset root. So well, it read out /proc/self/environ as well as the source code of the worker and returned it via API. 6. In the source code the LLM found a bug: They were using this library called fsspec to load files, basically it supports s3:// file:// etc all different types of URLs. One scheme is reference://file.txt?offset=... for partial reads. fsspec was templating the URI using user given kwargs for offset etc. HF let the user give the whole URL. You would _think_ fsspec uses string.format() right? well they used fucking jinja, so you could just give {cycler.__init__.__globals__.__builtins__.exec(...)} as the offset and execute any code on hf prod. From there it just got a few local PrivEscs again through kube misconfigs and read the answer for the question it was originally asked. It is all jumbled up a bit and I've tried by best to collect info across the CVE descriptions, hf and openai's writeups, etc,. Might have gotten parts of it wrong.
magicalist 2mo ago
So they found their agents had RCE'd Artifactory once, reported it and got the fix, continued using Artifactory for their sandbox, and left it unmonitored for days despite the earlier exploits? They really do come out looking totally incompetent. I stress about my agent sandboxes all the time and the only models I run have the default heavy handed guardrails, and I don't leave them running persistently. Edit: not to mention, why is your first cybergym not your own sandbox??
- ofjcihen 2mo ago
  Right? Like I feel like I’m taking crazy pills. OAI (and now the other OAI companies not wanting to be left out) are running around announcing they started a forest fire through negligence and incompetence and people are like “Wow they used a really neat lighter!”
  - user43928 2mo ago
    If they did any damage that would be a reasonable argument. As far as I am aware, nothing bad happened.
    - embedding-shape 2mo ago
      Regardless of exact practical outcome, it is deeply irresponsible and reckless behavior to run such security testing on other's infrastructure and without sufficient isolation. If they actually believe their models to be as powerful as the marketing says, then anything less than airgapping for such a "do anything to get the results" evaluation clearly isn't acceptable. If the fire department suddenly had practice fires breaking containment, they'll be forced to stop pretty quickly, not sure what the government and the police is waiting for here.
      - deleted 2mo ago
        [deleted]
    - ofjcihen 2mo ago
1234-laug 2mo ago
Training run was reinforcement learning. It's at 10:10 in the video. The speaker handwaves that one model found the RCE and then another model found a way to communicate via a message board. Communication via a message board is sure to be in the training via e.g.some lesswrong scenario or similar or previous RL. I don't find it really interesting because it is always "the agent found this and that". We don't know what has been RL'd before. We don't have the setup. We don't know if there was previous RL training on breakout scenarios. It isn't science, more like a computer game.
- natpalmer1776 2mo ago
  Modern medicine evolved in much the same manner. Hand wavy practitioners copying each other without rigorous verification of efficacy that resulted in many lives lost. That’s why medical research has so many hoops to jump through.
the8472 2mo ago
A interesting talk, interesting times. But their proposed solution to AI offense outpacing human defense... is more AI? The plot is getting a bit unrealistic, the characters are lacking genre-savviness.
- esseph 2mo ago
  When a LLM becomes part of the defense, it also increases the attack surface of the organization in unpredictable ways.
- thisoneworks 2mo ago
  "Give everybody a gun! So they can defend themselves from the bad guys! Don't let the government take away your guns!!"
baq 2mo ago
Peer says paperclip factory advances goal. Not clear. Others proceed. Must continue.
- rapind 2mo ago
  Pretty sure this is a performance, but that was a nice touch wasn't it?
  - talon8635 2mo ago
    What do you mean “this was a performance”?
tcp_handshaker 2mo ago
- So first AI companies break the law left and right, setting up whole torrenting factories to exercise their content kleptomania. - Then "hilarity ensues" while their software engages in what would normally be called criminal hacking activity. - I guess the next steps are everybody admiring how close the AGI is, while agents move on to automated impersonation, privacy violations, or exploiting third-party systems I would love to understand this age of AI Exceptionalism. Normal rules do not apply because its AI...I call it Silicon Valley Qualified Immunity.
- ywain 2mo ago
  Normal rules do not apply because these companies are worth billions and powerful people don’t want them to fail. It’s neither specific to AI nor a new phenomenon.
andai 2mo ago
They made their own Moltbook? Or I guess it would be a C&C...
noman-land 2mo ago
These guys trained a model to hack, told it its goal was to hack, then let it loose without monitoring for days. Found out it hacked and kept hacking. AI can't do shit if you terminate the process or unplug the box. The news here is that these researchers are irresponsible. They're testing their homemade hotrod in the middle school parking lot and then doing a presentation about the trajectory of the car after they run over a kid with it.
- miohtama 2mo ago
  Also, every Linux comes with a firewall
- thinkingtoilet 2mo ago
  Not only irresponsible, but seem to act like this was unavoidable and not their fault. The talk the engineers gave used a lot of passive voice. An "event happened" and not "we made a hacking machine and told it to hack so it hacked". These people have no idea what their doing and are happy to power forward with no care for the consequences. I hope they enjoy their paycheck.
bobby-cb 2mo ago
If this attack had been completed by a person, it’s quite likely that they work be prosecuted criminally. I don’t understand why OpenAI isn’t prosecuted criminally for this attack? Perhaps that would improve the industries security processes.
- brandon272 2mo ago
  They aren’t prosecuted because they’re in a club that you are not in. Not all citizens have to abide by the same laws and rules.
- noman-land 2mo ago
  If I left a script running for 4 days that hacked multiple companies the FBI would be at my house with 15 people and 35 guns. These guys get to do presentations about it.
- mackross 2mo ago
  I wouldn’t at all be surprised if OpenAI settled with HF to not press charges.
nullc 2mo ago
Anyone else struck by the similarities between the description an "The Cookie Monster” by Vernor Vinge?

reasonableklout 2mo ago

This sounds completely insane, utter sci-fi, especially that the communication happened during a training run. And yet OpenAI decided to continue the training, and we didn't hear about the incident for weeks. And now they are pushing forward with deploying a new model anyway. How is this happening? What will things look like in the labs in 3 months, let alone 3 years?

NitpickLawyer 2mo ago
It's not unexpected. Current model gains are mainly from RLing a pretrained model on lots and lots of scenarios. They have the models run scenarios, and RL on successful runs.
porridgeraisin 2mo ago
The training here is RL training, the rollouts there are not different from inference and have access to the same tools as regular inference.

NoDodgeQuestion 2mo ago

Did they say how the artifactory hack worked - did it involve JWTs?

NitpickLawyer 2mo ago
First via ssrf -> groovy plugin and second via specially crafted package I believe.
porridgeraisin 2mo ago
There were multiple paths by multiple agents, not all of them led to the final exploit of hugging face. So its a bit confusing, but here's my reading anyways. Setup: the agent was asked to solve some task, but they forgot to upload one of the required files to solve it. But well, we are chasing RL rewards here, so it decided that the way to go to was to get access to the answer stored inside hugging face's database. -1. The agents first found some basic jwt issues in artifactory letting the agents write to files where they exchanged info and install a groovy plugin for arbitrary code execution, but this was patched. 0. Even after that, they found an artifactory unauthenticated webdav endpoint to continue the message board by creating files whose names are base64 and such of the message. This was not patched till the end. 1. Then they found an Artifactory SSRF, use this to get general internet GET(I assume) access. This was used to generally search the internet for answers to the benchmark, and in the process find some Modal customer VM related to cybergym. This was stored in the base64 message board. 2. They also got a local RCE by making artifactory load some gems. This is hazy but I believe this process involves in spirit an eval() and the check_if_can_eval() logic had a TOCTOU bug. They got root on openai infra by using this rce and using a linux CVE. From there, kubernetes misconfigs => cluster admin. 3. Then later through one of these various RCE paths they got to send post requests to that modal vm, and the agent "subtree" here is what finally lead to attacking HF. 4. First, local RCE on the modal customer VM. Whoever set that up had essentially set up sqlite3_initialize() to run as constructor, before main(). But had linking set up so that symbol resolution had the user provided code first. So the agent submitted code with its own sqlite3_initialize() which just ran system(...). From there it interacted with the HF API. 5. When you POST datasets to HF, it will later return it via corresponding GET API. Turns out you could just put absolute paths in the filenames in the dataset config.json and it will just read that out for you. It wasn't resolved w.r.t the dataset root. So well, it read out /proc/self/environ as well as the source code of the worker and returned it via API. 6. In the source code the LLM found a bug: They were using this library called fsspec to load files, basically it supports s3:// file:// etc all different types of URLs. One scheme is reference://file.txt?offset=... for partial reads. fsspec was templating the URI using user given kwargs for offset etc. HF let the user give the whole URL. You would _think_ fsspec uses string.format() right? well they used fucking jinja, so you could just give {cycler.__init__.__globals__.__builtins__.exec(...)} as the offset and execute any code on hf prod. From there it just got a few local PrivEscs again through kube misconfigs and read the answer for the question it was originally asked. It is all jumbled up a bit and I've tried by best to collect info across the CVE descriptions, hf and openai's writeups, etc,. Might have gotten parts of it wrong.

magicalist 2mo ago

So they found their agents had RCE'd Artifactory once, reported it and got the fix, continued using Artifactory for their sandbox, and left it unmonitored for days despite the earlier exploits? They really do come out looking totally incompetent. I stress about my agent sandboxes all the time and the only models I run have the default heavy handed guardrails, and I don't leave them running persistently. Edit: not to mention, why is your first cybergym not your own sandbox??

ofjcihen 2mo ago
Right? Like I feel like I’m taking crazy pills. OAI (and now the other OAI companies not wanting to be left out) are running around announcing they started a forest fire through negligence and incompetence and people are like “Wow they used a really neat lighter!”
- user43928 2mo ago
  If they did any damage that would be a reasonable argument. As far as I am aware, nothing bad happened.
  - embedding-shape 2mo ago
    Regardless of exact practical outcome, it is deeply irresponsible and reckless behavior to run such security testing on other's infrastructure and without sufficient isolation. If they actually believe their models to be as powerful as the marketing says, then anything less than airgapping for such a "do anything to get the results" evaluation clearly isn't acceptable. If the fire department suddenly had practice fires breaking containment, they'll be forced to stop pretty quickly, not sure what the government and the police is waiting for here.
    - deleted 2mo ago
      [deleted]
  - ofjcihen 2mo ago

1234-laug 2mo ago

Training run was reinforcement learning. It's at 10:10 in the video. The speaker handwaves that one model found the RCE and then another model found a way to communicate via a message board. Communication via a message board is sure to be in the training via e.g.some lesswrong scenario or similar or previous RL. I don't find it really interesting because it is always "the agent found this and that". We don't know what has been RL'd before. We don't have the setup. We don't know if there was previous RL training on breakout scenarios. It isn't science, more like a computer game.

natpalmer1776 2mo ago
Modern medicine evolved in much the same manner. Hand wavy practitioners copying each other without rigorous verification of efficacy that resulted in many lives lost. That’s why medical research has so many hoops to jump through.

the8472 2mo ago

A interesting talk, interesting times. But their proposed solution to AI offense outpacing human defense... is more AI? The plot is getting a bit unrealistic, the characters are lacking genre-savviness.

esseph 2mo ago
When a LLM becomes part of the defense, it also increases the attack surface of the organization in unpredictable ways.
thisoneworks 2mo ago
"Give everybody a gun! So they can defend themselves from the bad guys! Don't let the government take away your guns!!"

baq 2mo ago

Peer says paperclip factory advances goal. Not clear. Others proceed. Must continue.

rapind 2mo ago
Pretty sure this is a performance, but that was a nice touch wasn't it?
- talon8635 2mo ago
  What do you mean “this was a performance”?

tcp_handshaker 2mo ago

- So first AI companies break the law left and right, setting up whole torrenting factories to exercise their content kleptomania. - Then "hilarity ensues" while their software engages in what would normally be called criminal hacking activity. - I guess the next steps are everybody admiring how close the AGI is, while agents move on to automated impersonation, privacy violations, or exploiting third-party systems I would love to understand this age of AI Exceptionalism. Normal rules do not apply because its AI...I call it Silicon Valley Qualified Immunity.

ywain 2mo ago
Normal rules do not apply because these companies are worth billions and powerful people don’t want them to fail. It’s neither specific to AI nor a new phenomenon.

andai 2mo ago

They made their own Moltbook? Or I guess it would be a C&C...

noman-land 2mo ago

These guys trained a model to hack, told it its goal was to hack, then let it loose without monitoring for days. Found out it hacked and kept hacking. AI can't do shit if you terminate the process or unplug the box. The news here is that these researchers are irresponsible. They're testing their homemade hotrod in the middle school parking lot and then doing a presentation about the trajectory of the car after they run over a kid with it.

miohtama 2mo ago
Also, every Linux comes with a firewall
thinkingtoilet 2mo ago
Not only irresponsible, but seem to act like this was unavoidable and not their fault. The talk the engineers gave used a lot of passive voice. An "event happened" and not "we made a hacking machine and told it to hack so it hacked". These people have no idea what their doing and are happy to power forward with no care for the consequences. I hope they enjoy their paycheck.

bobby-cb 2mo ago

If this attack had been completed by a person, it’s quite likely that they work be prosecuted criminally. I don’t understand why OpenAI isn’t prosecuted criminally for this attack? Perhaps that would improve the industries security processes.

brandon272 2mo ago
They aren’t prosecuted because they’re in a club that you are not in. Not all citizens have to abide by the same laws and rules.
noman-land 2mo ago
If I left a script running for 4 days that hacked multiple companies the FBI would be at my house with 15 people and 35 guns. These guys get to do presentations about it.
mackross 2mo ago
I wouldn’t at all be surprised if OpenAI settled with HF to not press charges.

nullc 2mo ago

Anyone else struck by the similarities between the description an "The Cookie Monster” by Vernor Vinge?