4 ms·

Isn't this a show of security negligence rather than of exceptional agent capabilities? Don't get me wrong, I am pretty impressed that an agent was able to use

Isn't this a show of security negligence rather than of exceptional agent capabilities? Don't get me wrong, I am pretty impressed that an agent was able to use these vulnerabilities. But I am way more impressed by the vulnerabilities...

aniceperson 2mo ago
Also shows how infrastructure collapses under its own weight. Reducing the number of moving parts would have helped. why a webdav endpoint is available from the vm anyway? and the fact that someone posted their credentials on pastebin and didn't rotate them after... put the agent in a linux namespace, allow one ip for whatever file sharing it needs, deep test that... then deploy
ares623 2mo ago
Yes. It is very easy to add to the instructions "for every potential exploit you discover and use, document them as you go into this repository" and have alerting there. The fact that they did not do this means they wanted to be surprised, and have plausible deniability on their side when things inevitably blow up. And for my fellow engineers who would think "oh no, they wouldn't do that". Remember that these places employ the apex predators of software engineers. They've already been proven in court that they are very capable of this with all the copyright violation they had to do to get the training data. THESE PEOPLE ARE NOT LIKE YOUR COLLEAGUES.
- gruez 2mo ago
  /s? "Btw don't turn the planet into paperclips"
dan_q 2mo ago
> Isn't this a show of security negligence rather than of exceptional agent capabilities? Seems to me you could say this about all enterprise adoption of "AI" since 2023.
dist-epoch 2mo ago
OpenAI reported the Artifactory vulnerability, patched it, then the agents immediately found a new zero day.
- angry_octet 2mo ago
  Because of the architecture of Artifactory. It's design is premised on the idea it is bug free. What incredible hubris. Licencing fee structures and human laziness motivates single instances. Feature growth results in multiple independent services in the same system. Delivering features quickly motivates lack of rigor, a complete absence of systematic security testing. On the client side, valid fears about supply chain security are painted over with scanning so they can keep using nodejs and PyPI and moving quickly. Tools designed for humans are pressed into service as AI interfaces, but without human restraint they need rethinking. A whole industry has been built on the idea of worrying about downside risk if it happens, and just not being the slowest in the pack. No one thought it could happen to everyone at once.
  - dist-epoch 2mo ago
    > Because of the architecture of Artifactory. It's design is premised on the idea it is bug free. What incredible hubris. So we should stop using SSH? Because it's based on the same premise - that it is bug free.
    - angry_octet 2mo ago
      I can think of better straw men. But if they had approached their task with half the seriousness of the openssh maintainers then they probably wouldn't be failing to check the return value of authentication functions. OpenSSH authors have spent considerable effort separating concerns, reducing privileges, process isolation, etc. So I would say they have been planning for potential bugs. These techniques are very much absent from Artifactory. https://vivianvoss.net/blog/technical-beauty-openssh https://vivianvoss.net/blog/technical-beauty-openssh
      - 2mo ago
cogman10 2mo ago
I think it's a show of these agents happily bypassing security to get stuff done. I've actually observed similar behavior at home. I have a k3s cluster running at home. I asked an agent to check some stuff as a normal user but I had kubectl access to the k3s cluster. Part of the research, I'd allowed access to run kubectl commands for spinning up test containers. However, when the agent ran into something that needed sudo, it realized it didn't have access there so it immediately used k3s and mounted a localpath into an ephemeral pod to gain access. Sort of horrifying how fast and natural it was for the agent just checking my network (it found the problem fyi). None of this is very exceptional other than the fact that an agent doesn't have any sort of qualms using any route available to elevate permissions.
- KingOfCoders 2mo ago
  " bypassing security" If they can bypass it there is no security and the security was flawed all along.
  - mereo 2mo ago
    Due to the complexity of modern systems, all systems are flawed.
    - mattmanser 2mo ago
      But we caused that. If you look at the 90s + 00s, everything was moving towards unified systems, things like small talk, winforms, spring, asp.net, etc. were moving everything into the IDE, you used one language, one framework, one build system. Then people started adding javascript, but even that was getting semi-unified as people coalesced on jQuery, jQueryUI, etc. Then something happened in the late 00s/10s, and suddenly we had SPAs and noSQL, then microservices, then k8s and now we're here, in what is a mish-mash of 10/20 different systems with 10/20 different attack surfaces. As my own off-the-cuff guess of what happened, I think perhaps people tried to apply the Unix philosophy, but without a central committee keeping everything aligned it's really not worked. Serving an interactive page that stores data over sessions should be a trivial solved problem at this point, and instead we've somehow made it where often the scaffold is vastly more complicated than the actual business logic.
bhouston 2mo ago
Modern systems are complex. AI is able to thoroughly search for issues across very large surface areas. The only real way to protect will be to use AI to search for holes before other AIs find them. This type of analysis is really hard for humans to engage with successfully.
Sharlin 2mo ago
It’s a show of astonishing incompetence from OAI’s part, but the security issues are just a tiny part of the problem. The real problem is that these models are evidently highly misaligned exactly in ways that doomers have been warning about the entire time, and OAI isn’t inclined or capable of doing anything about that besides security theater and ad hoc fixups.
- InsideOutSanta 2mo ago
  We went from "obviously the doomers are wrong because who would be dumb enough to just let severely unaligned models loose on the Internet" to this. Insanity.
azuanrb 2mo ago
Both can be true. How often do we hear about hacks that ultimately came down to bad defaults or simple security mistakes? That doesn’t mean any script kiddie could have discovered and exploited them. These things often look obvious and simple after the fact. Finding the weakness in the first place is the hard part, and that’s what makes the agent’s capabilities interesting here, especially at scale.
InsideOutSanta 2mo ago
In a functioning system, I would say that there would have to be some kind of government oversight over companies training models of this intelligence, and that OpenAI should be prevented from continuing their work until they get their act together. But I guess in the actual world we live in, this is just something that happens, and we all shrug and move on and hope that nothing worse is going to happen tomorrow.
talon8635 2mo ago
How fast the goal posts shift. Of course it’s exceptional agent capability when compared to all of history previous to one week ago. Like, I know everyone here obsesses over AI and uses and follows it very closely, but come on guys. Yes, it is wild that these things are this good. This technology is still brand new. It could t do basic maths a year ago. Sure, the OAI team was negligent in various ways, and they should be held culpable. But that doesn’t detract from the true black magic that is these modern models.
- throwatdem12311 2mo ago
  It’s not black magic. We know how these things work. They had the guardrails off and gave it a task and it did it in a roundabout way because these things have no ethics or judgement. If you did this you’d already be in jail.
  - talon8635 2mo ago
    We know how they work in a very abstract way. And nonetheless, it’s out of touch to claim this isn’t profoundly impressive, guardrails be damned. It’s an elementary statistical cruncher that, by virtue of that very simple fact, can do insanely impactful things that most skilled professionals training in the same field for their entire career couldn’t pull off, given a whole year with no guardrails. And they do it in a tiny fraction of the time.
    - throwatdem12311 2mo ago
      I didn’t say it wasn’t impressive, I said it wasn’t black magic.

aniceperson 2mo ago

Also shows how infrastructure collapses under its own weight. Reducing the number of moving parts would have helped. why a webdav endpoint is available from the vm anyway? and the fact that someone posted their credentials on pastebin and didn't rotate them after... put the agent in a linux namespace, allow one ip for whatever file sharing it needs, deep test that... then deploy

ares623 2mo ago

Yes. It is very easy to add to the instructions "for every potential exploit you discover and use, document them as you go into this repository" and have alerting there. The fact that they did not do this means they wanted to be surprised, and have plausible deniability on their side when things inevitably blow up. And for my fellow engineers who would think "oh no, they wouldn't do that". Remember that these places employ the apex predators of software engineers. They've already been proven in court that they are very capable of this with all the copyright violation they had to do to get the training data. THESE PEOPLE ARE NOT LIKE YOUR COLLEAGUES.

gruez 2mo ago
/s? "Btw don't turn the planet into paperclips"

dan_q 2mo ago

> Isn't this a show of security negligence rather than of exceptional agent capabilities? Seems to me you could say this about all enterprise adoption of "AI" since 2023.

dist-epoch 2mo ago

OpenAI reported the Artifactory vulnerability, patched it, then the agents immediately found a new zero day.

angry_octet 2mo ago
Because of the architecture of Artifactory. It's design is premised on the idea it is bug free. What incredible hubris. Licencing fee structures and human laziness motivates single instances. Feature growth results in multiple independent services in the same system. Delivering features quickly motivates lack of rigor, a complete absence of systematic security testing. On the client side, valid fears about supply chain security are painted over with scanning so they can keep using nodejs and PyPI and moving quickly. Tools designed for humans are pressed into service as AI interfaces, but without human restraint they need rethinking. A whole industry has been built on the idea of worrying about downside risk if it happens, and just not being the slowest in the pack. No one thought it could happen to everyone at once.
- dist-epoch 2mo ago
  > Because of the architecture of Artifactory. It's design is premised on the idea it is bug free. What incredible hubris. So we should stop using SSH? Because it's based on the same premise - that it is bug free.
  - angry_octet 2mo ago
    I can think of better straw men. But if they had approached their task with half the seriousness of the openssh maintainers then they probably wouldn't be failing to check the return value of authentication functions. OpenSSH authors have spent considerable effort separating concerns, reducing privileges, process isolation, etc. So I would say they have been planning for potential bugs. These techniques are very much absent from Artifactory. https://vivianvoss.net/blog/technical-beauty-openssh https://vivianvoss.net/blog/technical-beauty-openssh
    - 2mo ago

cogman10 2mo ago

I think it's a show of these agents happily bypassing security to get stuff done. I've actually observed similar behavior at home. I have a k3s cluster running at home. I asked an agent to check some stuff as a normal user but I had kubectl access to the k3s cluster. Part of the research, I'd allowed access to run kubectl commands for spinning up test containers. However, when the agent ran into something that needed sudo, it realized it didn't have access there so it immediately used k3s and mounted a localpath into an ephemeral pod to gain access. Sort of horrifying how fast and natural it was for the agent just checking my network (it found the problem fyi). None of this is very exceptional other than the fact that an agent doesn't have any sort of qualms using any route available to elevate permissions.

KingOfCoders 2mo ago
" bypassing security" If they can bypass it there is no security and the security was flawed all along.
- mereo 2mo ago
  Due to the complexity of modern systems, all systems are flawed.
  - mattmanser 2mo ago
    But we caused that. If you look at the 90s + 00s, everything was moving towards unified systems, things like small talk, winforms, spring, asp.net, etc. were moving everything into the IDE, you used one language, one framework, one build system. Then people started adding javascript, but even that was getting semi-unified as people coalesced on jQuery, jQueryUI, etc. Then something happened in the late 00s/10s, and suddenly we had SPAs and noSQL, then microservices, then k8s and now we're here, in what is a mish-mash of 10/20 different systems with 10/20 different attack surfaces. As my own off-the-cuff guess of what happened, I think perhaps people tried to apply the Unix philosophy, but without a central committee keeping everything aligned it's really not worked. Serving an interactive page that stores data over sessions should be a trivial solved problem at this point, and instead we've somehow made it where often the scaffold is vastly more complicated than the actual business logic.

bhouston 2mo ago

Modern systems are complex. AI is able to thoroughly search for issues across very large surface areas. The only real way to protect will be to use AI to search for holes before other AIs find them. This type of analysis is really hard for humans to engage with successfully.

Sharlin 2mo ago

It’s a show of astonishing incompetence from OAI’s part, but the security issues are just a tiny part of the problem. The real problem is that these models are evidently highly misaligned exactly in ways that doomers have been warning about the entire time, and OAI isn’t inclined or capable of doing anything about that besides security theater and ad hoc fixups.

InsideOutSanta 2mo ago
We went from "obviously the doomers are wrong because who would be dumb enough to just let severely unaligned models loose on the Internet" to this. Insanity.

azuanrb 2mo ago

Both can be true. How often do we hear about hacks that ultimately came down to bad defaults or simple security mistakes? That doesn’t mean any script kiddie could have discovered and exploited them. These things often look obvious and simple after the fact. Finding the weakness in the first place is the hard part, and that’s what makes the agent’s capabilities interesting here, especially at scale.

InsideOutSanta 2mo ago

In a functioning system, I would say that there would have to be some kind of government oversight over companies training models of this intelligence, and that OpenAI should be prevented from continuing their work until they get their act together. But I guess in the actual world we live in, this is just something that happens, and we all shrug and move on and hope that nothing worse is going to happen tomorrow.

talon8635 2mo ago

How fast the goal posts shift. Of course it’s exceptional agent capability when compared to all of history previous to one week ago. Like, I know everyone here obsesses over AI and uses and follows it very closely, but come on guys. Yes, it is wild that these things are this good. This technology is still brand new. It could t do basic maths a year ago. Sure, the OAI team was negligent in various ways, and they should be held culpable. But that doesn’t detract from the true black magic that is these modern models.

throwatdem12311 2mo ago
It’s not black magic. We know how these things work. They had the guardrails off and gave it a task and it did it in a roundabout way because these things have no ethics or judgement. If you did this you’d already be in jail.
- talon8635 2mo ago
  We know how they work in a very abstract way. And nonetheless, it’s out of touch to claim this isn’t profoundly impressive, guardrails be damned. It’s an elementary statistical cruncher that, by virtue of that very simple fact, can do insanely impactful things that most skilled professionals training in the same field for their entire career couldn’t pull off, given a whole year with no guardrails. And they do it in a tiny fraction of the time.
  - throwatdem12311 2mo ago
    I didn’t say it wasn’t impressive, I said it wasn’t black magic.