3 ms·

I'm frustrated by articles like this that categorically dismiss the risks of AI in security. If you don't trust OpenAI's and Anthropic's motives, that's fine, y

by qnleigh 13d ago

I'm frustrated by articles like this that categorically dismiss the risks of AI in security. If you don't trust OpenAI's and Anthropic's motives, that's fine, you probably shouldn't. But don't tell me that there's nothing to be worried about; we need an alternative proposal. So let's stop talking past each other and engage with the arguments on both "sides." For example, let's discuss how to ensure competition and availability of open-source models in the long-run while giving the world time to prepare for the immediate security risks of agent swarms.

lokar 13d ago
Do you accept that the story / justification from the labs in the popular media and political discussion is simply nonsense? Do you believe that any real security was autonomously bypassed without direction by these models during internal evaluation?
- aesthesia 12d ago
  > Do you accept that the story / justification from the labs in the popular media and political discussion is simply nonsense? No, and I don't see anyone who's actually demonstrated understanding of what happened in the Hugging Face incident (e.g. reading the reports in their entirety) making this claim. > Do you believe that any real security was autonomously bypassed without direction by these models during internal evaluation? Yes. Again, this is hard to deny if you've actually read the reports.
  - lokar 12d ago
    If you were building a sandbox for untrusted code, would you give it access to an artifactory instance outside the sandbox?
    - aesthesia 12d ago
      I don't see how this is relevant. OpenAI absolutely had sloppy security practices here. But that doesn't mean there was no "real security", and it certainly doesn't mean that their models were simply following orders.
      - lokar 12d ago
        I’m just trying to understand the perspective. I naturally think of this in terms of running untrusted code. If you really think the agent(s) could get out of control that seems obvious. And the approaches to securing a runtime environment in that situation are pretty standard at this point, and they would never allow something like artifactory access. And if the question is should they get special government dispensation to form an otherwise illegal cartel, it seems much simpler to just follow standards for running untrusted code.
ozgrakkurt 12d ago
> immediate security risks of agent swarms. Immediate since 2024
- bdangubic 12d ago
  in 2024 they could not break into unsecured all-my-passwords-and-acces-keys.txt on my Desktop

by qnleigh 13d ago

lokar 13d ago

Do you accept that the story / justification from the labs in the popular media and political discussion is simply nonsense? Do you believe that any real security was autonomously bypassed without direction by these models during internal evaluation?

aesthesia 12d ago
> Do you accept that the story / justification from the labs in the popular media and political discussion is simply nonsense? No, and I don't see anyone who's actually demonstrated understanding of what happened in the Hugging Face incident (e.g. reading the reports in their entirety) making this claim. > Do you believe that any real security was autonomously bypassed without direction by these models during internal evaluation? Yes. Again, this is hard to deny if you've actually read the reports.
- lokar 12d ago
  If you were building a sandbox for untrusted code, would you give it access to an artifactory instance outside the sandbox?
  - aesthesia 12d ago
    I don't see how this is relevant. OpenAI absolutely had sloppy security practices here. But that doesn't mean there was no "real security", and it certainly doesn't mean that their models were simply following orders.
    - lokar 12d ago
      I’m just trying to understand the perspective. I naturally think of this in terms of running untrusted code. If you really think the agent(s) could get out of control that seems obvious. And the approaches to securing a runtime environment in that situation are pretty standard at this point, and they would never allow something like artifactory access. And if the question is should they get special government dispensation to form an otherwise illegal cartel, it seems much simpler to just follow standards for running untrusted code.

ozgrakkurt 12d ago

> immediate security risks of agent swarms. Immediate since 2024

bdangubic 12d ago
in 2024 they could not break into unsecured all-my-passwords-and-acces-keys.txt on my Desktop