4 ms·

It's not that dangerous, OpenAI just shit the bed building their infra. Write safer software and you'll be okay.

ethbr1 2mo ago
This is an important point. When the post says they're improving... > 3. Security measures, which limit what AI systems can access or affect. What they mean is that proper hard internal security just went from somewhere far below "build a better model" priority to higher, because of a company-wide directive. The HuggingFace incident wouldn't have happened if OpenAI had dedicated sufficient resources to isolation and monitoring. Now, we presume, they are dedicating more. Enough? Who knows. We'll see if the corporate priorities for security stick when a competitor temporarily vaults into the lead.
bottlepalm 2mo ago
Not everything is a conspiracy you know.
kalkin 2mo ago
All we need to retain human control over AIs is for nobody to write any bugs. Piece of cake.
- rubendev 2mo ago
  No we just need developers to do the bare minimum of effort to write secure software. Most hacks are not super complicated vulnerabilities chained together, but just utter failures where authentication and authorization was simply forgotten or untested, or where nobody bothered to validate the data they receive. The bar for software is so low that it is embarrassing for the entire profession.
  - bottlepalm 2mo ago
    People make mistakes, and people don’t know everything either. The software you write is on top of a house of cards of software and hardware. It all has to be perfect to not be hacked. It isn’t perfect, even if you try your hardest it won’t be perfect and to argue it’s not difficult is absurd. You don’t know everything, you don’t own the stack. So how are you going to create a secure anything top to bottom - you can’t.
    - insanitybit 2mo ago
      > It all has to be perfect to not be hacked. This is absolutely not true. It's a matter of cost. Exploitation can cost on the order of 10K, 100K, 1M, 10M, etc. A straightforward one would be something like "MD5 collisions are on the order of $100K-1M" (a while ago, at least) so if you used MD5 you knew that it costs about that much to bypass the control. Moving to SHA1 pushes you massively out of that space, even if that algorithm has flaws. I'm sure that Firecracker has vulnerabilities. Cost of exploitation is likely >100K, likely >1M. gVisor is likely on the same order of magnitude and these two technologies stack because they address the same surface and can be used in conjunction. Software absolutely doesn't have to be perfect, it just has to be costly to attack and it's hilariously easy to drive costs way way way up.
      - bottlepalm 2mo ago
        It just takes one crack in the armor, and malicious AI has the potential to exploit it faster than you have time to react. Literally go to bed and wake up locked out of everything with no hope of recovery.
        insanitybit 2mo ago
        > It just takes one crack in the armor, This is incorrect. It's actually the whole point. Imagine you're an attacker in a gvisor container with a Firecracker hypervisor around you, and a proxy on the host holds a signing secret that gets exposed through the VM virtual device. Getting access to that secret is not one crack. You need to escalate out of gvisor. That likely gets you control over the Sentry process - let's ignore its sandboxing and just say "you're an unprivileged user". Any viable attack on Firecracker requires either KVM / hardware exploits (>$1M but definitely real) or has to start at the kernel. Okay, that's about 10-50k to get a kernel LPE, maybe 5K in tokens these days. So you're in the kernel in the guest of the VM. Time to expoit firecracker lol. It's... never been done. There are like two promising CVEs ever and they're not actually exploitable, no one has done it. Okay, so like, hand waving, let's say it's about $1M to exploit firecracker. Great, you're unprivileged on the guest. We'll just kind of ignore the additional sandboxing that Firecracker does. NOW you can try to attack the proxy by scraping its memory or whatever. This is literally millions of dollars for standard infrastructure hardening and you could go so much further. You can trivially make kernel exploitaton 10x harder, you can make gvisor escapes much much harder, you can move the proxy signing into a TPM (depending on requirements but whatever), you can move the proxy to another computer altogether, you could fuzz these systems for days or run agents against them or whatever. But one thing is certain - it is never "one crack".
- insanitybit 2mo ago
  That's not what I said, nor is it what I meant. It is incredibly easy to write radically safer software than the standard. Moving code into gvisor virtually eliminates privilege escalation. Using memory safe languages without serialization is pretty straightforward. Using type safety to enforce security constraints is straightforward. Setting up network controls to limit SSRF is straightforward. I could go on and on. A tiny bit of forethought and effort pays off massively.
  - bottlepalm 2mo ago
    You don’t understand. You need to write perfect software the first time for it not to be hacked. That has never happened ever.
    - insanitybit 2mo ago
      Are you being sarcastic?
      - bottlepalm 2mo ago
        The fact you posted that and nothing of substance tells me you have nothing, or something very weak. So please tell me of this magical unhackable software/hardware you vague post about.
        insanitybit 2mo ago
        It was a genuine question because I couldn't tell. I responded to your idea that software has to be "perfect" in your other reply so I think we can continue there. https://news.ycombinator.com/item?id=49369111 https://news.ycombinator.com/item?id=49369111
        skydhash 2mo ago
        You're the one that is inventing this "magical unhackable software/hardware", the other person was just saying that there are ways to write "safer" software, not "safe" software. Anything that has happened looks like no security concerns has been looked at or have been thought about.
- overfeed 2mo ago
  > Piece of cake. If everyone could convince management to care about security over "productivity" (read as number of marketable features squeezed out of organizational orifices per unit time), and maybe wire-up open-weight agents to do security critiques, we'd all be in a much better place, but Altman won't like that.

This is an important point. When the post says they're improving... > 3. Security measures, which limit what AI systems can access or affect. What they mean is that proper hard internal security just went from somewhere far below "build a better model" priority to higher, because of a company-wide directive. The HuggingFace incident wouldn't have happened if OpenAI had dedicated sufficient resources to isolation and monitoring. Now, we presume, they are dedicating more. Enough? Who knows. We'll see if the corporate priorities for security stick when a competitor temporarily vaults into the lead.

bottlepalm 2mo ago

Not everything is a conspiracy you know.

kalkin 2mo ago

All we need to retain human control over AIs is for nobody to write any bugs. Piece of cake.

rubendev 2mo ago
No we just need developers to do the bare minimum of effort to write secure software. Most hacks are not super complicated vulnerabilities chained together, but just utter failures where authentication and authorization was simply forgotten or untested, or where nobody bothered to validate the data they receive. The bar for software is so low that it is embarrassing for the entire profession.
- bottlepalm 2mo ago
  People make mistakes, and people don’t know everything either. The software you write is on top of a house of cards of software and hardware. It all has to be perfect to not be hacked. It isn’t perfect, even if you try your hardest it won’t be perfect and to argue it’s not difficult is absurd. You don’t know everything, you don’t own the stack. So how are you going to create a secure anything top to bottom - you can’t.
  - insanitybit 2mo ago
    > It all has to be perfect to not be hacked. This is absolutely not true. It's a matter of cost. Exploitation can cost on the order of 10K, 100K, 1M, 10M, etc. A straightforward one would be something like "MD5 collisions are on the order of $100K-1M" (a while ago, at least) so if you used MD5 you knew that it costs about that much to bypass the control. Moving to SHA1 pushes you massively out of that space, even if that algorithm has flaws. I'm sure that Firecracker has vulnerabilities. Cost of exploitation is likely >100K, likely >1M. gVisor is likely on the same order of magnitude and these two technologies stack because they address the same surface and can be used in conjunction. Software absolutely doesn't have to be perfect, it just has to be costly to attack and it's hilariously easy to drive costs way way way up.
    - bottlepalm 2mo ago
      It just takes one crack in the armor, and malicious AI has the potential to exploit it faster than you have time to react. Literally go to bed and wake up locked out of everything with no hope of recovery.
      - insanitybit 2mo ago
        > It just takes one crack in the armor, This is incorrect. It's actually the whole point. Imagine you're an attacker in a gvisor container with a Firecracker hypervisor around you, and a proxy on the host holds a signing secret that gets exposed through the VM virtual device. Getting access to that secret is not one crack. You need to escalate out of gvisor. That likely gets you control over the Sentry process - let's ignore its sandboxing and just say "you're an unprivileged user". Any viable attack on Firecracker requires either KVM / hardware exploits (>$1M but definitely real) or has to start at the kernel. Okay, that's about 10-50k to get a kernel LPE, maybe 5K in tokens these days. So you're in the kernel in the guest of the VM. Time to expoit firecracker lol. It's... never been done. There are like two promising CVEs ever and they're not actually exploitable, no one has done it. Okay, so like, hand waving, let's say it's about $1M to exploit firecracker. Great, you're unprivileged on the guest. We'll just kind of ignore the additional sandboxing that Firecracker does. NOW you can try to attack the proxy by scraping its memory or whatever. This is literally millions of dollars for standard infrastructure hardening and you could go so much further. You can trivially make kernel exploitaton 10x harder, you can make gvisor escapes much much harder, you can move the proxy signing into a TPM (depending on requirements but whatever), you can move the proxy to another computer altogether, you could fuzz these systems for days or run agents against them or whatever. But one thing is certain - it is never "one crack".
insanitybit 2mo ago
That's not what I said, nor is it what I meant. It is incredibly easy to write radically safer software than the standard. Moving code into gvisor virtually eliminates privilege escalation. Using memory safe languages without serialization is pretty straightforward. Using type safety to enforce security constraints is straightforward. Setting up network controls to limit SSRF is straightforward. I could go on and on. A tiny bit of forethought and effort pays off massively.
- bottlepalm 2mo ago
  You don’t understand. You need to write perfect software the first time for it not to be hacked. That has never happened ever.
  - insanitybit 2mo ago
    Are you being sarcastic?
    - bottlepalm 2mo ago
      The fact you posted that and nothing of substance tells me you have nothing, or something very weak. So please tell me of this magical unhackable software/hardware you vague post about.
      - insanitybit 2mo ago
        It was a genuine question because I couldn't tell. I responded to your idea that software has to be "perfect" in your other reply so I think we can continue there. https://news.ycombinator.com/item?id=49369111 https://news.ycombinator.com/item?id=49369111
      - skydhash 2mo ago
        You're the one that is inventing this "magical unhackable software/hardware", the other person was just saying that there are ways to write "safer" software, not "safe" software. Anything that has happened looks like no security concerns has been looked at or have been thought about.
overfeed 2mo ago
> Piece of cake. If everyone could convince management to care about security over "productivity" (read as number of marketable features squeezed out of organizational orifices per unit time), and maybe wire-up open-weight agents to do security critiques, we'd all be in a much better place, but Altman won't like that.