3 ms·

Have OpenAI or Anthropic ever had a model hacked/leaked? Is there any good reads on their cultures of preventing it from happening?

varun_ch 3mo ago
surely the weights for the model & the equipment to run them make it logistically challenging enough to deter that… also I’m sure models have leaked in their APIs before but those would be pretty easy and quick to catch/fix.
sarjann 3mo ago
I believe Nvidia chips have a secure way to run your model on other infra. https://www.nvidia.com/en-us/data-center/solutions/confidential-computing/ https://www.nvidia.com/en-us/data-center/solutions/confident...
- erwald 3mo ago
  Confidential computing is not secure against a potential attacker who has physical access to the hardware. The CC security guarantees explicitly assume the attacker has no physical access.
  - traceroute66 3mo ago
    > is not secure against a potential attacker who has physical access to the hardware. Well, yes, its the oldest adage in computing that "physical access == game over". So I would argue it is more about reducing your risk to a more acceptable level. And in that respect I would say using services such as Tinfoil or Privatemode is an enormous step up from "trust me dude, we won't look at your data". Remotely verifiable attestation combined with independent audits of the company hosting is a large step up from a Zero Data Retention clause in your contract that you have no way of verifying is actually happening other than "trust me dude". Clearly I absolutely agree, having it on your own infrastructure is best for confidentiality. But even then, what about evil-maid attacks in the datacentre ? Unless you have your own datacentre, you're going to be in a shared colo facility ...
    - erwald 3mo ago
      Yeah, to be clear I'm pretty excited about confidential computing and startups building on it, like Tinfoil, for some use cases. I just wanted to point out it's far from adequate for some important threat models (e.g., securing model weights for data centers located abroad, I think). (It's also not super widely adopted in AI yet, but that seems to be changing, at least for inference workloads.)
- traceroute66 3mo ago
  > I believe Nvidia chips have a secure way to run your model on other infra. Yes. And its already on offer today. See Tinfoil(US)[1] and Privatemode(Germany)[2] Tinfoil have not been independently audited, it is somewhere on their long-term radar. Privatemode have been thoroughly independently audited with documentation available on request. [1]https://tinfoil.sh/ https://tinfoil.sh/ [2] https://www.privatemode.ai/ https://www.privatemode.ai/
grun 3mo ago
- afaik no openai or anth model weight have been leaked to date - confidential computing and trusted execution environments (TEEs) are the strongest primitive yet invented to prevent model weight exfiltration. there are other techniques. like bandwidth limiting of GPU workers and key sealing. but none as strong as those afforded by confidential computing and TEEs - see https://confidential.ai/docs/confidential-computing-primer https://confidential.ai/docs/confidential-computing-primer for a primer on confidential computing - using confidential computing and TEEs to protect model weights in use is an area of active research note: i work on confidential.ai

varun_ch 3mo ago

surely the weights for the model & the equipment to run them make it logistically challenging enough to deter that… also I’m sure models have leaked in their APIs before but those would be pretty easy and quick to catch/fix.

sarjann 3mo ago

I believe Nvidia chips have a secure way to run your model on other infra. https://www.nvidia.com/en-us/data-center/solutions/confidential-computing/ https://www.nvidia.com/en-us/data-center/solutions/confident...

erwald 3mo ago
Confidential computing is not secure against a potential attacker who has physical access to the hardware. The CC security guarantees explicitly assume the attacker has no physical access.
- traceroute66 3mo ago
  > is not secure against a potential attacker who has physical access to the hardware. Well, yes, its the oldest adage in computing that "physical access == game over". So I would argue it is more about reducing your risk to a more acceptable level. And in that respect I would say using services such as Tinfoil or Privatemode is an enormous step up from "trust me dude, we won't look at your data". Remotely verifiable attestation combined with independent audits of the company hosting is a large step up from a Zero Data Retention clause in your contract that you have no way of verifying is actually happening other than "trust me dude". Clearly I absolutely agree, having it on your own infrastructure is best for confidentiality. But even then, what about evil-maid attacks in the datacentre ? Unless you have your own datacentre, you're going to be in a shared colo facility ...
  - erwald 3mo ago
    Yeah, to be clear I'm pretty excited about confidential computing and startups building on it, like Tinfoil, for some use cases. I just wanted to point out it's far from adequate for some important threat models (e.g., securing model weights for data centers located abroad, I think). (It's also not super widely adopted in AI yet, but that seems to be changing, at least for inference workloads.)
traceroute66 3mo ago
> I believe Nvidia chips have a secure way to run your model on other infra. Yes. And its already on offer today. See Tinfoil(US)[1] and Privatemode(Germany)[2] Tinfoil have not been independently audited, it is somewhere on their long-term radar. Privatemode have been thoroughly independently audited with documentation available on request. [1]https://tinfoil.sh/ https://tinfoil.sh/ [2] https://www.privatemode.ai/ https://www.privatemode.ai/

grun 3mo ago

- afaik no openai or anth model weight have been leaked to date - confidential computing and trusted execution environments (TEEs) are the strongest primitive yet invented to prevent model weight exfiltration. there are other techniques. like bandwidth limiting of GPU workers and key sealing. but none as strong as those afforded by confidential computing and TEEs - see https://confidential.ai/docs/confidential-computing-primer https://confidential.ai/docs/confidential-computing-primer for a primer on confidential computing - using confidential computing and TEEs to protect model weights in use is an area of active research note: i work on confidential.ai