3 ms·

What file format(s) are giant LLM models distributed in? I’m surprised they don’t get leaked by employees.

hnav 4mo ago
These are terabyte sized files (realistically a multi hour transfer) that you're unlikely to have access to in the first place. Every organization has exfiltration checks these days. You may succeed but you'll want to be on a plane to a non-extradition country no more than hours after you kick off the transfer.
qsxfthnkp2322 4mo ago
What’s the point? Anthropic and other frontier vendors already provide their models on other services like vertex, bedrock, or openrouter It’s not like anyone can home lab one of these models without quite a bit of hardware
- mips_avatar 4mo ago
  Yeah we can probably figure out how to run it on xiaomi gpus
05 4mo ago
I assume they’re encrypted/DRM’ed when deployed on inference hardware, so only core researchers/sec admins would potentially have some access to unprotected weights, and they are far too well paid to risk it leaking the model
- jltsiren 4mo ago
  Incentives matter on the average, but people are too unpredictable for categorical statements like that. They can always have other reasons beyond personal gain to leak secrets. There was no shortage of spies and defectors leaking American nuclear secrets to the USSR during the Cold War.
- Retr0id 4mo ago
  I wouldn't be surprised if they encrypt them at rest, but at some point the weights have to be loaded into vram.
  - 05 4mo ago
    Newer NVidia cards (H100 and up) support both in-memory model encryption and ‘trusted’ execution environment/remote attestation, not sure how widely used in frontier model deployments, but at least vendor claimed perf overhead is ‘3%’ [0] [0] https://www.spheron.network/blog/confidential-gpu-computing-nvidia-tee-encrypted-vram/#:~:text=Performance%20Overhead:%20CC%20Mode%20vs%20Standard https://www.spheron.network/blog/confidential-gpu-computing-...
borissk 4mo ago
The employees are hoping to become very very rich after the IPO and after they are allowed to sell the shares given to them - risking a likely multi-million dollar pay back to leak a model that will be superseded by publicly available models in a couple of years is not a likely decision.

hnav 4mo ago

These are terabyte sized files (realistically a multi hour transfer) that you're unlikely to have access to in the first place. Every organization has exfiltration checks these days. You may succeed but you'll want to be on a plane to a non-extradition country no more than hours after you kick off the transfer.

qsxfthnkp2322 4mo ago

What’s the point? Anthropic and other frontier vendors already provide their models on other services like vertex, bedrock, or openrouter It’s not like anyone can home lab one of these models without quite a bit of hardware

mips_avatar 4mo ago
Yeah we can probably figure out how to run it on xiaomi gpus

05 4mo ago

I assume they’re encrypted/DRM’ed when deployed on inference hardware, so only core researchers/sec admins would potentially have some access to unprotected weights, and they are far too well paid to risk it leaking the model

jltsiren 4mo ago
Incentives matter on the average, but people are too unpredictable for categorical statements like that. They can always have other reasons beyond personal gain to leak secrets. There was no shortage of spies and defectors leaking American nuclear secrets to the USSR during the Cold War.
Retr0id 4mo ago
I wouldn't be surprised if they encrypt them at rest, but at some point the weights have to be loaded into vram.
- 05 4mo ago
  Newer NVidia cards (H100 and up) support both in-memory model encryption and ‘trusted’ execution environment/remote attestation, not sure how widely used in frontier model deployments, but at least vendor claimed perf overhead is ‘3%’ [0] [0] https://www.spheron.network/blog/confidential-gpu-computing-nvidia-tee-encrypted-vram/#:~:text=Performance%20Overhead:%20CC%20Mode%20vs%20Standard https://www.spheron.network/blog/confidential-gpu-computing-...

borissk 4mo ago

The employees are hoping to become very very rich after the IPO and after they are allowed to sell the shares given to them - risking a likely multi-million dollar pay back to leak a model that will be superseded by publicly available models in a couple of years is not a likely decision.