4 ms·

Just two days ago I flipped through a slide deck from a security conference where the author, Jossef Harush Kadouri, found that using a model from a place like

by Aachen 2y ago

Just two days ago I flipped through a slide deck from a security conference where the author, Jossef Harush Kadouri, found that using a model from a place like Huggingface means the author of the model can execute any code on your machine. Not sure if the slides are uploaded elsewhere, I got them sent as file: https://dro.pm/c.pdf https://dro.pm/c.pdf (45MB) slide 188 I didn't realise at the time that I flipped through the slides that this means not only the model's author gets to run code on your machine, but also if Huggingface got a court-signed letter or if someone hacked them (especially if they don't notice for a while¹) As someone not in the AI scene, I've never run these models but was surprised at how quickly the industry standardised the format. I had assumed model files were big matrices of numbers and some metadata perhaps, but now I understand how they managed so quickly: a model is (eyeing slides 186 and 195) a Python script that can do whatever it wants. That makes "standardisation" exceedingly easy: everyone can do their own thing and you sidestep the problem altogether. But that comes with a cost. ¹ https://www.verizon.com/business/resources/articles/s/how-to-minimize-your-mean-time-to-detect-a-breach/ https://www.verizon.com/business/resources/articles/s/how-to... says 20% doesn't notice for months; of course, it depends on the situation and what actions the attackers take

bongodongobob 2y ago
Are you telling me that when I run software on my computer I could potentially be running software on my computer?
- foobiekr 2y ago
  Nobody expects a model file to be code thrat executes whatever.
  - foolishbard 2y ago
    I'd say anybody who is working in the field has this expectation. But the outside observer who is excited to try a new model does not expect it.
  - squigz 2y ago
    This is like expecting any random binary to only do what it 'claims' to do.
    - erikerikson 2y ago
      No, this is like PDFs that can contain and will execute PostScript but without a sandbox and with more general libraries
  - LunaSea 2y ago
    That is borderline incompetence
- hn92726819 2y ago
  This is more akin to downloading a jpeg and the jpeg running arbitrary code. Models should be like jpegs and I believe safetensors treat them that way, while the old pickle format didn't.
  - catgary 2y ago
    The weights are like jpegs, the model itself is still a piece of software that executes code.
  - amelius 2y ago
    You typically need more than the model to actually use it, as you might need to "massage" the data into the right form before and after the model call.
  - bravetraveler 2y ago
    I think the gap between 'should' and [short-lived] reality is the point - that's obviously the reasonable setup, but pickle happened. In the quest of being first or whatever, amateur rediscoveries are happening. Consistently. I'm making a disgusting amount of money in the field by doing day-one packaging work any commodity project would also be doing. They just wouldn't decide to redefine things in a clearly worse way. I don't actually know jack about AI, yet here we are. The hype/gold rush is real; competence in the shared fundamentals can go a long way. The trick is knowing/applying the basics for the wizards. There's a steep premium so they can play around, apparently. Case in point: most of these humongous container images and how they're naively moved around Now that more people know how to publish (containers/packages/articles), we're seeing more drivel published. It's ridiculous yet inevitable. I just wonder how much value the rush provides. IIRC 'safetensors' aren't necessarily even that safe. I could have sworn I read they're fairly nebulous. Grain of salt etc
    - fragmede 2y ago
      no such exploits have been publicised against known safetensors, so I'd need more than an unqualified Internet rumor to consider them unsafe. Not saying it's not possible but the entire point of them is that they're a blob of numbers, and not a known exploitable bit of python. How're you making money? is it a contracting via network thing or can anybody apply?
- pheatherlite 2y ago
  Data intended to be read as instructions for the interpreter or the cpu, is a whole different ballgame than data intended to convey values of something. High order sparse/dense matrices serialized in some xyz format is what most people think of when they hear the word "model". To switch it up and send some arbitrary python file and execute it on the client is a security nightmare. This outrageous.
strangecasts 2y ago
> I had assumed model files were big matrices of numbers and some metadata perhaps ONNX [1] is more or less this, but the challenge you immediately run into is models with custom layers/operators with their own inference logic - you either have to implement those operators in terms of the supported ops (not necessarily practical or viable) or provide the implementation of the operator to the runtime, putting you back at square one. [1] https://onnx.ai/ https://onnx.ai/
addandsubtract 2y ago
Isn't that why we have the .safetensors format, which can't execute code on the host?
- dartos 2y ago
  This is my understanding as well
  - SV_BubbleTime 2y ago
    I was at Info sec local meet up six months ago, and mentioned the Tensor models get control of your CPU and GPU… I was basically passed over like I was some idiot making LLM == TERMINATOR claims. Good stuff.
    - mistrial9 2y ago
      > Tensor models get control of your CPU and GPU "get control" is not accurate.. there is a code execution model on the CPU based on an operating system; the GPU is a different story. Executed code with an operating system has execution context and protections.. this is basic to operating system theory, all the common operating systems implement related concepts.
      - SV_BubbleTime 2y ago
        Cool… so… get control of a CPU means being able to execute code. It can’t necessarily defeat ring protections or secure zone depending on the context. But then again, I wouldn’t put it past a good exploit to do so.
        justinclift 2y ago
        If someone can inject bad code for execution into the GPU that's also pretty bad though yeah? The GPU should be able to pull and push stuff into memory, and maybe even access other devices directly over the PCIe bus?
koolala 2y ago
Hugging face standing right behind you, ready for hugs
d-z-m 2y ago
> using a model from a place like Huggingface means the author of the model can execute any code on your machine To my knowledge this is only a problem if the model is serialized/de-serialized via pickle[0]. [0]: https://huggingface.co/docs/hub/en/security-pickle https://huggingface.co/docs/hub/en/security-pickle
- hehdhdjehehegwv 2y ago
  The fact that pickle even exists is fundamentally wrong to start with. They should not be permitted as a distribution format, period.
  - 1oooqooq 2y ago
    it's pretty neat and functional if you are the one writing the code. you can do things like "hibernate" with zero developer cost.
    - wildzzz 2y ago
      That's what it's used for in a major application I use on a daily basis. The entire config for the application is saved off in a pickle binary file and then loaded at whim. Makes it super easy to backup and transfer configs to compatible systems rather than saving everything to a database. One time we were trying to get the system to do something it was not designed for but was probably capable of. Eventually someone got it working on one of the two identical systems we have through a lot of trial and error. Rather than two people sitting on the phone all day comparing hundreds of settings, we just grabbed the pickle file and emailed it over and had it instantly working.
worstspotgain 2y ago
As others have pointer out, this is format-dependent. One of the formats that hasn't been white-listed in this thread yet is GGUF, used by llama.cpp and derivates. It's pretty much "big matrices of numbers and some metadata." Some vulnerabilities were found [1] and patched. [1] https://www.databricks.com/blog/ggml-gguf-file-format-vulnerabilities https://www.databricks.com/blog/ggml-gguf-file-format-vulner...
mvandermeulen 2y ago
> Just two days ago I flipped through a slide deck from a security conference where the author, Jossef Harush Kadouri, found that using a model from a place like Huggingface means the author of the model can execute any code on your machine. Proceeds to link to pdf of unknown origins
- Aachen 2y ago
  That's precisely why it's unexpected that a data model can run code. Wouldn't expect a pdf to start executing code on my system either, it should be data!
Aachen 2y ago
(The dro.pm link will expire any minute now. It's so short because it's temporary, should maybe have used a more permanent service. I've found the talk here in case you're reading this later: https://m.youtube.com/watch?v=8XysLIq-e3s https://m.youtube.com/watch?v=8XysLIq-e3s)

by Aachen 2y ago

bongodongobob 2y ago

Are you telling me that when I run software on my computer I could potentially be running software on my computer?

foobiekr 2y ago
Nobody expects a model file to be code thrat executes whatever.
- foolishbard 2y ago
  I'd say anybody who is working in the field has this expectation. But the outside observer who is excited to try a new model does not expect it.
- squigz 2y ago
  This is like expecting any random binary to only do what it 'claims' to do.
  - erikerikson 2y ago
    No, this is like PDFs that can contain and will execute PostScript but without a sandbox and with more general libraries
- LunaSea 2y ago
  That is borderline incompetence
hn92726819 2y ago
This is more akin to downloading a jpeg and the jpeg running arbitrary code. Models should be like jpegs and I believe safetensors treat them that way, while the old pickle format didn't.
- catgary 2y ago
  The weights are like jpegs, the model itself is still a piece of software that executes code.
- amelius 2y ago
  You typically need more than the model to actually use it, as you might need to "massage" the data into the right form before and after the model call.
- bravetraveler 2y ago
  I think the gap between 'should' and [short-lived] reality is the point - that's obviously the reasonable setup, but pickle happened. In the quest of being first or whatever, amateur rediscoveries are happening. Consistently. I'm making a disgusting amount of money in the field by doing day-one packaging work any commodity project would also be doing. They just wouldn't decide to redefine things in a clearly worse way. I don't actually know jack about AI, yet here we are. The hype/gold rush is real; competence in the shared fundamentals can go a long way. The trick is knowing/applying the basics for the wizards. There's a steep premium so they can play around, apparently. Case in point: most of these humongous container images and how they're naively moved around Now that more people know how to publish (containers/packages/articles), we're seeing more drivel published. It's ridiculous yet inevitable. I just wonder how much value the rush provides. IIRC 'safetensors' aren't necessarily even that safe. I could have sworn I read they're fairly nebulous. Grain of salt etc
  - fragmede 2y ago
    no such exploits have been publicised against known safetensors, so I'd need more than an unqualified Internet rumor to consider them unsafe. Not saying it's not possible but the entire point of them is that they're a blob of numbers, and not a known exploitable bit of python. How're you making money? is it a contracting via network thing or can anybody apply?
pheatherlite 2y ago
Data intended to be read as instructions for the interpreter or the cpu, is a whole different ballgame than data intended to convey values of something. High order sparse/dense matrices serialized in some xyz format is what most people think of when they hear the word "model". To switch it up and send some arbitrary python file and execute it on the client is a security nightmare. This outrageous.

strangecasts 2y ago

> I had assumed model files were big matrices of numbers and some metadata perhaps ONNX [1] is more or less this, but the challenge you immediately run into is models with custom layers/operators with their own inference logic - you either have to implement those operators in terms of the supported ops (not necessarily practical or viable) or provide the implementation of the operator to the runtime, putting you back at square one. [1] https://onnx.ai/ https://onnx.ai/

addandsubtract 2y ago

Isn't that why we have the .safetensors format, which can't execute code on the host?

dartos 2y ago
This is my understanding as well
- SV_BubbleTime 2y ago
  I was at Info sec local meet up six months ago, and mentioned the Tensor models get control of your CPU and GPU… I was basically passed over like I was some idiot making LLM == TERMINATOR claims. Good stuff.
  - mistrial9 2y ago
    > Tensor models get control of your CPU and GPU "get control" is not accurate.. there is a code execution model on the CPU based on an operating system; the GPU is a different story. Executed code with an operating system has execution context and protections.. this is basic to operating system theory, all the common operating systems implement related concepts.
    - SV_BubbleTime 2y ago
      Cool… so… get control of a CPU means being able to execute code. It can’t necessarily defeat ring protections or secure zone depending on the context. But then again, I wouldn’t put it past a good exploit to do so.
      - justinclift 2y ago
        If someone can inject bad code for execution into the GPU that's also pretty bad though yeah? The GPU should be able to pull and push stuff into memory, and maybe even access other devices directly over the PCIe bus?

koolala 2y ago

Hugging face standing right behind you, ready for hugs

d-z-m 2y ago

> using a model from a place like Huggingface means the author of the model can execute any code on your machine To my knowledge this is only a problem if the model is serialized/de-serialized via pickle[0]. [0]: https://huggingface.co/docs/hub/en/security-pickle https://huggingface.co/docs/hub/en/security-pickle

hehdhdjehehegwv 2y ago
The fact that pickle even exists is fundamentally wrong to start with. They should not be permitted as a distribution format, period.
- 1oooqooq 2y ago
  it's pretty neat and functional if you are the one writing the code. you can do things like "hibernate" with zero developer cost.
  - wildzzz 2y ago
    That's what it's used for in a major application I use on a daily basis. The entire config for the application is saved off in a pickle binary file and then loaded at whim. Makes it super easy to backup and transfer configs to compatible systems rather than saving everything to a database. One time we were trying to get the system to do something it was not designed for but was probably capable of. Eventually someone got it working on one of the two identical systems we have through a lot of trial and error. Rather than two people sitting on the phone all day comparing hundreds of settings, we just grabbed the pickle file and emailed it over and had it instantly working.

worstspotgain 2y ago

As others have pointer out, this is format-dependent. One of the formats that hasn't been white-listed in this thread yet is GGUF, used by llama.cpp and derivates. It's pretty much "big matrices of numbers and some metadata." Some vulnerabilities were found [1] and patched. [1] https://www.databricks.com/blog/ggml-gguf-file-format-vulnerabilities https://www.databricks.com/blog/ggml-gguf-file-format-vulner...

mvandermeulen 2y ago

> Just two days ago I flipped through a slide deck from a security conference where the author, Jossef Harush Kadouri, found that using a model from a place like Huggingface means the author of the model can execute any code on your machine. Proceeds to link to pdf of unknown origins

Aachen 2y ago
That's precisely why it's unexpected that a data model can run code. Wouldn't expect a pdf to start executing code on my system either, it should be data!

Aachen 2y ago

(The dro.pm link will expire any minute now. It's so short because it's temporary, should maybe have used a more permanent service. I've found the talk here in case you're reading this later: https://m.youtube.com/watch?v=8XysLIq-e3s https://m.youtube.com/watch?v=8XysLIq-e3s)