4 ms·

As a non ML person I have been playing around with torch the past few weeks. I see that people will just share pretrained models on github with random links to

As a non ML person I have been playing around with torch the past few weeks. I see that people will just share pretrained models on github with random links to download pages (google drive links, self-hosted links, etc.) I was quite surprised by this. Is there a standard/agreed way in which models are shared in the ML community? Is there some agreed model integrity check or signature when pulling random files?

jrumbut 4y ago
I'm sure there are several ways but in practice there is a lot of ad hoc.
londons_explore 4y ago
Models are hard to train. So if someone is offering you a large model, you can be fairly sure whoever is offering it has substantial compute resources. Turns out most bad guys don't yet have access to compute on the necessary scale. That in turn means you can be fairly sure most big models you find online are in fact made by a trustworthy party, even if you download them from a random WeTransfer link...
- mtlmtlmtlmtl 4y ago
  If it's in pickle format, containing arbitrary code, what's stopping a bad actor from simply generating a random untrained model, with a malicious payload attached? If you want to embed some sort of sneaky backdoor into a model, sure I buy this logic, but most malicious actors just want to take over your machine or something. No need to actually train a model to do that.
  - aflag 4y ago
    That said, it's probably a much better investment to do supply chain sort of attacks than trying to trick people into downloading your pickled model. Although, I would be surprised if there aren't pickled models with some malicious code out there. It doesn't feel like it's a very sought after target.
    - londons_explore 4y ago
      There are recent attackers who want to steal ssh keys of developers to do things like inject malicious code into any git repos that developer owns. That malicious code in turn, when pulled and installed by another developer does the same - so it's a worm that spreads via npm, makefiles, requirements.txt, etc. No reason it couldn't also spread by pickle files.
- aflag 4y ago
  I think plenty bad guys have the necessary resources, but models tend to be just a large array of numbers. I think the main reason it's unlikely is just that there's not that much value in messing with your model. What are they realistically going to get out of it?
- kwertyoowiyop 4y ago
  Can all those crypto-hashing computers now be used for ML training?
jeroenhd 4y ago
The most fun are the ML models shared in pickle format. They can contain executable code and who knows if that Stable Diffusion model you just downloaded will make your image generation dreams come true or is just full of viruses! There are ways to verify the safety of these models but I doubt most users will go through the effort.
- DoingIsLearning 4y ago
  > There are ways to verify the safety of these models but I doubt most users will go through the effort. Could you expand on this? I assume it's some sort of serialization format, other than parsing it what can you do to inspect?
  - jeroenhd 4y ago
    It's Python's serialisation format: https://docs.python.org/3/library/pickle.html https://docs.python.org/3/library/pickle.html There are tools to check the format for suspicious behaviour: https://github.com/mmaitre314/picklescan https://github.com/mmaitre314/picklescan seems to be the most developed one. You can also check the format manually (being careful not to call into it), like demonstrated by this more rudimentary scanner: https://github.com/zxix/stable-diffusion-pickle-scanner https://github.com/zxix/stable-diffusion-pickle-scanner It you do check for security issues yourself, you'll need to read up on what magical methods/variables may cause code execution. Simple demonstrations of dangerous code can be found all over the web (https://stackoverflow.com/questions/47705202/pickle-exploiting https://stackoverflow.com/questions/47705202/pickle-exploiti...) but I'm sure there are obfuscation tricks that simple scans won't catch.

jrumbut 4y ago

I'm sure there are several ways but in practice there is a lot of ad hoc.

londons_explore 4y ago

Models are hard to train. So if someone is offering you a large model, you can be fairly sure whoever is offering it has substantial compute resources. Turns out most bad guys don't yet have access to compute on the necessary scale. That in turn means you can be fairly sure most big models you find online are in fact made by a trustworthy party, even if you download them from a random WeTransfer link...

mtlmtlmtlmtl 4y ago
If it's in pickle format, containing arbitrary code, what's stopping a bad actor from simply generating a random untrained model, with a malicious payload attached? If you want to embed some sort of sneaky backdoor into a model, sure I buy this logic, but most malicious actors just want to take over your machine or something. No need to actually train a model to do that.
- aflag 4y ago
  That said, it's probably a much better investment to do supply chain sort of attacks than trying to trick people into downloading your pickled model. Although, I would be surprised if there aren't pickled models with some malicious code out there. It doesn't feel like it's a very sought after target.
  - londons_explore 4y ago
    There are recent attackers who want to steal ssh keys of developers to do things like inject malicious code into any git repos that developer owns. That malicious code in turn, when pulled and installed by another developer does the same - so it's a worm that spreads via npm, makefiles, requirements.txt, etc. No reason it couldn't also spread by pickle files.
aflag 4y ago
I think plenty bad guys have the necessary resources, but models tend to be just a large array of numbers. I think the main reason it's unlikely is just that there's not that much value in messing with your model. What are they realistically going to get out of it?
kwertyoowiyop 4y ago
Can all those crypto-hashing computers now be used for ML training?

jeroenhd 4y ago

The most fun are the ML models shared in pickle format. They can contain executable code and who knows if that Stable Diffusion model you just downloaded will make your image generation dreams come true or is just full of viruses! There are ways to verify the safety of these models but I doubt most users will go through the effort.

DoingIsLearning 4y ago
> There are ways to verify the safety of these models but I doubt most users will go through the effort. Could you expand on this? I assume it's some sort of serialization format, other than parsing it what can you do to inspect?
- jeroenhd 4y ago
  It's Python's serialisation format: https://docs.python.org/3/library/pickle.html https://docs.python.org/3/library/pickle.html There are tools to check the format for suspicious behaviour: https://github.com/mmaitre314/picklescan https://github.com/mmaitre314/picklescan seems to be the most developed one. You can also check the format manually (being careful not to call into it), like demonstrated by this more rudimentary scanner: https://github.com/zxix/stable-diffusion-pickle-scanner https://github.com/zxix/stable-diffusion-pickle-scanner It you do check for security issues yourself, you'll need to read up on what magical methods/variables may cause code execution. Simple demonstrations of dangerous code can be found all over the web (https://stackoverflow.com/questions/47705202/pickle-exploiting https://stackoverflow.com/questions/47705202/pickle-exploiti...) but I'm sure there are obfuscation tricks that simple scans won't catch.