2 ms·
Disclosure I work for the company that released this: https://github.com/protectai/modelscan https://github.com/protectai/modelscan but we do have a tool to sup
by wolftickets 3y ago
Disclosure I work for the company that released this: https://github.com/protectai/modelscan https://github.com/protectai/modelscan but we do have a tool to support scanning many models for this kind of problem.
That said you should be using something like safe-tensors.
- lawlessone 3y agoYou have me curious now. The models generate text. Could a model hypothetically be trained in such a way that could create a buffer overflow when given certain prompts? I am guessing the way inference works in such a way that cant happen
- wolftickets 3y agoAbsolutely, though that isn't strictly what we're talking about here. In this case, models themselves are fundamentally files. These files can have malicious code embedded into them that is executed when the model is loaded for further training or inference. When executed it isn't obvious to the user at all. It's a very nasty potential vector. I wrote a blog about it here: https://protectai.com/blog/announcing-modelscan https://protectai.com/blog/announcing-modelscan