3 ms·

There's a big difference, and there's absolutely a safe way to run models locally. Pytorch files use pickle serialization[0] which is insecure and can allow you

by Me1000 3y ago

There's a big difference, and there's absolutely a safe way to run models locally. Pytorch files use pickle serialization[0] which is insecure and can allow you to embed arbitrary code. Regular users should not be using that, they should instead be using safetensors or something like gguf which is (more or less) just a set of weights. A gguf model file can be thought of (at a very high level) as a jpg. We have safe and secure ways of decoding and using a jpg. The "black box" you refer to is more about not understanding why a model does what it does. We don't know why a particular node in the network has a value that ends up influencing the output. We understand how a deep neural net works. [0] https://docs.python.org/3/library/pickle.html https://docs.python.org/3/library/pickle.html

rvz 3y ago
> Regular users should not be using that, they should instead be using safetensors or something like gguf which is (more or less) just a set of weights. Regular users do not know any of this and do not care. All they know is to download .exe and run and never check whatever they are downloading is malicious or not. > The "black box" you refer to is more about not understanding why a model does what it does. That is my additional point which makes this situation absolutely even worse. > We understand how a deep neural net works. No one does and certainly not even the AI scientists even understand the unpredictable behaviours of these models after training.
- Me1000 3y ago
  I'm sorry I think I didn't explain my point clearly. I'm trying to point out there's a difference between not understanding why a model outputs a specific token, and not understand what the computer is doing under the hood. We understand very well how fed forward networks work computationally, and there's nothing really insecure about that. The insecure problem here is in the serialization format that some of these models are distributed as. As for my "regular users" comment, I don't think it's hard to imagine a world where users have a trusted program that runs the models. This is how basically all file formats work. Excel was insecure at one point for similar reasons, you could embed malicious code in macros, but today excel spreadsheets can run computations on files downloaded from the web and it's just as secure as open a .txt file.
  - rvz 3y ago
    > I'm trying to point out there's a difference between not understanding why a model outputs a specific token, and not understand what the computer is doing under the hood. Whenever there is trust involved, there is no difference to your point. How you're running the model requires trusting that the model, parser, etc isn't compromised and especially if it can be trusted to behave correctly after training - thus transparent explanations rather than hallucinations and its easy to trick and compromise them to do something else. Given that we already don't trust the outputs of these AI models, the above security issue make this even worse and untrustworthy. The plain old regular average joe users do not care about the neural network format, etc and will run and open anything without checking regardless even if it is a text file disguised as a program. Thus, it is entirely no different and we are back to square -1 (with the unexplainable properties of these models that people will try it out and trust its outputs)
  - saltcured 3y ago
    I agree that we can define a safe serialization format for models with given assumptions about architecture. I.e. when the model is just the matrices and cannot supply custom inference code needed to process the matrices. But, I expect we're going to have additional rounds of insecure practice just like we've had in every other popularized tech movement. People are going to develop frameworks with code-injection flaws, where they assume mode outputs (tokens) can be trusted and can contain executable content. Then, the inscrutable and untrustworthy models are going to be a problem as well. Think everything from MS Office macro abuse to human drivers blindly following their GPS guidance into a lake. This can and most probabaly will be repeated with AI models, due to the prevalence of naive and over-trusting practitioners and consumers.
    - Me1000 3y ago
      100%. Software takes time to make it secure, since after all it’s written by us flawed humans. The runtime that consumes the model files might have bugs, but those will be fixed over time. How people use model outputs (or inputs; I.e prompt injections) is a whole other area ripe for exploits, especially while these technologies are being adopted by people who don’t really understand them. But I view this as fundamentally different than the above. A model format can be secure in that it won’t just randomly delete files on your computer. This is a computer science problem and thus provably securable. I guess that was the point I was trying to make when differentiating why they work vs how they work.

by Me1000 3y ago

rvz 3y ago

> Regular users should not be using that, they should instead be using safetensors or something like gguf which is (more or less) just a set of weights. Regular users do not know any of this and do not care. All they know is to download .exe and run and never check whatever they are downloading is malicious or not. > The "black box" you refer to is more about not understanding why a model does what it does. That is my additional point which makes this situation absolutely even worse. > We understand how a deep neural net works. No one does and certainly not even the AI scientists even understand the unpredictable behaviours of these models after training.

Me1000 3y ago
I'm sorry I think I didn't explain my point clearly. I'm trying to point out there's a difference between not understanding why a model outputs a specific token, and not understand what the computer is doing under the hood. We understand very well how fed forward networks work computationally, and there's nothing really insecure about that. The insecure problem here is in the serialization format that some of these models are distributed as. As for my "regular users" comment, I don't think it's hard to imagine a world where users have a trusted program that runs the models. This is how basically all file formats work. Excel was insecure at one point for similar reasons, you could embed malicious code in macros, but today excel spreadsheets can run computations on files downloaded from the web and it's just as secure as open a .txt file.
- rvz 3y ago
  > I'm trying to point out there's a difference between not understanding why a model outputs a specific token, and not understand what the computer is doing under the hood. Whenever there is trust involved, there is no difference to your point. How you're running the model requires trusting that the model, parser, etc isn't compromised and especially if it can be trusted to behave correctly after training - thus transparent explanations rather than hallucinations and its easy to trick and compromise them to do something else. Given that we already don't trust the outputs of these AI models, the above security issue make this even worse and untrustworthy. The plain old regular average joe users do not care about the neural network format, etc and will run and open anything without checking regardless even if it is a text file disguised as a program. Thus, it is entirely no different and we are back to square -1 (with the unexplainable properties of these models that people will try it out and trust its outputs)
- saltcured 3y ago
  I agree that we can define a safe serialization format for models with given assumptions about architecture. I.e. when the model is just the matrices and cannot supply custom inference code needed to process the matrices. But, I expect we're going to have additional rounds of insecure practice just like we've had in every other popularized tech movement. People are going to develop frameworks with code-injection flaws, where they assume mode outputs (tokens) can be trusted and can contain executable content. Then, the inscrutable and untrustworthy models are going to be a problem as well. Think everything from MS Office macro abuse to human drivers blindly following their GPS guidance into a lake. This can and most probabaly will be repeated with AI models, due to the prevalence of naive and over-trusting practitioners and consumers.
  - Me1000 3y ago
    100%. Software takes time to make it secure, since after all it’s written by us flawed humans. The runtime that consumes the model files might have bugs, but those will be fixed over time. How people use model outputs (or inputs; I.e prompt injections) is a whole other area ripe for exploits, especially while these technologies are being adopted by people who don’t really understand them. But I view this as fundamentally different than the above. A model format can be secure in that it won’t just randomly delete files on your computer. This is a computer science problem and thus provably securable. I guess that was the point I was trying to make when differentiating why they work vs how they work.