3 ms·
This is only true for models that are python pickles. gguf doesn't suffer from this. The ml community needs to move away from pickle files as soon as possible.
by fzzzy 3y ago
This is only true for models that are python pickles. gguf doesn't suffer from this.
The ml community needs to move away from pickle files as soon as possible. Worst idea ever.
- water-data-dude 3y agoWHAT? Python warns you in the most apocalyptic language possible that pickles are dangerous as hell. Are the pickle files buried in the HF models in a non-obvious way? I can’t imagine people would just run random pickles from the internet (but then again, people do run `curl [URL] | bash` all the time…)
- fzzzy 3y agoI'm not responsible for whatever led the ml community to the widespread use of pickles for weights. It happened regardless of the warnings that pickles are dangerous. Yes, ml researchers have been running random pickles from the internet for years now. No they are not buried in a non obvious way. They are at the very top of many models. safetensors and gguf have started to replace this horrible practice, but there are still tons of models that use pickles.