3 ms·
> Most developers don't realize that standard PyTorch files are just Zip archives containing Python Pickle bytecode. This is outrageous. Why not deprecate this
by nextaccountic 11mo ago
> Most developers don't realize that standard PyTorch files are just Zip archives containing Python Pickle bytecode.
This is outrageous. Why not deprecate this cursed format and use something from the data frame community? Like, Parquet or something
Actually almost any binary format is better than this
- tennysont 11mo agoPickle files are probably still useful saving exploratory work, collaborating inside a company, and use inside a pipeline. Safetensors is supposed to be the successor for distribution. I believe that it's the "safe" subset of pickle's data format.
- rhdunn 11mo agoThe safetensors file format is a header length, JSON header, and serialized tensor weights. [1] [1] https://github.com/huggingface/safetensors https://github.com/huggingface/safetensors