3 ms·

Yeah, I'm also surprised by this. The standard library itself explicitly warns you in a big red box that pickle is not secure. It also doesn't support NumPy, th

Yeah, I'm also surprised by this. The standard library itself explicitly warns you in a big red box that pickle is not secure. It also doesn't support NumPy, though NumPy has its own native persistence modules. I'd have expected people to be using something like hdf5 for this.

hprotagonist 6y ago
the soopar geniuses at facebook clearly know better! /s (in truth, because pytorch serialised models can include python code for jit scripts, it's nonobvious what a good way to store python code is -- but torch has recently moved to a zipfile impl as of 1.6: https://pytorch.org/docs/stable/generated/torch.save.html https://pytorch.org/docs/stable/generated/torch.save.html) >I'd have expected people to be using something like hdf5 for this. Amusingly, matlab was way ahead of the pack here; matfiles have been hdf5 since r2006b, back when we just called it "matlab 7.3".
- a-dub 6y ago
  yep! in past projects i've actually used the .mat file format for both C and python bits because their wrapping of hdf5 is sufficiently well done. solved a lot of problems! :)
liuliu 6y ago
Soumith and I had discussions on hdf5 in the beginning of the AI renaissance (around 2014?). Torch maintainers definitely aware of better format at that time (original Torch in Lua has high-level File APIs for serialization). It becomes muddy when we moved from Caffe / TensorFlow to dynamic models with PyTorch where it is harder to see how to persist model (which means both the executable objects and the weights) efficiently and safely. At the end of the day, I think "export" and "checkpointing" should be two different things. An "exported" model should be safe to deploy and run on platforms like Azure ML while a "checkpointing" model should be treated like code and everything goes. That is probably where ONNX should be (for exporting).

hprotagonist 6y ago

the soopar geniuses at facebook clearly know better! /s (in truth, because pytorch serialised models can include python code for jit scripts, it's nonobvious what a good way to store python code is -- but torch has recently moved to a zipfile impl as of 1.6: https://pytorch.org/docs/stable/generated/torch.save.html https://pytorch.org/docs/stable/generated/torch.save.html) >I'd have expected people to be using something like hdf5 for this. Amusingly, matlab was way ahead of the pack here; matfiles have been hdf5 since r2006b, back when we just called it "matlab 7.3".

a-dub 6y ago
yep! in past projects i've actually used the .mat file format for both C and python bits because their wrapping of hdf5 is sufficiently well done. solved a lot of problems! :)

liuliu 6y ago

Soumith and I had discussions on hdf5 in the beginning of the AI renaissance (around 2014?). Torch maintainers definitely aware of better format at that time (original Torch in Lua has high-level File APIs for serialization). It becomes muddy when we moved from Caffe / TensorFlow to dynamic models with PyTorch where it is harder to see how to persist model (which means both the executable objects and the weights) efficiently and safely. At the end of the day, I think "export" and "checkpointing" should be two different things. An "exported" model should be safe to deploy and run on platforms like Azure ML while a "checkpointing" model should be treated like code and everything goes. That is probably where ONNX should be (for exporting).