5 ms·

Pickles and .dlls solve completely different purposes. A .dll, which stands for "dynamic link library", is code - it's how you package parts of your implementat

Pickles and .dlls solve completely different purposes. A .dll, which stands for "dynamic link library", is code - it's how you package parts of your implementation in certain environments (Windows). A pickle file is data - it's a way of serializing out the contents of the state of a data structure, using some magic built into Python reflection, and then loading it back later. Pickling is very convenient for its purpose. It's also... very bad in production. The problem is that the serialization format is derived from the implementation and not very stable, so if the implementation changes - sometimes in subtle ways like upgrading your Python version - the serialized data isn't readable anymore. You're almost always better off either using an explicit data serialization format (JSON with schemas, protocol buffers, etc.) and writing your own logic for saving and restoring to/from it. > the pickle file is just a black box input data flows in, response flows out of it There's 100% more to it than that - there's got to be something that's evaluating the data in the pickle file and actually using it to compute the response. That substrate layer probably doesn't change often, but it should have its own versioning and deployment story.

bigger_cheese 4y ago
> There's 100% more to it than that - there's got to be something that's evaluating the data in the pickle file and actually using it to compute the response. I suspect that is the Microsoft Azure ML part I'm not sure. As I said I don't write the models, I just come along afterwards and review the results. I don't know anything about Python.
fbdab103 4y ago
>the serialization format is derived from the implementation and not very stable, so if the implementation changes - sometimes in subtle ways like upgrading your Python version - the serialized data isn't readable anymore. Could be wrong, but I do not believe this is true anymore. There are pickle versions, but I believe the format has been standardized such that you should be able to freely move between Python releases.
- jononor 4y ago
  The base serialization is stable and compatible across Python versions. However pickle files can serialize entire Python objects, including from custom and third party code. Compatibility of serialization then depends on the versions of the code that implements those objects, which is often not well managed.
  - fbdab103 4y ago
    Ah that's fair. I was only considering serializing native types (for which there are much better options).

bigger_cheese 4y ago

> There's 100% more to it than that - there's got to be something that's evaluating the data in the pickle file and actually using it to compute the response. I suspect that is the Microsoft Azure ML part I'm not sure. As I said I don't write the models, I just come along afterwards and review the results. I don't know anything about Python.

fbdab103 4y ago

>the serialization format is derived from the implementation and not very stable, so if the implementation changes - sometimes in subtle ways like upgrading your Python version - the serialized data isn't readable anymore. Could be wrong, but I do not believe this is true anymore. There are pickle versions, but I believe the format has been standardized such that you should be able to freely move between Python releases.

jononor 4y ago
The base serialization is stable and compatible across Python versions. However pickle files can serialize entire Python objects, including from custom and third party code. Compatibility of serialization then depends on the versions of the code that implements those objects, which is often not well managed.
- fbdab103 4y ago
  Ah that's fair. I was only considering serializing native types (for which there are much better options).