4 ms·

> Pickle is slow ... Python is slow. But "slow" means "plenty fast" nowadays and the development speed advantage is immense. > unpickling malicious data can c

> Pickle is slow ... Python is slow. But "slow" means "plenty fast" nowadays and the development speed advantage is immense. > unpickling malicious data can cause security issues Why would I do that? I can't read the linked page because it seems to be down/the link is broken, so I don't know whether this includes user data that is present before pickling and then turns to be an issue after pickling. Then I would worry, otherwise ... yeah, I'm not gonna unpickle random data. > Just use JSON How do I effortlessly restore objects including their methods from JSON?

cratermoon 4y ago
>> unpickling malicious data can cause security issues > Why would I do that? If you pickle data from an untrusted source, say a web form submission and then later unpickle it. See https://cwe.mitre.org/data/definitions/502.html https://cwe.mitre.org/data/definitions/502.html
- ademarre 4y ago
  > If you pickle data from an untrusted source . . . and then later unpickle it That is not exactly right. The risk is when you unpickle data that was pickled by someone else or that was tampered with after you pickled it.
  - cratermoon 4y ago
    Look closer at the CWE and the linked examples: An attacker can construct a illegitimate, serialized object, like an auth token or sessionID, that instantiates one of Python's subprocesses to execute arbitrary commands
    - ademarre 4y ago
      That quote supports my statement. Notice that the serialized object is the thing that was constructed by the attacker, not some user data that you serialized yourself.
      - cratermoon 4y ago
        No the input was not serialized, it was carefully crafted so that when it gets serialized and deserialized, it triggers the malicious payload.
        ademarre 4y ago
        No. That is not how it works. https://docs.python.org/3/library/pickle.html https://docs.python.org/3/library/pickle.html https://blog.nelhage.com/2011/03/exploiting-pickle/ https://blog.nelhage.com/2011/03/exploiting-pickle/ (referenced from https://cwe.mitre.org/data/definitions/502.html#REF-467 https://cwe.mitre.org/data/definitions/502.html#REF-467)
TremendousJudge 4y ago
There's also the much faster cPickle. It may just be fast enough for your needs. If it isn't, then you start exploring other options.
- IshKebab 4y ago
  That was included in the benchmarks.
- kzrdude 4y ago
  the regular pickle module uses "cPickle" transparently. It should not be worth mentioning since Python 3.x. The article is 8 years old, so it kind of misses this detail.
  - TremendousJudge 4y ago
    That's good to know. I haven't worked professionaly with Python for a couple of years now, and back then we were maintaining a Python 2 system.
IshKebab 4y ago
> But "slow" means "plenty fast" nowadays Not in my experience. "Slow" means "it seems fast enough now and I'm sure we'll have time to rewrite it in a fast language once it's grown to a monster that processes 1000 times the data it does now... right?". > Why would I do that? Because you are using someone else's code and make the fairly reasonable assumption that deserialising data doesn't cause arbitrary code execution... But of course it's all your fault because you didn't read their code to see that it's using Pickle! > How do I effortlessly restore objects including their methods from JSON? You don't. You shouldn't.
- solarkraft 4y ago
  > You don't. You shouldn't. There's a good reason the functionality exists: It's a breeze for quickly persisting some state without worrying about anything else. It's not pretty, it's not clean, but it just works (which is arguably a use case Python is very popular for). A sibling comment pointed out that pickled data makes it annoying to deal with eventual schema changes. For sure. But so do other things that go hand in hand with quick-and-dirty approaches. "Probably don't use pickle in production" is something I could get behind, but that would, of course, not be such an inflammatory title.
vore 4y ago
One thing that's not mentioned is that pickled data is effectively fossilized once you've pickled it. If you want to change the layout of a class and have objects unpickle correctly, it can be an ordeal, as objects are unpickled by their class name, and you need both the original class and the new class to correctly unpickle and migrate. If you instead selectively pick what you want to serialize about your data and keep the representations separate, you can change the internal model easily without having a huge impact on the serialized model.
marcosdumay 4y ago
> How do I effortlessly restore objects including their methods from JSON? The recommendation from the title is usually made instead of something like "deserializing executable data is harmful". That is exactly the one question where the answer is "don't". It's not exactly the unpickling process that is the problem. It's how you established that the data isn't malicious. It is very hard to use pickle without creating some local privilege escalation possibilities. And at the end of the process, you usually don't get any capability that replicating the code on both sides of the communication channel wouldn't give you. (The problem isn't specific to Python either. There was a time when that kind of functionality was very hyped on both the industry and academia. For example, Java also got something similar that they had to retract. The famous Gnu-Hurd OS (the one that would never finish) was supposed to do that on the system level.)
- xhevahir 4y ago
  The Mozart/Oz people came up with pickle, I think.
- mananaysiempre 4y ago
  Do you, Programmer, take this Object to be part of the persistent state of your application, to have and to hold, through maintenance and iterations, for past and future versions, as long as the application shall live? Arturo Bejar, as quoted[1] in Mark Miller’s “Safe serialization under mutual suspicion”, which describes what it takes to make reasonable and compatible serialization restoring “all you can do is to send a message” objects. (The Smalltalk school actually spent quite a bit of time on the upgrade problem, see e.g. Fuel[2] and its references, but it was after the industry took the object orientation shiny and ran away with it, so that work seems to be little-known outside it.) [1] http://www.erights.org/data/serial/jhu-paper/upgrade.html http://www.erights.org/data/serial/jhu-paper/upgrade.html [2] http://wiki.squeak.org/squeak/6221 http://wiki.squeak.org/squeak/6221
NotTameAntelope 4y ago
Instantiate a new object of the class with the JSON as arguments, is one way. I’ve built a bunch of these systems, keeping your data separate solves a lot of future problems.
LtWorf 4y ago
The benchmark is bad. Because after you load a json you can't really use it. Well to use it you must check lists are lists for real, objects are really objects and have the keys you think they should have and so on. The alternative is using something like typedload (which I wrote) or pydantic in addition to json load, to avoid cluttering the code with the countless and error prone checks one must do to use untrusted json. In the end dealing with untrusted json directly is terrible.
- theamk 4y ago
  if you are dealing with untrusted data, pickle is not an option at all, it lacks security.
- vore 4y ago
  Isn't pickle just as bad if not worse? There's no guarantee that the unpickled object has the right shape either, AND it can arbitrarily execute code while unpickling.
  - LtWorf 4y ago
    Indeed, but if you say json is better and faster, you have to add all the costs, not just the parsing. Otherwise while json will not execute code, it will crash your program. And while in python it's ok, in C++ you can still execute code in that way :)

cratermoon 4y ago

>> unpickling malicious data can cause security issues > Why would I do that? If you pickle data from an untrusted source, say a web form submission and then later unpickle it. See https://cwe.mitre.org/data/definitions/502.html https://cwe.mitre.org/data/definitions/502.html

ademarre 4y ago
> If you pickle data from an untrusted source . . . and then later unpickle it That is not exactly right. The risk is when you unpickle data that was pickled by someone else or that was tampered with after you pickled it.
- cratermoon 4y ago
  Look closer at the CWE and the linked examples: An attacker can construct a illegitimate, serialized object, like an auth token or sessionID, that instantiates one of Python's subprocesses to execute arbitrary commands
  - ademarre 4y ago
    That quote supports my statement. Notice that the serialized object is the thing that was constructed by the attacker, not some user data that you serialized yourself.
    - cratermoon 4y ago
      No the input was not serialized, it was carefully crafted so that when it gets serialized and deserialized, it triggers the malicious payload.
      - ademarre 4y ago
        No. That is not how it works. https://docs.python.org/3/library/pickle.html https://docs.python.org/3/library/pickle.html https://blog.nelhage.com/2011/03/exploiting-pickle/ https://blog.nelhage.com/2011/03/exploiting-pickle/ (referenced from https://cwe.mitre.org/data/definitions/502.html#REF-467 https://cwe.mitre.org/data/definitions/502.html#REF-467)

TremendousJudge 4y ago

There's also the much faster cPickle. It may just be fast enough for your needs. If it isn't, then you start exploring other options.

IshKebab 4y ago
That was included in the benchmarks.
kzrdude 4y ago
the regular pickle module uses "cPickle" transparently. It should not be worth mentioning since Python 3.x. The article is 8 years old, so it kind of misses this detail.
- TremendousJudge 4y ago
  That's good to know. I haven't worked professionaly with Python for a couple of years now, and back then we were maintaining a Python 2 system.

IshKebab 4y ago

> But "slow" means "plenty fast" nowadays Not in my experience. "Slow" means "it seems fast enough now and I'm sure we'll have time to rewrite it in a fast language once it's grown to a monster that processes 1000 times the data it does now... right?". > Why would I do that? Because you are using someone else's code and make the fairly reasonable assumption that deserialising data doesn't cause arbitrary code execution... But of course it's all your fault because you didn't read their code to see that it's using Pickle! > How do I effortlessly restore objects including their methods from JSON? You don't. You shouldn't.

solarkraft 4y ago
> You don't. You shouldn't. There's a good reason the functionality exists: It's a breeze for quickly persisting some state without worrying about anything else. It's not pretty, it's not clean, but it just works (which is arguably a use case Python is very popular for). A sibling comment pointed out that pickled data makes it annoying to deal with eventual schema changes. For sure. But so do other things that go hand in hand with quick-and-dirty approaches. "Probably don't use pickle in production" is something I could get behind, but that would, of course, not be such an inflammatory title.

vore 4y ago

One thing that's not mentioned is that pickled data is effectively fossilized once you've pickled it. If you want to change the layout of a class and have objects unpickle correctly, it can be an ordeal, as objects are unpickled by their class name, and you need both the original class and the new class to correctly unpickle and migrate. If you instead selectively pick what you want to serialize about your data and keep the representations separate, you can change the internal model easily without having a huge impact on the serialized model.

marcosdumay 4y ago

> How do I effortlessly restore objects including their methods from JSON? The recommendation from the title is usually made instead of something like "deserializing executable data is harmful". That is exactly the one question where the answer is "don't". It's not exactly the unpickling process that is the problem. It's how you established that the data isn't malicious. It is very hard to use pickle without creating some local privilege escalation possibilities. And at the end of the process, you usually don't get any capability that replicating the code on both sides of the communication channel wouldn't give you. (The problem isn't specific to Python either. There was a time when that kind of functionality was very hyped on both the industry and academia. For example, Java also got something similar that they had to retract. The famous Gnu-Hurd OS (the one that would never finish) was supposed to do that on the system level.)

xhevahir 4y ago
The Mozart/Oz people came up with pickle, I think.
mananaysiempre 4y ago
Do you, Programmer, take this Object to be part of the persistent state of your application, to have and to hold, through maintenance and iterations, for past and future versions, as long as the application shall live? Arturo Bejar, as quoted[1] in Mark Miller’s “Safe serialization under mutual suspicion”, which describes what it takes to make reasonable and compatible serialization restoring “all you can do is to send a message” objects. (The Smalltalk school actually spent quite a bit of time on the upgrade problem, see e.g. Fuel[2] and its references, but it was after the industry took the object orientation shiny and ran away with it, so that work seems to be little-known outside it.) [1] http://www.erights.org/data/serial/jhu-paper/upgrade.html http://www.erights.org/data/serial/jhu-paper/upgrade.html [2] http://wiki.squeak.org/squeak/6221 http://wiki.squeak.org/squeak/6221

NotTameAntelope 4y ago

Instantiate a new object of the class with the JSON as arguments, is one way. I’ve built a bunch of these systems, keeping your data separate solves a lot of future problems.

LtWorf 4y ago

The benchmark is bad. Because after you load a json you can't really use it. Well to use it you must check lists are lists for real, objects are really objects and have the keys you think they should have and so on. The alternative is using something like typedload (which I wrote) or pydantic in addition to json load, to avoid cluttering the code with the countless and error prone checks one must do to use untrusted json. In the end dealing with untrusted json directly is terrible.

theamk 4y ago
if you are dealing with untrusted data, pickle is not an option at all, it lacks security.
vore 4y ago
Isn't pickle just as bad if not worse? There's no guarantee that the unpickled object has the right shape either, AND it can arbitrarily execute code while unpickling.
- LtWorf 4y ago
  Indeed, but if you say json is better and faster, you have to add all the costs, not just the parsing. Otherwise while json will not execute code, it will crash your program. And while in python it's ok, in C++ you can still execute code in that way :)