3 ms·

Shelve uses pickle, which executes untrusted code without asking. These types of serialisation are a technical debt from day one. There are countless stories o

Shelve uses pickle, which executes untrusted code without asking. These types of serialisation are a technical debt from day one. There are countless stories of ecosystems doing a very painful migration to get away from it. See java, or recently the ML model files trying to go to safetensors etc.

OutOfHere 2y ago
There is nothing stopping you from storing just strings or bytes having safe representations.
- ysleepy 2y ago
  The same way that it's fine to use `eval()` to parse JSON I assume.
  - OutOfHere 2y ago
    This thread has bastardized serialization with storage even though the two are orthogonal concepts. If you don't want shelve to unpickle a hazardous object, give it a string or byte representation.
    - Dunedan 2y ago
      And that's exactly why it's so dangerous to use pickle, because it doesn't prevent you from executing untrusted code. It doesn't matter if you only pickle strings and bytes, as long as you unpickle any data, an attacker can replace that data with something which results in arbitrary code execution. If you only want to store bytes, the safe way to do so is to use the dbm module directly, which doesn't pickle/unpickle the data you provide.
      - OutOfHere 2y ago
        I find pickle useful for trusted objects that I created myself. pkl.xz is for example a fast way to reproducibly store and read Pandas dataframes. I have also found pickle useful for custom serialization of some objects that needed it.
kstrauser 2y ago
If you unpickle data someone else sends you, you’re playing with fire. I don’t see the risk in pickling your own data, say to cache it to disk, then reloading it later. An attacker could hack the store? If they have that access then all bets are already off. I suppose if a remote store is involved then the attack would be easier: you don’t need to hack the server running my code, just the DB/KV-store/whatever storing the pickles. Is that a common use case though? It seems like JSON and other dedicated formats are vastly more common for serializing data across machine boundaries.
cqqxo4zV46cp 2y ago
This is just flat out poor / lazy engineering. “Yeah, well, I’ve heard stories of people moving away from pickle” is no substitute for actually analysing your use case and determining what tool/s are a good fit. The fact that absolutist claims like “tech debt from day one” are so easy to prove wrong should make it obvious that the reality is more nuanced than this. And the correct response is not to lecture me about why untrusted code execution is bad. Threat models are not cookie cutter, either.

OutOfHere 2y ago

There is nothing stopping you from storing just strings or bytes having safe representations.

ysleepy 2y ago
The same way that it's fine to use `eval()` to parse JSON I assume.
- OutOfHere 2y ago
  This thread has bastardized serialization with storage even though the two are orthogonal concepts. If you don't want shelve to unpickle a hazardous object, give it a string or byte representation.
  - Dunedan 2y ago
    And that's exactly why it's so dangerous to use pickle, because it doesn't prevent you from executing untrusted code. It doesn't matter if you only pickle strings and bytes, as long as you unpickle any data, an attacker can replace that data with something which results in arbitrary code execution. If you only want to store bytes, the safe way to do so is to use the dbm module directly, which doesn't pickle/unpickle the data you provide.
    - OutOfHere 2y ago
      I find pickle useful for trusted objects that I created myself. pkl.xz is for example a fast way to reproducibly store and read Pandas dataframes. I have also found pickle useful for custom serialization of some objects that needed it.

kstrauser 2y ago

If you unpickle data someone else sends you, you’re playing with fire. I don’t see the risk in pickling your own data, say to cache it to disk, then reloading it later. An attacker could hack the store? If they have that access then all bets are already off. I suppose if a remote store is involved then the attack would be easier: you don’t need to hack the server running my code, just the DB/KV-store/whatever storing the pickles. Is that a common use case though? It seems like JSON and other dedicated formats are vastly more common for serializing data across machine boundaries.

cqqxo4zV46cp 2y ago

This is just flat out poor / lazy engineering. “Yeah, well, I’ve heard stories of people moving away from pickle” is no substitute for actually analysing your use case and determining what tool/s are a good fit. The fact that absolutist claims like “tech debt from day one” are so easy to prove wrong should make it obvious that the reality is more nuanced than this. And the correct response is not to lecture me about why untrusted code execution is bad. Threat models are not cookie cutter, either.