5 ms·

Maps and Sets are entirely irrelevant when it comes to serialization. You may as well store or transmit that information as an array (of pairs, in case of Map).

by pitaj 8y ago

Maps and Sets are entirely irrelevant when it comes to serialization. You may as well store or transmit that information as an array (of pairs, in case of Map). The point of a Map or Set (O(1) insertion / removal / contains) don't matter when you're talking about serialization.

repsilat 8y ago
Well, unlees you're using something like capnp, where O(1) contains is meaningful when O(n) deserialise is not acceptable. And really, if you have consumers/producers in different languages/codebases, built-in support for these things is a great convenience. You wouldn't say that objects/dicts in JSON are irrelevant, would you?
- derefr 8y ago
  > You wouldn't say that objects/dicts in JSON are irrelevant, would you? I kind of would. Coming from Erlang, I don't see a point to there being a whole separate syntax for encoding object/map types. Just encode objects/maps as arrays of pairs (2-Tuples, or just length-2 arrays). Then, if you want ser-des type fidelity, stick an annotation onto the array (like in YAML) to say what type it should decode to. E.g., something like: [1, 2, 3] # array [['a', 1], ['b', 2], ['c', 3]] # array of pairs [@object, ['a', 1], ['b', 2], ['c', 3]] # array of pairs hinted so it should decode to an Object [@map, ['a', 1], ['b', 2], ['c', 3]] # array of pairs hinted so it should decode to a Map The nice thing about this approach is that JSON libraries could do as much or as little work in parsing out the annotations as they want: they could recognize the annotations and construct the referred-to type; or they could just pass back the array with the annotation expressed as an Annotation value.
tonyg 8y ago
Do you want remote code execution? Because that's how you get remote code execution. http://docs.couchdb.org/en/2.1.1/cve/2017-12635.html http://docs.couchdb.org/en/2.1.1/cve/2017-12635.html Map and set values are important for many applications. The fact that JSON doesn't have a way of denoting a map or set value (or anything else, but that's another issue) is a problem: it means there's no understanding common to all JSON consumers about what syntax denotes a map or a set. Shortcuts are taken, consumers disagree on the details, and the linked CVE is the result. Being able to reliably convey "this is intended to be a map", or "this is intended to be a set" is crucial for secure and robust interoperability.
- skolemtotem 8y ago
  That's really weird - why didn't they parse it into a map in the first place, immediately removing duplicate keys? This seems like more of an argument for automatic deserializer generation than it does for adding undue complexity to a serialization format.
  - tonyg 8y ago
    Even that's not enough to save you in general. JSON deliberately (!) permits duplicate keys. It leaves it up to the implementation to decide how to handle them. And there are two (sensible) ways to add key/value pairs to a map: either keep the first occurrence of key X, or the last occurrence of key X. Some JSON libraries pick one way; others pick the other. Getting the two to interoperate is, as we see, not easy. It would have been better for JSON to forbid duplicate keys, or to specify a mandatory first-wins or last-wins policy. Then there'd be no room for error.
- camgunz 8y ago
  > Do you want remote code execution? Because that's how you get remote code execution. It's fundamentally not the case that any data serialization format without support for map and set values allows for remote code execution. This CVE was CouchDB's auth code not handling some edge cases in the JSON standard. Your argument that edge cases like this lead to problems is definitely a good one though. I'm more inclined to say that a lot of these kinds of issues are due to people thinking JSON is a simple, straightforward format when it definitely isn't -- and that's due to mostly two things: - It fits on a business card! - You can serialize/deserialize in one line in most implementations Serialization/Deserialization is something you should always pay careful attention to. Making it a one-liner and advertising it as such was pretty irresponsible.

by pitaj 8y ago

repsilat 8y ago

Well, unlees you're using something like capnp, where O(1) contains is meaningful when O(n) deserialise is not acceptable. And really, if you have consumers/producers in different languages/codebases, built-in support for these things is a great convenience. You wouldn't say that objects/dicts in JSON are irrelevant, would you?

derefr 8y ago
> You wouldn't say that objects/dicts in JSON are irrelevant, would you? I kind of would. Coming from Erlang, I don't see a point to there being a whole separate syntax for encoding object/map types. Just encode objects/maps as arrays of pairs (2-Tuples, or just length-2 arrays). Then, if you want ser-des type fidelity, stick an annotation onto the array (like in YAML) to say what type it should decode to. E.g., something like: [1, 2, 3] # array [['a', 1], ['b', 2], ['c', 3]] # array of pairs [@object, ['a', 1], ['b', 2], ['c', 3]] # array of pairs hinted so it should decode to an Object [@map, ['a', 1], ['b', 2], ['c', 3]] # array of pairs hinted so it should decode to a Map The nice thing about this approach is that JSON libraries could do as much or as little work in parsing out the annotations as they want: they could recognize the annotations and construct the referred-to type; or they could just pass back the array with the annotation expressed as an Annotation value.

tonyg 8y ago

Do you want remote code execution? Because that's how you get remote code execution. http://docs.couchdb.org/en/2.1.1/cve/2017-12635.html http://docs.couchdb.org/en/2.1.1/cve/2017-12635.html Map and set values are important for many applications. The fact that JSON doesn't have a way of denoting a map or set value (or anything else, but that's another issue) is a problem: it means there's no understanding common to all JSON consumers about what syntax denotes a map or a set. Shortcuts are taken, consumers disagree on the details, and the linked CVE is the result. Being able to reliably convey "this is intended to be a map", or "this is intended to be a set" is crucial for secure and robust interoperability.

skolemtotem 8y ago
That's really weird - why didn't they parse it into a map in the first place, immediately removing duplicate keys? This seems like more of an argument for automatic deserializer generation than it does for adding undue complexity to a serialization format.
- tonyg 8y ago
  Even that's not enough to save you in general. JSON deliberately (!) permits duplicate keys. It leaves it up to the implementation to decide how to handle them. And there are two (sensible) ways to add key/value pairs to a map: either keep the first occurrence of key X, or the last occurrence of key X. Some JSON libraries pick one way; others pick the other. Getting the two to interoperate is, as we see, not easy. It would have been better for JSON to forbid duplicate keys, or to specify a mandatory first-wins or last-wins policy. Then there'd be no room for error.
camgunz 8y ago
> Do you want remote code execution? Because that's how you get remote code execution. It's fundamentally not the case that any data serialization format without support for map and set values allows for remote code execution. This CVE was CouchDB's auth code not handling some edge cases in the JSON standard. Your argument that edge cases like this lead to problems is definitely a good one though. I'm more inclined to say that a lot of these kinds of issues are due to people thinking JSON is a simple, straightforward format when it definitely isn't -- and that's due to mostly two things: - It fits on a business card! - You can serialize/deserialize in one line in most implementations Serialization/Deserialization is something you should always pay careful attention to. Making it a one-liner and advertising it as such was pretty irresponsible.