5 ms·
The github for this project doesn't use this terminology, but the keyword here is 'canonicalization'. That's the process in which you transform a document which
by niftich 9y ago
The github for this project doesn't use this terminology, but the keyword here is 'canonicalization'. That's the process in which you transform a document which can take any number of forms to a specific document that only takes one form.
You do this to ensure that documents that contain the same "information" will look identical on the wire, not just after-the-fact once you've processed them into some execution-specific data structure.
For example, X.509 certificates use ASN.1 DER, a restricted subset of ASN.1 BER where each value takes a deterministic form, so that two certificates that contain the same information will look identical when serialized into bytes. This is a stricter application of ASN.1, while, say, when you talk to a directory service over LDAP, you can speak BER, the looser encoding, because the exact bytes by which you're making yourself understood don't really matter.
Son is a canonicalized form of JSON, and the author maintains a page of JSON subsets and supersets [1] that lists two other canonicalized encodings, both of them containing silly flaws in their design. Son is a better effort.
[1] https://housejeffries.com/page/7 https://housejeffries.com/page/7
- mindB 9y agoThis isn't really a true canonicalization since it doesn't define an ordering. I was actually hoping it was. The matrix project [0] invented their own json canonicalization so that they could have servers meaningfully sign it. Was hoping some external standard of json canonicalization existed I could recommend them switching to. [0] https://matrix.org https://matrix.org
- Arathorn 9y agoYou could argue that Matrix’s canonical JSON form /is/ an open external standard ;) Meanwhile, Mastodon looks to have been implementing its own canonical JSON form recently too for signing purposes.
- pegasuscollins 9y agoWhat do you mean by ordering? This is easy to miss in the linked "spec" (I did too at first) but there is a sentence: "Object members must be sorted by ascending lexicographic order of their keys."
- Someone 9y agoThey need to tighten that a bit, though. “Lexicographic” isn’t clear enough even ignoring locale dependencies. Looking at the reference implementation, it contains encode = encodeUtf8 . generate So, the sorting is done before conversion to UTF-8. Relevant line, I think is genObject hm = "{" <> foldl' go mempty (sortOn fst (HM.toList hm)) <> "}" So, it uses Data.String’s sort order, which seems to be to lexicographic by Unicode code point (https://stackoverflow.com/a/3126287 https://stackoverflow.com/a/3126287) ⇒ implementations cannot sort the UTF-8 byte st sequences lexicographically. I think that’s a bad choice (if it was a choice and not an oversight) I’ve never written a single line of Haskell, so corrections welcome.
- pegasuscollins 9y agoHave you seen the relevant reference in RFC3629, page 2? It's explicitly listed there as a feature: "The byte-value lexicographic sorting order of UTF-8 strings is the same as if ordered by character numbers" Agree that specifying the keys to be ordered >by unicode code points< instead of >lexicographically< would be less ambiguous though.
- seagreen 9y agoI definitely meant ordering by Unicode code points. Someone very helpfully opened an issue and we're trying to figure out the right wording there: https://github.com/seagreen/Son/issues/13 https://github.com/seagreen/Son/issues/13
- Someone 9y agoLooks good. I still foresee interoperability problems between implementations, though. It just is too easy to mix up the ‘sort by key’ and ‘escape various control characters’ steps (CR sorts before ascii characters, but “\n” sorts after it) Even if the spec requires it, I fear implementations will also canonicalize strings differently, breaking sort order.
- lobster_johnson 9y agoActually, it does define an ordering: > Object members must be sorted by ascending lexicographic order of their keys. It also specifies that object keys must be unique, which is another canonicalization issue with JSON.
- mindB 9y agoyep. I completely missed that. My bad.
- nly 9y agoA stricter subset of JSON doesn't interest me without more specification surrounding how numerics should be decoded. How do I decode 18446744073709551615? What is the behaviour of a decoder that can't handle numeric types of that size? Should it cause a parse error? Should it truncate to 1.84467e+19? Should it overflow to 4294967295? Presumably if you want a canonical textual representation, the parse error is the only acceptable solution.
- d33 9y ago> What is the behaviour of a decoder that can't handle numeric types of that size? Is this actually relevant? It sounds like an implementation detail to me and it seems to boil down to using arbitrary size bigint. If you can't handle a bigint because it's too big for your machine, it sounds like a problem with the system, not parser.
- Spivak 9y agoWe can play this game all day. What if I pass an integer that takes 32GiB of memory to store? Also the system's fault if the resources aren't available?
- d33 9y agoI'd say so. This is kind of a security question - the data is valid from the protocol's perspective, it's the contents that are malicious. It's up to the developer to decide how to handle that - if it's super important to be able to process such big numbers, modify the parser to handle that (for example by storing them on the HDD). Same goes for another serialization format: bencoding. It supports arbitrarily nested data structures and most implementations are recursive, which can easily result in stack overflow if you mess up the implementation. But I wouldn't blame the standard - that's the price you pay for flexibility. At the end of the day, you need to sanitize your input anyway and keep your edge cases in mind. Same goes with XML bombs...
- rmrfrmrf 9y ago