10 ms·
Son: A minimal subset of JSON for machine-to-machine communication
- fanf2 9y agoThis spec mostly concentrates on simplifying the syntax of numbers, but it does not do anything about one of the big omissions in JSON: it does not specify the range of integers or the precision of fractions. There are a lot of interoperability problems in that omission. (The other thing SON does is require unique keys, which is a good improvement that fixes an interop bug that can cause security vulnerabilities.)
- mehrdadn 9y ago> but it does not do anything about one of the big omissions in JSON: it does not specify the range of integers or the precision of fractions Could you give an example of a storage format that does do this and the problems that doing so avoids?
- based2 9y agohttps://stackoverflow.com/questions/6175592/xml-schema-precision-of-decimal-value https://stackoverflow.com/questions/6175592/xml-schema-preci... http://www.ddialliance.org/Specification/DDI-Codebook/2.5/XMLSchema/field_level_documentation_files/schemas/codebook_xsd/elements/range.html http://www.ddialliance.org/Specification/DDI-Codebook/2.5/XM... https://www.obj-sys.com/docs/xbv23/JavaCSUsersGuide/ch06s03s02.html https://www.obj-sys.com/docs/xbv23/JavaCSUsersGuide/ch06s03s...
- majewsky 9y agoI'm not working with binary protocols that often, so I cannot point to an existing protocol. However, if I had to implement a serialization format that doesn't need to be human-readable, I would never use number literals. I would write numbers as one byte whose value means e.g. "the next 4 bytes are an uint32 in little-endian", followed by just these 4 bytes. Then you also don't have to worry about whitespace, because there is none. The next field can follow directly afterwards, because the type header byte defines how long exactly the current field is. (For variable-length strings, use something like netstrings.) The only problem that I can see with this is that Son wants to ensure deterministic encoding, but in my example, 0 could be encoded as an integer of any size or signedness. You could add a rule like "encode every integer as the smallest possible type", but that may have practical implications elsewhere. I wouldn't be too worried about it, seeing how half-baked Son appears to be. For example: > Object members must be sorted by ascending lexicographic order of their keys. Yeah right, because there is only one lexicographic order in the world.
- pegasuscollins 9y ago> Yeah right, because there is only one lexicographic order in the world. The specification tells you all strings (keys) must be UTF8. There is only one lexicographic ordering of UTF8 strings (RFC3629, page 2). This should probably be explicitly mentioned in the linked specification though.
- majewsky 9y ago> There is only one lexicographic ordering of UTF8 strings. Then why is LC_COLLATE still a thing? Also, you don't get to second-guess a spec. If it does not specify everything that it needs to, it deserves the "half-baked" label.
- pegasuscollins 9y ago> The only problem that I can see with this is that Son wants to ensure deterministic encoding, but in my example, 0 could be encoded as an integer of any size or signedness. To be pedantic: Not really. Assuming your example uses twos complement (what else would it use) and you meant "int32" instead of "uint32", negative zero and positive zero would still be the same encoded value. Also since you said "[u]int32", there is only one size the integer can be: 32 bit. So, in your example, the literal value zero has one and only one unique binary representation, which is "0x00000000"
- majewsky 9y agoI implied that a useful serialization format would include integers of multiple size. When your field only needs a uint16, you don't want to pay the extra 6 bytes to encode as, say, uint64.
- pegasuscollins 9y agoYou could either stipulate (as you suggested) to always use the smallest width that still fits a number and then prefix with a length byte but that would be wasteful. Ideally something like LEB128 would be used. At any rate, it's not an issue in terms of guaranteeing a bijective mapping.
- pegasuscollins 9y agoPretty much every binary format will encode integers using a fixed width or varlen scheme in "base 2". This generally is done for two major reasons: First of all, such an encoding is significantly easier and cheaper to parse than a base10+ascii (human-readable) encoding. I encourage you to write a parser that reads a fixed, 32-bit binary number (1) and another parser that reads a JSON-formatted number string into an internal variable in a classic language like Java or C++. You will immediately see the big difference in complexity. Make sure your parser can also deal with a message that contains more than just one number, i.e. the parser should be able to tell at which byte index an encoded number begins and ends. Even if you're using a language or library where this is hidden from you (e.g. by using parseInt or std::stod) the same work still happens behind the scenes. The other reason is that for most numbers and fixed/varint encoding schemes, the "binary" representation will be much more compact. Storing the number "1000000" in base10+ascii (human readable) takes at least eight bytes. Storing the same number in a fixed 32-bit integer encoding takes four bytes. Using a varint encoding scheme might allow you to get down to three bytes. (1) Ignoring stuff like byte order and representation of negative numbers; this is usually fixed in the protocol/format specification.
- user5994461 9y agoThere are no integers in javascript, numbers are always represented as double. Most implementations of json respect that. The error people get are because numbers are converted from double to something else that's not equivalent. Precision is around 15 digits and the usual rounding issues are applicable.
- hanbura 9y agoJSON is not javascript, and you can write valid JSON that isn't valid javascript (using unusual line seperators) No JSON standard restricts the range of numbers (except for universally prohibiting NaN and infinity, making JSON numbers clearly not IEEE doubles). The standards warns of potential interoperability issues with numbers outside the numbers representable as double, but they do allow all numbers representable in Base 10.
- nly 9y agoYou're wrong. There isn't a JSON specification anywhere that says numerics are IEEE 754 doubles. The RFC mentions "good interoperability" can be achieved by using doubles, but does not say this MUST or even SHOULD be supported.
- emodendroket 9y agoI'd also like to see a built-in date format instead of requiring the parsing of numbers or strings to get them.
- sundarurfriend 9y agoThere's a github issue on this, titled "Son inherits the weakness of JSON by not specifying the numeric value ranges" [1], in case anyone has anything concrete to suggest. [1] https://github.com/seagreen/Son/issues/3 https://github.com/seagreen/Son/issues/3
- algesten 9y agoI remember reading this a while back. I think the SON specification could tighten up a few more things, like number ranges and further string escaping problems. http://seriot.ch/parsing_json.php http://seriot.ch/parsing_json.php
- seagreen 9y agoLimiting the size of numbers is intentionally left out of Son right now, because I'm not sure where that limit should be. My hope is that we eventually have a good schema language for JSON (something like JSON Schema, though I'm not sure about that one in particular) and then people can set number ranges in their schemas based on their particular use case. There shouldn't be any remaining string escaping problems though! If you find one definitely open a GitHub issue.
- Deid2AMa 9y agoWhy not just use MsgPack or CBOR?
- js8 9y agoI wish people wouldn't use JSON for M2M communication and used e.g. CBOR instead. It's such a waste of resources. In the 70's, using text formats for protocols was understandable (7-bit serial lines..). Today, it is not.
- emodendroket 9y agoI think people like the ease of human-readable formats for debugging.
- IshKebab 9y agoJust decode the data before logging it... I mean, I can't think of many situations where I'm debugging something by looking at raw bytes when I couldn't easily just dump the decoded representation instead. People don't use BCD for ease of debugging do they?
- emodendroket 9y agoI mean, maybe it's worth doing anyway, but it's an extra bit of friction. Plus you probably can avoid issues where the binary doesn't decode properly because of version changes, different environment, etc.
- foobarian 9y agoIn Javaland BSON for production messages performs close to JDK serialization, and for debugging it's not hard to override the content types at runtime (or log the messages in readable form where they are getting serialized).
- deleted 9y ago[deleted]
- maltalex 9y agoJSON loses its appeal once you're not targeting browsers and/or anything else that uses Javascript. Sure, it's simple and familiar, but it has serious drawbacks such as the time and CPU resources it takes for parsing and the lack of schema. For those of you interested, I suggest using Google's protobufs or even full blown gRPC for such endeavors. See discussion about gRPC here: https://news.ycombinator.com/item?id=12344995 https://news.ycombinator.com/item?id=12344995 Protobufs: https://developers.google.com/protocol-buffers/ https://developers.google.com/protocol-buffers/
- Anderkent 9y agoCBOR is pretty good if you just want to move data around.
- pagnol 9y agoI didn't know CBOR, but it looks similar to BSON and MessagePack, or is there a significant difference?
- heavenlyhash 9y agoRoughly that it isn't shit. I say that a bit tongue in cheek, but really: BSON has a monstrous amount of basic data types, some of which make very little sense and are very difficult to round-trip through other formats; and MessagePack had crippling basic sanity issues with what-are-bytes-vs-strings in its early history and that's something you simply can't recover from without declaring game over and making a new spec. CBOR has neither of these issues. It's almost completely isomorphic to json, plus supports byte arrays. It's very nice.
- joelthelion 9y agoIf I could change JSON, I would add comments to it, not make it simpler.
- IshKebab 9y agoHJSON.
- joelthelion 9y agoThanks, that looks useful.
- lioeters 9y agoThere's also JSON5: http://json5.org/ http://json5.org/ I use HJSON fairly often, in browser and on server, mostly during development.
- matharmin 9y agoI see many comments here on the efficiency of binary formats versus JSON. While it is certainly possible, especially with a well designed and implemented protocol, just remember that you don't automatically get massive gains by using a binary protocol: * JSON compresses well, and gzipped JSON is usually smaller than a corresponding binary format without compression. * Being such a common format, practically every language has a highly optimized JSON parser. Just because a binary format can be faster to parse, doesn't mean the implementation is necessarily faster. As a specific example, I've had experience in the past where the most common protobuf implementation for Ruby at the time was an order of magnitude slower than the default JSON parser (it has probably improved by now). So by all means use a binary format for (internal) machine-to-machine communication if the performance gains are significant, but don't just assume that JSON will be too inefficient.
- alfanick 9y ago"Just because a binary format can be faster to parse, doesn't mean the implementation is necessarily faster." - except "JSON compresses well, and gzipped JSON is usually smaller than a corresponding binary format without compression", gzip takes some cycles...
- k__ 9y agoHow about JSAN? With pre-defined record structure you could simply use arrays instead of objects and save much space.
- tzahola 9y agoDaily reminder that there must be a sane timeline where telcos didn't proliferate ASN.1 to death, and machine-to-machine communication was solved decades ago.
- niftich 9y agoThe github for this project doesn't use this terminology, but the keyword here is 'canonicalization'. That's the process in which you transform a document which can take any number of forms to a specific document that only takes one form. You do this to ensure that documents that contain the same "information" will look identical on the wire, not just after-the-fact once you've processed them into some execution-specific data structure. For example, X.509 certificates use ASN.1 DER, a restricted subset of ASN.1 BER where each value takes a deterministic form, so that two certificates that contain the same information will look identical when serialized into bytes. This is a stricter application of ASN.1, while, say, when you talk to a directory service over LDAP, you can speak BER, the looser encoding, because the exact bytes by which you're making yourself understood don't really matter. Son is a canonicalized form of JSON, and the author maintains a page of JSON subsets and supersets [1] that lists two other canonicalized encodings, both of them containing silly flaws in their design. Son is a better effort. [1] https://housejeffries.com/page/7 https://housejeffries.com/page/7
- mindB 9y agoThis isn't really a true canonicalization since it doesn't define an ordering. I was actually hoping it was. The matrix project [0] invented their own json canonicalization so that they could have servers meaningfully sign it. Was hoping some external standard of json canonicalization existed I could recommend them switching to. [0] https://matrix.org https://matrix.org
- Arathorn 9y agoYou could argue that Matrix’s canonical JSON form /is/ an open external standard ;) Meanwhile, Mastodon looks to have been implementing its own canonical JSON form recently too for signing purposes.
- pegasuscollins 9y agoWhat do you mean by ordering? This is easy to miss in the linked "spec" (I did too at first) but there is a sentence: "Object members must be sorted by ascending lexicographic order of their keys."
- mlawren 9y agoSomewhat off-topic, but I had a need recently for a simple serialization format that satisfied the following: * Support for undef/null * Support for binary data * Support for UTF8 strings * Universally-recognized canonical form for hashing * Trivial to construct on the fly from SQLite triggers Of all of the formats I looked at nothing matched. So I created Bifcode [https://metacpan.org/pod/Bifcode https://metacpan.org/pod/Bifcode]. It is a bit of a mix between Netstrings, Bencode and JSON. It is not really human readable, but it is extremely simple and robust, easy to generate and parse, and therefore relatively secure. I'm not promoting it for any particular use case, and I only have a Perl implementation. Posting here as I think some of the audience in this thread may find it useful.
- iainmerrick 9y agoFirst bullet point: - No exponential notation That's going to give you some pretty big numbers. It would be better to require exponential notation.
- seagreen 9y agohttps://github.com/seagreen/Son/issues/10 https://github.com/seagreen/Son/issues/10
- grogenaut 9y agoMan I want the opposite. If it is machine to machine I can use a machine to decide it. For Jon I want comments, trailing commas, and a few other things I can't think of at 9am Saturday
- seagreen 9y agoYou're looking for supersets of JSON, see the last section here: https://housejeffries.com/page/7 https://housejeffries.com/page/7. I'm partial to JSON5.
- kbumsik 9y agoNo exponential notation?? I don’t see why this change makes it suitable for machine-to-machine communication. This looks like rather for the human readability, not for the machine. It surely removes some branches but I don’t think it will much improve decoding performance. At least this has to provide convincing reason to use it over JSON e.g. performance comparison.
- seagreen 9y agoI opened an issue for this here: https://github.com/seagreen/Son/issues/10 https://github.com/seagreen/Son/issues/10 I'm curious about how often enough trailing zeros occur in real world documents for lack of exponential notation to have a noticeable impact on document size.
- alfanick 9y agoCBOR, FlatBuffers, ProtoBuf, MsgPack, BSON - so many better choices. Please do not make JSON a standard for M2M communication, it's as bad as calling Electron apps "native".
- megous 9y agoIt's already a standard for M2M communication, unless you see someone writing JSON by hand when doing anything on the web.
- c54 9y agoI love the parser/state machine diagrams for all the matchers, does anyone (OP?) know what tool was used to generate those?
- alfanick 9y agoPlease do read: "# Notes + The diagrams were generated with [GrammKit](https://github.com/dundalek/GrammKit https://github.com/dundalek/GrammKit)."