6 ms·

Bits on the Wire

116 points by yarapavan 7y ago

sarosh 7y ago
"The take-away is that if you’re going to start emitting events from a piece of software, put just as much care into it as you would as you do in specifying an API. Because event formats are a contract, too."
kragen 7y ago
> Then XML came along in 1998. Useful context for the youngsters: Tim invented XML. I mean, techically, there were a few other people who helped write the spec, but Tim was using more-or-less XML in shipped applications before they got involved. In my book, that means his opinion about the importance of bits on the wire is pretty weighty, but you also shouldn't discount the possibility that he's in some sense constitutionally inclined toward standardized data formats. About other data formats: I have some experience with CBOR which I regret; it's more compact than JSON, but it's usually not significantly more compact than gzipped JSON, and it's usually slower than JSON, which is saying something, and is surprising given that no scanning for delimiters is needed. Its only real advantage is that you can safely include binary blobs. It's worth noting that originally CBOR was part of MessagePack until its author decided to fork it off and name it after himself, Carsten Bormann. Cap'n Proto, FlatBuffers, and SBE (which I guess is too obscure for Tim to mention) get their "deserialization in negative nanoseconds" by virtue of not deserializing; instead, the respective libraries access the desired fields of the serialized data in place, which involves an extra pointer indirection or two and possibly some byte swapping, so it's a little slower than accessing fields of native structs. But it means you don't have to parse the entire data structure, just the parts you're accessing, so you can do things like memory-map a read-only data file instead of parsing it. In many cases, even if you're accessing the whole thing, saving the extra memory copy is actually worth more performance than avoiding the byte-swapping and indirections. This is not a new idea — COBOL was based on it, though without these systems' extensibility and consequently without the cost of the extra indirections — but it's an extremely effective idea in many contexts. All of FlatBuffers, Cap'n Proto, SBE, Protobufs, Thrift, and I think Avro do have support for adding new fields, contrary to what Tim says about binary formats in his note; but they mostly do presuppose that those new fields are being defined by some central authority, because there are things like sequentially assigned field IDs, so certainly you don't have the level of safe extensibility that you have in JSON (or HTTP or RFC-822 mail or HTML forms or SNMP ASN.1 BER data or, of course, XML and HTML tags and attributes).
- timbray 7y ago
  Ahem. No I didn't: https://www.tbray.org/ongoing/When/200x/2008/11/22/Not-the-Inventor-of-XML https://www.tbray.org/ongoing/When/200x/2008/11/22/Not-the-I... But, I was there, and it's an interesting story: https://www.tbray.org/ongoing/When/200x/2008/02/10/XML-People https://www.tbray.org/ongoing/When/200x/2008/02/10/XML-Peopl...
  - kragen 7y ago
    Heh, sorry to have fulfilled your prophecy. But, weren't you deploying a schema-free SGML app by virtue of not using implied end tags a couple of years before? Because that sure sounds like XML to me...
    - timbray 7y ago
      Yeah, but others were as well, it was obvious to anyone who understood both SGML and the Web. Mind you, there were only maybe a dozen people in the world like that.
      - kragen 7y ago
        And one of them was Erik Naggum, who understood the Web but hated it. He probably wasn't the only one who self-selected out of the pool of possible XML inventors... BTW, it's mentioned in another thread above that you're not sending a Content-Type header — are you doing that on purpose? I haven't tried doing that since HTTP/0.9 days and I'm surprised it works (i.e., that modern browsers sniff the body and render it as HTML).
        tempguy9999 7y ago
        I'm curious; could you explain/link to some explanation of Eric Naggum's views that showed why he disliked the web. Thanks.
- tialaramex 7y ago
  To be fair, most of those "extensible" formats aren't safely extensible. ASN.1 is because it uses OIDs, and some scenarios in XML and parts of HTTP are safely extensible because they use URIs, but anywhere which relies on just arbitrary text, like JSON, is just shrugging when it comes to global collisions. What I'm getting at here is, OIDs and URIs are globally namespaced. When I decide to add my own custom foo to a format, OIDs let me go get myself an arc and assign foo an OID from that arc, and URIs let me go get myself an FQDN and then assign foo a URL with that FQDN. In HTML if I just add a foo attribute that collides with everybody else's foo attribute, in RFC-822 mail my foo header can collide with anybody else's foo header. And so on.
Thorentis 7y ago
Interesting article, but lots of "probably the next biggest consumer" etc. Was hoping he would have some kind of network traffic analysis report in there that backs up his ranking of the formats. Seems to be anecdotal more than anything. Still good overview of the different formats available.
deleted 7y ago
[deleted]
AdamN 7y ago
To me the biggest annoyance of json is there's no standard way to put comments in there. What's the point of a text-based serialization format if it can't carry comments??? It has one advantage over YAML though, with JSON you know when you're not at the end of the file :-)
- akersten 7y ago
  The downside of this is that the comment becomes just another piece of the data model, because anything that consumes that JSON will need to decide: do I preserve comments, or remove them, when re-emitting the JSON? If I remove them, comments are kind of only useful for things like initial config files or what have you. Which might be fine for your use case, but JSON broadly is not about making configuration files - this is more suited for something like YAML. If you preserve them - how is that different from a key-value just named "comment": "Here's my comment"? (And if the answer is "because it might not crash poorly coded applications," that's more of a discussion about the application than it is about comments in JSON)
  - vageli 7y ago
    Often times people add comments in-line with things to prevent the comment from becoming stale should the referent change (because the inlining encourages modifying the comment if the referent is modified), whereas it may be hard to determine the referent of a comment in its own field. Further, having multiple comments with the same field name is invalid JSON I believe, as keys must be unique? Though I may be wrong. (however, if keys _must_ be unique, then comment authors could encode the referent in the comment key, possibly).
  - bmm6o 7y ago
    >If I remove them, comments are kind of only useful for things like initial config files or what have you.* True, but that's a pretty big use case to dismiss with "only". I don't think people want their json api responses to be able to include comments, but it's definitely convenient to be able to comment out parts of a config file. How/if to expose that through the deserializer is an interesting question, it would be fine with me if it dropped them.
jkaptur 7y ago
> ... schemas aren’t free. You have to coördinate between sender and receiver, and you have to worry what happens when someone wants to add a new field to a message type — but in raw JSON, you don’t have to worry, you just toss in the new field and things don’t break. Flexibility is a good thing. Much like type systems in general, you always have a schema, and you always have to worry about what happens when someone adds a field. You just get to choose whether it's explicit and you have to type it alllll the way out allllll the time, or whether it's implicit, and is defined as "whatever all the usages are everywhere". Just as gradual typing is taking over the world (TypeScript, MyPy, etc.), I predict success for something that lets you gradually add schemas to JSON.
- vincnetas 7y ago
  > ... you just toss in the new field and things don’t break. Flexibility is a good thing. Except when it's not you that are flexible but other parties, and they express that flexibility not only by adding but also removing or modifying fields without notice :/
  - erik_seaberg 7y ago
    I very rarely see system that distinguish between "this new feature can be safely ignored" and "it's not safe to process this message if you don't support this new feature", but I've always thought it a good idea. Like semantic versioning without a global ordering over schema changes.
jolmg 7y ago
My browser opens up a download dialog on clicking this link. It's probably because the response headers don't include "Content-Type", and so it doesn't know if it will be able to interpret the response.
- kragen 7y ago
  Interesting, what browser?
  - jolmg 7y ago
    Firefox on Archlinux. Though, now I think the issue is with an extension I have.
    - kragen 7y ago
      Weird, I'm not having that problem in Firefox on Debian. But it's sure enough true that there's no Content-Type header, either with LWP or with Firefox. Tim, are you doing that on purpose?
      - jolmg 7y ago
        I found it's my extension[1]. That's what's interpreting the lack of Content-Type as indication that Firefox is going to force a download (it actually doesn't) and showing me its own download dialog. [1] https://github.com/Rob--W/open-in-browser https://github.com/Rob--W/open-in-browser
commandersaki 7y ago
A quick note about protobuf. It's just way too complicated in my opinion. If you need a binary protocol just use TLVs (Tag-Length-Value) - so much less cognitive load and the parsing is easy to implement and read compared to the complicated autogenerated mess that is protobuf.
geraldbauer 7y ago
A quick note about json next. I collect new JSON variants, extensions and formats at the Awesome JSON Next page [1]. [1]: https://github.com/json-next/awesome-json-next https://github.com/json-next/awesome-json-next

116 points by yarapavan 7y ago

"The take-away is that if you’re going to start emitting events from a piece of software, put just as much care into it as you would as you do in specifying an API. Because event formats are a contract, too."

kragen 7y ago

> Then XML came along in 1998. Useful context for the youngsters: Tim invented XML. I mean, techically, there were a few other people who helped write the spec, but Tim was using more-or-less XML in shipped applications before they got involved. In my book, that means his opinion about the importance of bits on the wire is pretty weighty, but you also shouldn't discount the possibility that he's in some sense constitutionally inclined toward standardized data formats. About other data formats: I have some experience with CBOR which I regret; it's more compact than JSON, but it's usually not significantly more compact than gzipped JSON, and it's usually slower than JSON, which is saying something, and is surprising given that no scanning for delimiters is needed. Its only real advantage is that you can safely include binary blobs. It's worth noting that originally CBOR was part of MessagePack until its author decided to fork it off and name it after himself, Carsten Bormann. Cap'n Proto, FlatBuffers, and SBE (which I guess is too obscure for Tim to mention) get their "deserialization in negative nanoseconds" by virtue of not deserializing; instead, the respective libraries access the desired fields of the serialized data in place, which involves an extra pointer indirection or two and possibly some byte swapping, so it's a little slower than accessing fields of native structs. But it means you don't have to parse the entire data structure, just the parts you're accessing, so you can do things like memory-map a read-only data file instead of parsing it. In many cases, even if you're accessing the whole thing, saving the extra memory copy is actually worth more performance than avoiding the byte-swapping and indirections. This is not a new idea — COBOL was based on it, though without these systems' extensibility and consequently without the cost of the extra indirections — but it's an extremely effective idea in many contexts. All of FlatBuffers, Cap'n Proto, SBE, Protobufs, Thrift, and I think Avro do have support for adding new fields, contrary to what Tim says about binary formats in his note; but they mostly do presuppose that those new fields are being defined by some central authority, because there are things like sequentially assigned field IDs, so certainly you don't have the level of safe extensibility that you have in JSON (or HTTP or RFC-822 mail or HTML forms or SNMP ASN.1 BER data or, of course, XML and HTML tags and attributes).

timbray 7y ago
Ahem. No I didn't: https://www.tbray.org/ongoing/When/200x/2008/11/22/Not-the-Inventor-of-XML https://www.tbray.org/ongoing/When/200x/2008/11/22/Not-the-I... But, I was there, and it's an interesting story: https://www.tbray.org/ongoing/When/200x/2008/02/10/XML-People https://www.tbray.org/ongoing/When/200x/2008/02/10/XML-Peopl...
- kragen 7y ago
  Heh, sorry to have fulfilled your prophecy. But, weren't you deploying a schema-free SGML app by virtue of not using implied end tags a couple of years before? Because that sure sounds like XML to me...
  - timbray 7y ago
    Yeah, but others were as well, it was obvious to anyone who understood both SGML and the Web. Mind you, there were only maybe a dozen people in the world like that.
    - kragen 7y ago
      And one of them was Erik Naggum, who understood the Web but hated it. He probably wasn't the only one who self-selected out of the pool of possible XML inventors... BTW, it's mentioned in another thread above that you're not sending a Content-Type header — are you doing that on purpose? I haven't tried doing that since HTTP/0.9 days and I'm surprised it works (i.e., that modern browsers sniff the body and render it as HTML).
      - tempguy9999 7y ago
        I'm curious; could you explain/link to some explanation of Eric Naggum's views that showed why he disliked the web. Thanks.
tialaramex 7y ago
To be fair, most of those "extensible" formats aren't safely extensible. ASN.1 is because it uses OIDs, and some scenarios in XML and parts of HTTP are safely extensible because they use URIs, but anywhere which relies on just arbitrary text, like JSON, is just shrugging when it comes to global collisions. What I'm getting at here is, OIDs and URIs are globally namespaced. When I decide to add my own custom foo to a format, OIDs let me go get myself an arc and assign foo an OID from that arc, and URIs let me go get myself an FQDN and then assign foo a URL with that FQDN. In HTML if I just add a foo attribute that collides with everybody else's foo attribute, in RFC-822 mail my foo header can collide with anybody else's foo header. And so on.

Thorentis 7y ago

Interesting article, but lots of "probably the next biggest consumer" etc. Was hoping he would have some kind of network traffic analysis report in there that backs up his ranking of the formats. Seems to be anecdotal more than anything. Still good overview of the different formats available.

deleted 7y ago

[deleted]

AdamN 7y ago

To me the biggest annoyance of json is there's no standard way to put comments in there. What's the point of a text-based serialization format if it can't carry comments??? It has one advantage over YAML though, with JSON you know when you're not at the end of the file :-)

akersten 7y ago
The downside of this is that the comment becomes just another piece of the data model, because anything that consumes that JSON will need to decide: do I preserve comments, or remove them, when re-emitting the JSON? If I remove them, comments are kind of only useful for things like initial config files or what have you. Which might be fine for your use case, but JSON broadly is not about making configuration files - this is more suited for something like YAML. If you preserve them - how is that different from a key-value just named "comment": "Here's my comment"? (And if the answer is "because it might not crash poorly coded applications," that's more of a discussion about the application than it is about comments in JSON)
- vageli 7y ago
  Often times people add comments in-line with things to prevent the comment from becoming stale should the referent change (because the inlining encourages modifying the comment if the referent is modified), whereas it may be hard to determine the referent of a comment in its own field. Further, having multiple comments with the same field name is invalid JSON I believe, as keys must be unique? Though I may be wrong. (however, if keys _must_ be unique, then comment authors could encode the referent in the comment key, possibly).
- bmm6o 7y ago
  >If I remove them, comments are kind of only useful for things like initial config files or what have you.* True, but that's a pretty big use case to dismiss with "only". I don't think people want their json api responses to be able to include comments, but it's definitely convenient to be able to comment out parts of a config file. How/if to expose that through the deserializer is an interesting question, it would be fine with me if it dropped them.

jkaptur 7y ago

> ... schemas aren’t free. You have to coördinate between sender and receiver, and you have to worry what happens when someone wants to add a new field to a message type — but in raw JSON, you don’t have to worry, you just toss in the new field and things don’t break. Flexibility is a good thing. Much like type systems in general, you always have a schema, and you always have to worry about what happens when someone adds a field. You just get to choose whether it's explicit and you have to type it alllll the way out allllll the time, or whether it's implicit, and is defined as "whatever all the usages are everywhere". Just as gradual typing is taking over the world (TypeScript, MyPy, etc.), I predict success for something that lets you gradually add schemas to JSON.

vincnetas 7y ago
> ... you just toss in the new field and things don’t break. Flexibility is a good thing. Except when it's not you that are flexible but other parties, and they express that flexibility not only by adding but also removing or modifying fields without notice :/
- erik_seaberg 7y ago
  I very rarely see system that distinguish between "this new feature can be safely ignored" and "it's not safe to process this message if you don't support this new feature", but I've always thought it a good idea. Like semantic versioning without a global ordering over schema changes.

jolmg 7y ago

My browser opens up a download dialog on clicking this link. It's probably because the response headers don't include "Content-Type", and so it doesn't know if it will be able to interpret the response.

kragen 7y ago
Interesting, what browser?
- jolmg 7y ago
  Firefox on Archlinux. Though, now I think the issue is with an extension I have.
  - kragen 7y ago
    Weird, I'm not having that problem in Firefox on Debian. But it's sure enough true that there's no Content-Type header, either with LWP or with Firefox. Tim, are you doing that on purpose?
    - jolmg 7y ago
      I found it's my extension[1]. That's what's interpreting the lack of Content-Type as indication that Firefox is going to force a download (it actually doesn't) and showing me its own download dialog. [1] https://github.com/Rob--W/open-in-browser https://github.com/Rob--W/open-in-browser

commandersaki 7y ago

A quick note about protobuf. It's just way too complicated in my opinion. If you need a binary protocol just use TLVs (Tag-Length-Value) - so much less cognitive load and the parsing is easy to implement and read compared to the complicated autogenerated mess that is protobuf.

geraldbauer 7y ago

A quick note about json next. I collect new JSON variants, extensions and formats at the Awesome JSON Next page [1]. [1]: https://github.com/json-next/awesome-json-next https://github.com/json-next/awesome-json-next