3 ms·

JSON has drawbacks, but I think you skipped one of its biggest benefits: it encodes in reasonably readable plain text. Readable plain-text protocols remove a c

JSON has drawbacks, but I think you skipped one of its biggest benefits: it encodes in reasonably readable plain text. Readable plain-text protocols remove a constant overhead from all debugging/introspection tasks. If you use, say protobufs, you'll still have to convert back into strings to see what was in individual messages. This makes reading, for example, logfiles or wireshark captures with "grep" more difficult. Sure, there are tools that you can throw before "| grep" that convert to text, but that's one more piece of cognitive/debugging overhead you have to remember, and one more thing that new developers have to know to do if they're used to the standard unix/text-oriented way. That benefit applies even if your messages (or logs of messages) are stored/viewed in a database that speaks your binary protocol and provides tools to search/see the data as text. Whether or not you have something like that, it will always (always) be routinely necessary, or at least incredibly helpful to productivity, to "print out or search the textual content of messages sent/received by my code" when developing and testing apps that communicate over they network. A text protocol has a huge advantage in that department--or rather, it has a very small advantage that is present in a very large number of development/testing tasks. Also, readable plain text is key. XML dialects, or e.g. Thrift's JSON format are plain text, but are not terribly readable. This is subjective, to be sure, but makes a big difference in practice. There are great reasons to use binary protocols, but any comparison that leaves out the benefits of text content that is readable "for free" without using per-protocol tools is lacking.

ThrowawayR2 9y ago
> Readable plain-text protocols remove a constant overhead from all debugging/introspection tasks. As others have pointed out, text-based serialization has significant overhead. To save a few hours of debugging, you're willing to spend gigabytes of bandwidth, an uncountable number of CPU cycles and memory, and all the associated electricity and hosting costs over the lifetime of your service? Penny wise and pound foolish, to say the least. Text-based serialization formats are just plain stupid for any type of service that handles over 100 requests per second. A good developer is not hampered in the slightest by the fact that their serialization format isn't text.
- pjmlp 9y ago
  Yep, we just replaced XML with JSON, while keeping the same loss of performance.
- zbentley 9y ago
  Would I switch to a more computationally expensive format to save a few hours debugging? Probably not. Would I switch to a more computationally expensive format to save a few hours debugging each time someone makes substantial changes to my application's message format? Very possibly. Would I switch to a more computationally expensive format to prevent bugs (the easier it is to view, or remember how to view, what your code is sending/receiving, the less likely it is that lazy people will skip bug-hunting/QA steps that would show it)? Almost certainly. Should you use a binary format if you're sending tons of data per second? Well, it depends. It's like microservices: if you have the tooling to make dealing with that format from the dev/debug/tracing side a breeze, go for it; using binary provides substantial savings, as you point out. If you don't have the tooling/time to make it as easy as text to introspect, and I mean that in the most absolute way, there might still be persuasive reasons to switch, but know that if you do, you are also buying a lot of "debuggability debt".
foobarian 9y ago
So this comes up constantly and I tend to agree with it, but it has always bugged me. We are programmers and we control the machines, so why can't we code our way out of this problem? Why isn't the very first thing one does when working on a protobuf system installing a custom version of utils and tools that deals with the wire format better? E.g. grep, less and wireshark that autoconvert protobuf messages? (I know there are good reasons for this involving the effort required and maintenance etc. it just bugs me.)
- teddyh 9y ago
  Because Unix. Unix was designed as using ASCII text as a lowest common format, and all its tooling follows that design, and nothing has displaced this. If, for example, a Lisp system or a Smalltalk system had won out, I am sure we would all now be lamenting the inadequacies of s-expressions or Smalltalk objects instead of complaining about ASCII.
  - zbentley 9y ago
    While the "everything assumes unstructured text; destructure and restructure at will" paradigm is still incredibly powerful, and set the bar super high for data interoperability and uniformity, I think it lowered expectations for [meta]data complexity that could be supported by simple, small programs. In Windows, PowerShell is a very good example of a low-ish-level (composable commandline apps and snippets) programming environment that isn't shackled by that paradigm.
- skybrian 9y ago
  To do this right you'd need a common registry with all the proto schemas you'll ever want to use. It's a solvable problem, but it's comparable to setting up a new build ecosystem and getting people to use it. (Consider TypeScript and .d.ts files.)

ThrowawayR2 9y ago

> Readable plain-text protocols remove a constant overhead from all debugging/introspection tasks. As others have pointed out, text-based serialization has significant overhead. To save a few hours of debugging, you're willing to spend gigabytes of bandwidth, an uncountable number of CPU cycles and memory, and all the associated electricity and hosting costs over the lifetime of your service? Penny wise and pound foolish, to say the least. Text-based serialization formats are just plain stupid for any type of service that handles over 100 requests per second. A good developer is not hampered in the slightest by the fact that their serialization format isn't text.

pjmlp 9y ago
Yep, we just replaced XML with JSON, while keeping the same loss of performance.
zbentley 9y ago
Would I switch to a more computationally expensive format to save a few hours debugging? Probably not. Would I switch to a more computationally expensive format to save a few hours debugging each time someone makes substantial changes to my application's message format? Very possibly. Would I switch to a more computationally expensive format to prevent bugs (the easier it is to view, or remember how to view, what your code is sending/receiving, the less likely it is that lazy people will skip bug-hunting/QA steps that would show it)? Almost certainly. Should you use a binary format if you're sending tons of data per second? Well, it depends. It's like microservices: if you have the tooling to make dealing with that format from the dev/debug/tracing side a breeze, go for it; using binary provides substantial savings, as you point out. If you don't have the tooling/time to make it as easy as text to introspect, and I mean that in the most absolute way, there might still be persuasive reasons to switch, but know that if you do, you are also buying a lot of "debuggability debt".

foobarian 9y ago

So this comes up constantly and I tend to agree with it, but it has always bugged me. We are programmers and we control the machines, so why can't we code our way out of this problem? Why isn't the very first thing one does when working on a protobuf system installing a custom version of utils and tools that deals with the wire format better? E.g. grep, less and wireshark that autoconvert protobuf messages? (I know there are good reasons for this involving the effort required and maintenance etc. it just bugs me.)

teddyh 9y ago
Because Unix. Unix was designed as using ASCII text as a lowest common format, and all its tooling follows that design, and nothing has displaced this. If, for example, a Lisp system or a Smalltalk system had won out, I am sure we would all now be lamenting the inadequacies of s-expressions or Smalltalk objects instead of complaining about ASCII.
- zbentley 9y ago
  While the "everything assumes unstructured text; destructure and restructure at will" paradigm is still incredibly powerful, and set the bar super high for data interoperability and uniformity, I think it lowered expectations for [meta]data complexity that could be supported by simple, small programs. In Windows, PowerShell is a very good example of a low-ish-level (composable commandline apps and snippets) programming environment that isn't shackled by that paradigm.
skybrian 9y ago
To do this right you'd need a common registry with all the proto schemas you'll ever want to use. It's a solvable problem, but it's comparable to setting up a new build ecosystem and getting people to use it. (Consider TypeScript and .d.ts files.)