3 ms·

OK, we're going to see a lot of rehashing of the old arguments: Pro-text: Human-readability saves immeasurable human developer time! Pro-binary: Text parsing

by kentonv 11y ago

OK, we're going to see a lot of rehashing of the old arguments: Pro-text: Human-readability saves immeasurable human developer time! Pro-binary: Text parsing wastes a lot of very measurable machine time! Problem is, half of this argument is based entirely on anecdotal evidence and gut feelings. We really have no idea how much developer time is saved by having messages be human-readable. You will find smart people who believe very strongly both ways. In a seemingly very high fraction of these cases, it seems that people are really just taking the practice they are more comfortable with (because they've used it more) and rationalizing an argument for it because it makes them feel good about their choices. When hard evidence is lacking, confirmation bias, unfortunately, takes over. As the author of Cap'n Proto and former maintainer of Protobufs, I obviously come down pro-binary... but I won't bore you with my argument as I don't really have any hard facts either.

wmil 11y ago
You're missing a major feature. Text formats don't have any natural integer size limits. That's been a big win for HTTP. It's easy to imagine the original HTTP authors thinking that 24 bits is plenty of space for file size (anything bigger should use FTP) or using seconds since 1970 in 32 bits instead of text dates. Those are a bit obvious, but a developer could easily put in something like a 255 chunk limit without thinking too hard about it. Even John Carmack accidentally screwed over OpenGL driver developers by copying the GL_EXTENSIONS list into a 1000 char buffer in GL Quake (that's way too small). A binary format makes hard limits too easy.
- bdarnell 11y ago
  > You're missing a major feature. Text formats don't have any natural integer size limits. Sure they do. Where do you think Y2K came from? RFC 822 said "years are two digits" and people wrote parsers to that spec. Even now the official rule (AFAIK) is two or four digits, not an arbitrary number of digits. And even when the spec has no limit, implementations often intrude. For a more current example, JSON has no "natural" numeric size limits, but interoperability of integers greater than 2^53 is problematic because some implementations assume double-precision floating point while others can parse larger numbers into a 64-bit integer. Limits are a part of protocol design; a good protocol should make them clearly defined, suitably sized, and feasible to change.
- ThrustVectoring 11y ago
  Every format has limits, sometimes people just choose to not document them. For example, try transmitting 3↑↑↑3 as an integer over HTTP.
- dnr 11y ago
  If you use protocol buffers or similar libraries, you get variable-length integers (usually a 7 bits per byte encoding), which have no natural size limit. (You can go out of your way to get fixed-length integers in protocol buffers, but presumably you'd have a good reason because the default is variable-length.)
- kentonv 11y ago
  All good formats -- text or binary -- are capable of being extended over time in a backwards-compatible way, in order to fix mistakes. Formats that don't allow this are bad and should not be used. Protobufs and Cap'n Proto in particular have very strong support for forwards- and backwards-compatibility. Choosing a too-small fixed integer size is one way that a format can screw up and need correction -- one which happens to be (somewhat) exclusive to binary formats. But there are plenty of ways text formats can screw up too, like trying to pack fundamentally structured data into an ad-hoc hard-to-parse string format in order to make it look more pleasant to humans (I'm looking at you, MIME type specifications), or failing to use consistent parsing rules (cookie and user-agent headers totally flout otherwise-consistent HTTP grammar rules, confounding well-written parsers).
  - deleted 11y ago
    [deleted]
  - mcv 11y ago
    Is this an unbiased argument for binary formats in general, or is it a sales pitch for Protobufs and Cap'n Proto? If a fairly recent tool is the only way to get binary formats right, then it's entirely understandable that they've fallen out of use.
    - kentonv 11y ago
      Obviously I am biased, but my intent is to make a technical argument, not a sales pitch. I don't make any money from these (and I no longer have anything to do with protobufs). Protobufs was open sourced in 2008 (after being in use inside Google since around 2001). I suspect other binary formats prior to that had a solution for compatibility as well, but I haven't done a survey.
peteretep 11y ago
> You will find smart people who believe very strongly > both ways. In a seemingly very high fraction of these > cases, it seems that people are really just taking the > practice they are more comfortable with I think you've just solved the Internet, and every programming holy war ever. The irony of course being that tech folk hold themselves up as being particularly rational and immune to this kind of thing. It's a strong indicator that a problem is considerably less important than people intuitively feel it is. Likewise, a great way to tell that a news source is genuinely unbiased is both sides of an argument complaining of its bias. One of the few (only?) things to love about Go is that a large portion of dull holy war decisions have been premade for you. Generally (imho) wrongly, but who cares? They've been made, and we can get on with solving problems effectively.
bsder 11y ago
The problem is that binary format is good for ephemeral, structured data, but everybody starts using it for persistent, unstructured data where text format is far superior in terms of persistence. So, the problem is that most people wind up talking past one another in different domains. HTTP is unusual in that it is highly structured now in spite of having started life as highly unstructured. In addition, once people start using https more and more, we're going to have to use binary decoders on HTTP streams anyhow. So, binary http is really fait accompli, we might as well standardize it. However, I have a very strong aversion to binary formats for anything persistent that has a lifetime in the range of year+. I have had to grovel through FAR too many binary formats with a hex editor because the program that created them no longer runs because of OS, language, or computer upgrade. This is especially bad when closed-source programs are involved (dredging through SPICE circuit simulator data formats produced by FORTRAN programs of yesteryear is an exercise in pain).
- kentonv 11y ago
  I disagree. Binary formats with proper affordances for forwards- and backwards-compatibility (like Protobuf and Cap'n Proto) are great for persistence. Among many other points I could make, with large quantities of persistent data, random access is usually important, and text formats are woefully poor at random access. It's true that binary formats present more challenges to "digital archaeology" than text-based formats, but I don't think this is the most important thing to optimize for. Moreover, tools for reading Protobufs are not likely to go away any time soon -- not any more likely than tools for reading MySQL databases or your preferred compression format or your filesystem, all of which are also binary. (Cap'n Proto is perhaps not as deeply-rooted yet, but I'm working on that...) Of course, again, this is a judgment call and there's no way I can prove that my judgment is "correct".
cetra3 11y ago
How would you envisage Cap'n Proto or Protobufs working in a web application that has to talk to a number of 3rd party APIs and services, where you may not be provided with a .proto or .capnp file?
- peteretep 11y ago
  Have you seen Go Challenge 1? Like that: hex dumper, investigative work, some captured packets.
- kentonv 11y ago
  You need a .proto or .capnp file, just like with JSON you realistically need a spec and/or documentation. It just happens to be that with Protobuf and Cap'n Proto, the spec is in part machine-readable.
felix_g 11y ago
I think that text and binary can both be used together: Internally (between services) the format should be binary. It's important to have appropriate tooling support in order to be able to debug production service communications if necessary. For development purposes it would be nice to be able to switch to a text format (i.e. JSON) for debugging. Since JSON is a very convenient format for the web, it would be great to have a library that is compatible to both modes. I'm currently evaluating Flatbuffers vs. Cap'n Proto. I think that for my purpose they don't differ much. But Flatbuffers implements JSON reading/writing. I like that Cap'n Proto has forwards- and backwards-compatibility. So I would be interested in your plans on implementing the JSON codec support (mentioned on the Cap'n Proto Roadmap).

by kentonv 11y ago

wmil 11y ago

You're missing a major feature. Text formats don't have any natural integer size limits. That's been a big win for HTTP. It's easy to imagine the original HTTP authors thinking that 24 bits is plenty of space for file size (anything bigger should use FTP) or using seconds since 1970 in 32 bits instead of text dates. Those are a bit obvious, but a developer could easily put in something like a 255 chunk limit without thinking too hard about it. Even John Carmack accidentally screwed over OpenGL driver developers by copying the GL_EXTENSIONS list into a 1000 char buffer in GL Quake (that's way too small). A binary format makes hard limits too easy.

bdarnell 11y ago
> You're missing a major feature. Text formats don't have any natural integer size limits. Sure they do. Where do you think Y2K came from? RFC 822 said "years are two digits" and people wrote parsers to that spec. Even now the official rule (AFAIK) is two or four digits, not an arbitrary number of digits. And even when the spec has no limit, implementations often intrude. For a more current example, JSON has no "natural" numeric size limits, but interoperability of integers greater than 2^53 is problematic because some implementations assume double-precision floating point while others can parse larger numbers into a 64-bit integer. Limits are a part of protocol design; a good protocol should make them clearly defined, suitably sized, and feasible to change.
ThrustVectoring 11y ago
Every format has limits, sometimes people just choose to not document them. For example, try transmitting 3↑↑↑3 as an integer over HTTP.
dnr 11y ago
If you use protocol buffers or similar libraries, you get variable-length integers (usually a 7 bits per byte encoding), which have no natural size limit. (You can go out of your way to get fixed-length integers in protocol buffers, but presumably you'd have a good reason because the default is variable-length.)
kentonv 11y ago
All good formats -- text or binary -- are capable of being extended over time in a backwards-compatible way, in order to fix mistakes. Formats that don't allow this are bad and should not be used. Protobufs and Cap'n Proto in particular have very strong support for forwards- and backwards-compatibility. Choosing a too-small fixed integer size is one way that a format can screw up and need correction -- one which happens to be (somewhat) exclusive to binary formats. But there are plenty of ways text formats can screw up too, like trying to pack fundamentally structured data into an ad-hoc hard-to-parse string format in order to make it look more pleasant to humans (I'm looking at you, MIME type specifications), or failing to use consistent parsing rules (cookie and user-agent headers totally flout otherwise-consistent HTTP grammar rules, confounding well-written parsers).
- deleted 11y ago
  [deleted]
- mcv 11y ago
  Is this an unbiased argument for binary formats in general, or is it a sales pitch for Protobufs and Cap'n Proto? If a fairly recent tool is the only way to get binary formats right, then it's entirely understandable that they've fallen out of use.
  - kentonv 11y ago
    Obviously I am biased, but my intent is to make a technical argument, not a sales pitch. I don't make any money from these (and I no longer have anything to do with protobufs). Protobufs was open sourced in 2008 (after being in use inside Google since around 2001). I suspect other binary formats prior to that had a solution for compatibility as well, but I haven't done a survey.

peteretep 11y ago

> You will find smart people who believe very strongly > both ways. In a seemingly very high fraction of these > cases, it seems that people are really just taking the > practice they are more comfortable with I think you've just solved the Internet, and every programming holy war ever. The irony of course being that tech folk hold themselves up as being particularly rational and immune to this kind of thing. It's a strong indicator that a problem is considerably less important than people intuitively feel it is. Likewise, a great way to tell that a news source is genuinely unbiased is both sides of an argument complaining of its bias. One of the few (only?) things to love about Go is that a large portion of dull holy war decisions have been premade for you. Generally (imho) wrongly, but who cares? They've been made, and we can get on with solving problems effectively.

bsder 11y ago

The problem is that binary format is good for ephemeral, structured data, but everybody starts using it for persistent, unstructured data where text format is far superior in terms of persistence. So, the problem is that most people wind up talking past one another in different domains. HTTP is unusual in that it is highly structured now in spite of having started life as highly unstructured. In addition, once people start using https more and more, we're going to have to use binary decoders on HTTP streams anyhow. So, binary http is really fait accompli, we might as well standardize it. However, I have a very strong aversion to binary formats for anything persistent that has a lifetime in the range of year+. I have had to grovel through FAR too many binary formats with a hex editor because the program that created them no longer runs because of OS, language, or computer upgrade. This is especially bad when closed-source programs are involved (dredging through SPICE circuit simulator data formats produced by FORTRAN programs of yesteryear is an exercise in pain).

kentonv 11y ago
I disagree. Binary formats with proper affordances for forwards- and backwards-compatibility (like Protobuf and Cap'n Proto) are great for persistence. Among many other points I could make, with large quantities of persistent data, random access is usually important, and text formats are woefully poor at random access. It's true that binary formats present more challenges to "digital archaeology" than text-based formats, but I don't think this is the most important thing to optimize for. Moreover, tools for reading Protobufs are not likely to go away any time soon -- not any more likely than tools for reading MySQL databases or your preferred compression format or your filesystem, all of which are also binary. (Cap'n Proto is perhaps not as deeply-rooted yet, but I'm working on that...) Of course, again, this is a judgment call and there's no way I can prove that my judgment is "correct".

cetra3 11y ago

How would you envisage Cap'n Proto or Protobufs working in a web application that has to talk to a number of 3rd party APIs and services, where you may not be provided with a .proto or .capnp file?

peteretep 11y ago
Have you seen Go Challenge 1? Like that: hex dumper, investigative work, some captured packets.
kentonv 11y ago
You need a .proto or .capnp file, just like with JSON you realistically need a spec and/or documentation. It just happens to be that with Protobuf and Cap'n Proto, the spec is in part machine-readable.

felix_g 11y ago

I think that text and binary can both be used together: Internally (between services) the format should be binary. It's important to have appropriate tooling support in order to be able to debug production service communications if necessary. For development purposes it would be nice to be able to switch to a text format (i.e. JSON) for debugging. Since JSON is a very convenient format for the web, it would be great to have a library that is compatible to both modes. I'm currently evaluating Flatbuffers vs. Cap'n Proto. I think that for my purpose they don't differ much. But Flatbuffers implements JSON reading/writing. I like that Cap'n Proto has forwards- and backwards-compatibility. So I would be interested in your plans on implementing the JSON codec support (mentioned on the Cap'n Proto Roadmap).