5 ms·

A case against text protocols

24 points by unmdplyr 5y ago

gary_0 5y ago
Back when HTTP and SMTP were designed, the Internet was mostly old Unix machines talking to each other. Everything was a file full of `char`, piped to an 80-column terminal. Text-based made sense. Decades later, when the computer world was bigger, faster, and more unified, other systems kind of cargo-culted off of those earlier successes. And isn't it neat that you can telnet into port 80? I think another big reason text-based protocols are seductive is that they're an engineering path of least resistance. When you start off text-based, how to debug and analyze and interoperate with other implementations can be put off for later, or be Someone Else's Problem. Whereas if you design the same protocol but in binary, these tricky considerations are harder to ignore -- even though text-based protocols will still run into the same problems eventually, because there's no way I'm typing in a Cookie header by hand or decoding Base64 in my head.
- phyrex 5y ago
  And yet I have definitely copy&pasted cookie headers from the database or log files into the browser - something I’ve never done it could imagine doing with binary protocols
  - IshKebab 5y ago
    You could use something like Wireshark to do it with binary protocols. Also lots of binary protocols/formats support a text format to make them readable, e.g. Wasm and Capnp
- swiley 5y ago
  >internet was mostly old Unix machines talking to each other. Was it? By my reading there was way more weird stuff than there is today but I certainly wasn't alive back then.
  - gary_0 5y ago
    In the context of popular things like HTTP and SMTP, yes, it was mostly Unix/BSD/NeXT[0][1]. Obviously the entire history of the Internet is a different story. [0] https://en.wikipedia.org/wiki/World_Wide_Web#/media/File:NeXTcube_first_webserver.JPG https://en.wikipedia.org/wiki/World_Wide_Web#/media/File:NeX... [1] https://en.wikipedia.org/wiki/Simple_Mail_Transfer_Protocol#Original_SMTP https://en.wikipedia.org/wiki/Simple_Mail_Transfer_Protocol#...
- deleted 5y ago
  [deleted]
- deleted 5y ago
  [deleted]
- deleted 5y ago
  [deleted]
pwdisswordfish8 5y ago
Another argument: text-based protocols often admit too many degrees of freedom in constructing messages, the handling of which is left underspecified and completely overlooked during implementation. (What happens if you separate lines with LF instead of CRLF in HTTP headers? What if the opening and closing HTML tags don't match? I know this should not usually happen, but how should I handle it when it does anyway?) It's not by any means exclusive to text-based protocols, but there's this tendency to assume everything about a text-based protocols is ‘obvious’, ‘self-documenting’ and doesn't need specifying, and to think that just because the individual elements of the protocol are human-readable, this will somehow magically make the computers using the protocol follow the Gricean maxims (if it doesn't make sense, nobody will ever say that, therefore I don't need to think about it).
- gary_0 5y ago
  > the handling of which is left underspecified I used to see Postel's Law ("be conservative in what you send, be liberal in what you accept") quoted as some sort of antidote, but it seems to have fallen out of fashion -- I think enough people saw how that ideal played out in reality. Nowadays a JSON library feels justified throwing a fit if it sees a comment string instead of playing along with such shenanigans. > It's not by any means exclusive to text-based protocols Plus, I would argue text-based greatly increases the surface area for ambiguity, whereas, for instance, there are only a few ways a sane person would send an integer as bytes.
  - mananaysiempre 5y ago
    I’d say it’s played out ... ambiguously. Conservative HTTP unworkable, liberal TLS dangerous. JSON for internal APIs and data succeeded because it’s conservative, XHTML and XML+XSLT on the open web failed for the same reason. Postel’s law is less of a universal principle than it initially seemed to be, sure, but it appears to me that part of the reason for its increasing irrelevance is our moving away from open ecosystems and not deficiencies valid in its original context. Integer encoding (as opposed to e.g. encoding of opaque binary strings) actually appears to be a bad example to me: various universal binary encoding protocols, self-describing or not, have an astounding number of unsigned and signed integer encodings among them. It’s like inventing a new one is a rite of passage or something.
Joker_vD 5y ago
> It is as simple as dumping struct ntp_packet on wire and reading it off it -- no parsing involved except for calling ntohX()/htonX() on all fields except li, vn and mode. Nope, you may still need to call ntoh/hton, depending on how the compiler you use orders the bitfields inside an int. Plus you need "__attribute__((packed))" or whatever the compiler you use supports to make that C struct definition mean what it looks like it means: even then I am not sure those three bitfields are required to occupy exactly 8 bits.
- pwdisswordfish8 5y ago
  Better yet is using wrappers for char arrays paired with helper functions: typedef struct { unsigned char data[2]; } u16be_t; inline static uint16_t get_u16be(const u16be_t *p) { uint16_t result = 0; result |= (uint16_t) p->data[0] << 8; result |= (uint16_t) p->data[1]; return result; } inline static void put_u16be(u16be_t *p, uint16_t value) { p->data[0] = value >> 8; p->data[1] = value; } Perfectly portable (as long as CHAR_BIT = 8), type-safe, and on modern compilers it generates no overhead over direct memory accesses.
zajio1am 5y ago
There is one thing that is often neglected in text vs binary protocols debate and that is self-terminating vs prior-length. Although it is not strictly connected, text protocols are usually self-terminating (e.g. closing tags), while binary protocols are usually prior-length (e.g. type-length-value approach). The first approach leads to escaping and all associated problems.
horsawlarway 5y ago
I don't really agree... with any of this, actually. It IS simpler to use text - He claims "Text was never as portable as it's believed to be.", but ascii/unicode are probably the most portable formats we've ever created. I can't think of a computer that won't be able to parse and display one of those two formats (From embedded hardware, to old f16 parts, to my modern laptop, to the raspberry pi, to the fucking computer I designed in my EE classes). Being able to type out messages is hugely helpful while debugging and developing (I copy and paste things that look exactly like the code he claims no one would ever write - It's like he doesn't understand the value of a clipboard, or a text editor I can dump a message into and change a single value in - Something I can conveniently do on pretty much any system ANYWHERE without having to install any extra software if the format is text) His parsing example is hilarious - See that readable text above? Psh, Folly! That's hard to read so lets use specialized tools that depend entirely on system specific details and configuration (Int size, byte order, struct packing, etc) and claim that's better! Extensibility, meh - I find this one rarely matters as much as people believe it does, but to me, the big benefit of text is that I can easily craft messages with new fields myself without having to write code to do it. Error recovery... I can sort of agree (in transit over a noisy channel, use a format that supports ECCs) but he misses that there are two different types of error here - An unexpected field value/type, and a generally malformed payload. The first will break binary but not something like a json parser. The second will break both (he only talks about the second, since he assumed the failure happens at tokenization time...) Basically - My whole point devolves into "It sure seems like he's arguing for premature optimization". If you have a spot where text is particularly expensive or inefficient, suck it up and move to a binary protocol that requires more documentation, tooling, and work. Everywhere else... it seems like a bad move.
- arpyzo 5y ago
  I've taken advantage of text protocols countless (hundreds? thousands?) of times in my career to troubleshoot, learn, and experiment. Just a few weeks ago when I needed to peek into some StatsD packets to ensure we were sending what we expected when monitoring wasn't working. If it was a binary format this simply would not have been an option as this was a remote environment with limited tooling available to it.
- 5y ago
IshKebab 5y ago
This is all 100% correct but he missed probably the biggest reason! It's really really hard to write an unambiguous text protocol specification and equally hard to write something that implements it properly. Think about all the extra ambiguities text adds: where is whitespace allowed? How are newlines encoded? Are windows line endings ok? Does case matter? Which bits are ASCII and which are UTF-8? How are values quoted? It's just insanely more complicated, and there are many subtle differences that seem reasonable to different people. "Of course whitespace is allowed there!" It makes everything way less robust and way more prone to quirks-mode style degradation.
- Joker_vD 5y ago
  > How are newlines encoded? Are windows line endings ok? sigh Those are not "Windows line endings". CRLF has been used as a line-delimiter in every single ASCII-based network protocol since 1971 to this day. UNIX is not the single progenitor of every thing in the modern computing, not by a long shot.

24 points by unmdplyr 5y ago

Back when HTTP and SMTP were designed, the Internet was mostly old Unix machines talking to each other. Everything was a file full of `char`, piped to an 80-column terminal. Text-based made sense. Decades later, when the computer world was bigger, faster, and more unified, other systems kind of cargo-culted off of those earlier successes. And isn't it neat that you can telnet into port 80? I think another big reason text-based protocols are seductive is that they're an engineering path of least resistance. When you start off text-based, how to debug and analyze and interoperate with other implementations can be put off for later, or be Someone Else's Problem. Whereas if you design the same protocol but in binary, these tricky considerations are harder to ignore -- even though text-based protocols will still run into the same problems eventually, because there's no way I'm typing in a Cookie header by hand or decoding Base64 in my head.

phyrex 5y ago
And yet I have definitely copy&pasted cookie headers from the database or log files into the browser - something I’ve never done it could imagine doing with binary protocols
- IshKebab 5y ago
  You could use something like Wireshark to do it with binary protocols. Also lots of binary protocols/formats support a text format to make them readable, e.g. Wasm and Capnp
swiley 5y ago
>internet was mostly old Unix machines talking to each other. Was it? By my reading there was way more weird stuff than there is today but I certainly wasn't alive back then.
- gary_0 5y ago
  In the context of popular things like HTTP and SMTP, yes, it was mostly Unix/BSD/NeXT[0][1]. Obviously the entire history of the Internet is a different story. [0] https://en.wikipedia.org/wiki/World_Wide_Web#/media/File:NeXTcube_first_webserver.JPG https://en.wikipedia.org/wiki/World_Wide_Web#/media/File:NeX... [1] https://en.wikipedia.org/wiki/Simple_Mail_Transfer_Protocol#Original_SMTP https://en.wikipedia.org/wiki/Simple_Mail_Transfer_Protocol#...
deleted 5y ago
[deleted]
deleted 5y ago
[deleted]
deleted 5y ago
[deleted]

pwdisswordfish8 5y ago

Another argument: text-based protocols often admit too many degrees of freedom in constructing messages, the handling of which is left underspecified and completely overlooked during implementation. (What happens if you separate lines with LF instead of CRLF in HTTP headers? What if the opening and closing HTML tags don't match? I know this should not usually happen, but how should I handle it when it does anyway?) It's not by any means exclusive to text-based protocols, but there's this tendency to assume everything about a text-based protocols is ‘obvious’, ‘self-documenting’ and doesn't need specifying, and to think that just because the individual elements of the protocol are human-readable, this will somehow magically make the computers using the protocol follow the Gricean maxims (if it doesn't make sense, nobody will ever say that, therefore I don't need to think about it).

gary_0 5y ago
> the handling of which is left underspecified I used to see Postel's Law ("be conservative in what you send, be liberal in what you accept") quoted as some sort of antidote, but it seems to have fallen out of fashion -- I think enough people saw how that ideal played out in reality. Nowadays a JSON library feels justified throwing a fit if it sees a comment string instead of playing along with such shenanigans. > It's not by any means exclusive to text-based protocols Plus, I would argue text-based greatly increases the surface area for ambiguity, whereas, for instance, there are only a few ways a sane person would send an integer as bytes.
- mananaysiempre 5y ago
  I’d say it’s played out ... ambiguously. Conservative HTTP unworkable, liberal TLS dangerous. JSON for internal APIs and data succeeded because it’s conservative, XHTML and XML+XSLT on the open web failed for the same reason. Postel’s law is less of a universal principle than it initially seemed to be, sure, but it appears to me that part of the reason for its increasing irrelevance is our moving away from open ecosystems and not deficiencies valid in its original context. Integer encoding (as opposed to e.g. encoding of opaque binary strings) actually appears to be a bad example to me: various universal binary encoding protocols, self-describing or not, have an astounding number of unsigned and signed integer encodings among them. It’s like inventing a new one is a rite of passage or something.

Joker_vD 5y ago

> It is as simple as dumping struct ntp_packet on wire and reading it off it -- no parsing involved except for calling ntohX()/htonX() on all fields except li, vn and mode. Nope, you may still need to call ntoh/hton, depending on how the compiler you use orders the bitfields inside an int. Plus you need "__attribute__((packed))" or whatever the compiler you use supports to make that C struct definition mean what it looks like it means: even then I am not sure those three bitfields are required to occupy exactly 8 bits.

pwdisswordfish8 5y ago
Better yet is using wrappers for char arrays paired with helper functions: typedef struct { unsigned char data[2]; } u16be_t; inline static uint16_t get_u16be(const u16be_t *p) { uint16_t result = 0; result |= (uint16_t) p->data[0] << 8; result |= (uint16_t) p->data[1]; return result; } inline static void put_u16be(u16be_t *p, uint16_t value) { p->data[0] = value >> 8; p->data[1] = value; } Perfectly portable (as long as CHAR_BIT = 8), type-safe, and on modern compilers it generates no overhead over direct memory accesses.

zajio1am 5y ago

There is one thing that is often neglected in text vs binary protocols debate and that is self-terminating vs prior-length. Although it is not strictly connected, text protocols are usually self-terminating (e.g. closing tags), while binary protocols are usually prior-length (e.g. type-length-value approach). The first approach leads to escaping and all associated problems.

horsawlarway 5y ago

I don't really agree... with any of this, actually. It IS simpler to use text - He claims "Text was never as portable as it's believed to be.", but ascii/unicode are probably the most portable formats we've ever created. I can't think of a computer that won't be able to parse and display one of those two formats (From embedded hardware, to old f16 parts, to my modern laptop, to the raspberry pi, to the fucking computer I designed in my EE classes). Being able to type out messages is hugely helpful while debugging and developing (I copy and paste things that look exactly like the code he claims no one would ever write - It's like he doesn't understand the value of a clipboard, or a text editor I can dump a message into and change a single value in - Something I can conveniently do on pretty much any system ANYWHERE without having to install any extra software if the format is text) His parsing example is hilarious - See that readable text above? Psh, Folly! That's hard to read so lets use specialized tools that depend entirely on system specific details and configuration (Int size, byte order, struct packing, etc) and claim that's better! Extensibility, meh - I find this one rarely matters as much as people believe it does, but to me, the big benefit of text is that I can easily craft messages with new fields myself without having to write code to do it. Error recovery... I can sort of agree (in transit over a noisy channel, use a format that supports ECCs) but he misses that there are two different types of error here - An unexpected field value/type, and a generally malformed payload. The first will break binary but not something like a json parser. The second will break both (he only talks about the second, since he assumed the failure happens at tokenization time...) Basically - My whole point devolves into "It sure seems like he's arguing for premature optimization". If you have a spot where text is particularly expensive or inefficient, suck it up and move to a binary protocol that requires more documentation, tooling, and work. Everywhere else... it seems like a bad move.

arpyzo 5y ago
I've taken advantage of text protocols countless (hundreds? thousands?) of times in my career to troubleshoot, learn, and experiment. Just a few weeks ago when I needed to peek into some StatsD packets to ensure we were sending what we expected when monitoring wasn't working. If it was a binary format this simply would not have been an option as this was a remote environment with limited tooling available to it.
5y ago

IshKebab 5y ago

This is all 100% correct but he missed probably the biggest reason! It's really really hard to write an unambiguous text protocol specification and equally hard to write something that implements it properly. Think about all the extra ambiguities text adds: where is whitespace allowed? How are newlines encoded? Are windows line endings ok? Does case matter? Which bits are ASCII and which are UTF-8? How are values quoted? It's just insanely more complicated, and there are many subtle differences that seem reasonable to different people. "Of course whitespace is allowed there!" It makes everything way less robust and way more prone to quirks-mode style degradation.

Joker_vD 5y ago
> How are newlines encoded? Are windows line endings ok? sigh Those are not "Windows line endings". CRLF has been used as a line-delimiter in every single ASCII-based network protocol since 1971 to this day. UNIX is not the single progenitor of every thing in the modern computing, not by a long shot.