4 ms·

Designing Network Protocols

66 points by tomazmuraus 15y ago

jacques_chester 15y ago
I think the short version is: in the past 5 years, simple wire-oriented serialisation formats have become much more common. At the time you had your pick of ASN (humongous) or Thrift (brand new).
- fabricode 15y ago
  I don't believe a tl;dr is necessary for this short article. I enjoyed reading through his thought process for designing a simple protocol which is: * Easy to use (requiring little/no additional libraries) * Easy to extend (simple keyword/value extensions) * Immune to changes in technology * (above all) easy to understand
  - jacques_chester 15y ago
    Yet he himself says he considered and rejected a binary protocol for the reason I gave: "I considered using a binary format, but the immediate problem was having extendable fields.", going on to point out that he rejected Thrift because it was too new and ASN.DER because it was too big. That said, I think he didn't want a binary format in any case -- his "doing it again today" remarks point to JSON.
- groby_b 15y ago
  Actually, the short version is: I wanted it easily debuggable by a network admin. (It's a point he made repeatedly). All the other arguments were moot.
  - huhtenberg 15y ago
    So the network protocol must be in text so that a network admin could debug it? This is absurd.
    - groby_b 15y ago
      Not as absurd as you'd think. You don't want to debug the protocol itself, but you want to be able to easily read what messages were exchanged. I get the rationale. But I think it's weak, and this entire post is lots of fluff around that core rationale. (I've been writing extensible binary protocols back in 1988 - and it never struck me as particularly difficult even back then.)
    - tptacek 15y ago
      How does a network admin debug a binary protocol for which no dissector has been implemented/merged into core for Wireshark, and no decoder has been written for tcpdump? It's obviously doable, but it's very painful.
      - jacques_chester 15y ago
        On the other hand, are Wireshark and tcpdump now the gatekeepers for new protocols?
        tptacek 15y ago
        What's your point? I'm not making a value judgement.
        jacques_chester 15y ago
        You say that to me a lot. My point is that I imagine a network designer shouldn't focus on Wireshark or tcpdump integration over other non-functional requirements such as, well, network performance. Network performance isn't as visible as the non-functional requirement of inspectability because it is amortised over potentially millions of machines, whereas inspectability is an immediately visible issue to the select few who "pop the hood" to fix an issue or simply to have a look. For example: in terms of network capacity, I wonder how much HTTP headers cost all of us collectively. Probably a lot more than the cost of making a Wireshark plugin and having sysadmins install it as necessary. Edit: put another way, I think designers should prioritise the needs of the people who pay the cost of network operation over the convenience of the operators. There's a feedback loop here -- if it's too hard and thus very expensive to operate a system, then optimising for performance was a false win. But I don't think this is such a case, especially since as you pointed out elsewhere there are a number of very mature binary wire formats that were extant in 2007.
- tptacek 15y ago
  Not particularly fair. You also had XDR (ubiquitous on Unix systems), IIOP, TLV, ICE... not to mention what ever protocol designer for the past 20 years has used: network byte order integers and ASCII/UTF8 strings. Some people just like ASCII, human readable protocols. There's nothing wrong with that, but it's a little silly to suggest that the options for a packed binary encoding in 2007 were limited because Thrift and Protocol Buffers were too new.
  - jacques_chester 15y ago
    You're right, to my shame. I should've thought of XDR myself, after suffering through a lecture on it at uni. I'd be interested in the original designer's remarks on using existing wire serialisation formats.
  - pquerna 15y ago
    I mentioned ASN.1 DER, but I honestly didn't think I should go into a history of XDR or other encodings. I guess I can't skip any history in a blog post....
    - tptacek 15y ago
      I was responding to a comment, not your post. I don't think you really need to justify using an ASCII protocol (though, again, I think HTTP query arguments are a poor choice).
trout 15y ago
As someone who troubleshoots networks for living, it can't be overstated what value easily understood information has. Normal factors such as lack of understanding of a system, miscommunication, false assumptions, false information, all types of bugs, and operator error already plague troubleshooting. Anything you can do to simplify is extremely important. Little things like being able to read debugs directly out of wireshark can make the difference between solving something between minutes and hours, days and weeks. Unfortunately serviceability is not normally high in the initial product requirements, but I've seen a direct correlation between customer satisfaction in support and products/protocols/designs with good serviceability.
tptacek 15y ago
HTTP-style query strings are a horrible format, whether you like ASCII or not.
alexchamberlain 15y ago
I would accept the ease of debugging argument if these messages weren't so small and so common. 1000 servers constructing strings and sending them over the network once a second is a nontrivial waste of resources.
alfiejohn_ 15y ago
There's a complete chapter in The Art of Unx Programming about the importance of being textual: http://catb.org/~esr/writings/taoup/html/textualitychapter.html

66 points by tomazmuraus 15y ago

I think the short version is: in the past 5 years, simple wire-oriented serialisation formats have become much more common. At the time you had your pick of ASN (humongous) or Thrift (brand new).

fabricode 15y ago
I don't believe a tl;dr is necessary for this short article. I enjoyed reading through his thought process for designing a simple protocol which is: * Easy to use (requiring little/no additional libraries) * Easy to extend (simple keyword/value extensions) * Immune to changes in technology * (above all) easy to understand
- jacques_chester 15y ago
  Yet he himself says he considered and rejected a binary protocol for the reason I gave: "I considered using a binary format, but the immediate problem was having extendable fields.", going on to point out that he rejected Thrift because it was too new and ASN.DER because it was too big. That said, I think he didn't want a binary format in any case -- his "doing it again today" remarks point to JSON.
groby_b 15y ago
Actually, the short version is: I wanted it easily debuggable by a network admin. (It's a point he made repeatedly). All the other arguments were moot.
- huhtenberg 15y ago
  So the network protocol must be in text so that a network admin could debug it? This is absurd.
  - groby_b 15y ago
    Not as absurd as you'd think. You don't want to debug the protocol itself, but you want to be able to easily read what messages were exchanged. I get the rationale. But I think it's weak, and this entire post is lots of fluff around that core rationale. (I've been writing extensible binary protocols back in 1988 - and it never struck me as particularly difficult even back then.)
  - tptacek 15y ago
    How does a network admin debug a binary protocol for which no dissector has been implemented/merged into core for Wireshark, and no decoder has been written for tcpdump? It's obviously doable, but it's very painful.
    - jacques_chester 15y ago
      On the other hand, are Wireshark and tcpdump now the gatekeepers for new protocols?
      - tptacek 15y ago
        What's your point? I'm not making a value judgement.
        jacques_chester 15y ago
        You say that to me a lot. My point is that I imagine a network designer shouldn't focus on Wireshark or tcpdump integration over other non-functional requirements such as, well, network performance. Network performance isn't as visible as the non-functional requirement of inspectability because it is amortised over potentially millions of machines, whereas inspectability is an immediately visible issue to the select few who "pop the hood" to fix an issue or simply to have a look. For example: in terms of network capacity, I wonder how much HTTP headers cost all of us collectively. Probably a lot more than the cost of making a Wireshark plugin and having sysadmins install it as necessary. Edit: put another way, I think designers should prioritise the needs of the people who pay the cost of network operation over the convenience of the operators. There's a feedback loop here -- if it's too hard and thus very expensive to operate a system, then optimising for performance was a false win. But I don't think this is such a case, especially since as you pointed out elsewhere there are a number of very mature binary wire formats that were extant in 2007.
tptacek 15y ago
Not particularly fair. You also had XDR (ubiquitous on Unix systems), IIOP, TLV, ICE... not to mention what ever protocol designer for the past 20 years has used: network byte order integers and ASCII/UTF8 strings. Some people just like ASCII, human readable protocols. There's nothing wrong with that, but it's a little silly to suggest that the options for a packed binary encoding in 2007 were limited because Thrift and Protocol Buffers were too new.
- jacques_chester 15y ago
  You're right, to my shame. I should've thought of XDR myself, after suffering through a lecture on it at uni. I'd be interested in the original designer's remarks on using existing wire serialisation formats.
- pquerna 15y ago
  I mentioned ASN.1 DER, but I honestly didn't think I should go into a history of XDR or other encodings. I guess I can't skip any history in a blog post....
  - tptacek 15y ago
    I was responding to a comment, not your post. I don't think you really need to justify using an ASCII protocol (though, again, I think HTTP query arguments are a poor choice).

trout 15y ago

As someone who troubleshoots networks for living, it can't be overstated what value easily understood information has. Normal factors such as lack of understanding of a system, miscommunication, false assumptions, false information, all types of bugs, and operator error already plague troubleshooting. Anything you can do to simplify is extremely important. Little things like being able to read debugs directly out of wireshark can make the difference between solving something between minutes and hours, days and weeks. Unfortunately serviceability is not normally high in the initial product requirements, but I've seen a direct correlation between customer satisfaction in support and products/protocols/designs with good serviceability.

tptacek 15y ago

HTTP-style query strings are a horrible format, whether you like ASCII or not.

alexchamberlain 15y ago

I would accept the ease of debugging argument if these messages weren't so small and so common. 1000 servers constructing strings and sending them over the network once a second is a nontrivial waste of resources.

alfiejohn_ 15y ago

There's a complete chapter in The Art of Unx Programming about the importance of being textual: http://catb.org/~esr/writings/taoup/html/textualitychapter.html