7 ms·

Network protocols, sans I/O

123 points by kawera 10y ago

dividuum 10y ago
I used that approach for my minecraft protocol parsing library (https://github.com/dividuum/fastmc https://github.com/dividuum/fastmc). It's indeed very useful to decouple parsing a protocol from the underlying transport. I used my library both for socket communication as well as to read and write files. I'd love to see more libraries implemented that way. So thanks for posting that link. I imagine it might be difficult sometimes depending on the protocol. IIRC TLS might sometimes require writes while reading.
- Lukasa 10y ago
  It turns out TLS isn't that bad. OpenSSL exposes a "memory BIO", which is basically exactly this. You send in writes, and then emit data when you can, and in the meantime it buffers. It's a bit more limited than that, but generally speaking it's not too bad.
  - JoshTriplett 10y ago
    I assume you handle decompression the same way? Feed in data, and the decompressor emits data when it can?
- oxymoron 10y ago
  On the topic of IO. I'm actually a fan of java.io.Reader/Writer and java.io.InputStream/OutputStream. It seems to me that most java protocols and parsers are quite composable by default due to these simple little abstractions. Granted, they might not be a good fit for async IO, but I'd still argue that they've held up quite well during the past fifteen years. I'll go out on a limb and claim that standard C++ IO haven't worked out quite as well, although it might just be my skip-the-oo-and-template-everything skepticism. Rust seems closer to java, although I feel like the Jury's still out. I'm not a enthusiastic of doing everything asynchronously as in Node. Haskell does seem to get the composability right. Python has some of javas thinking through a standard convention on the method names for reading and writing (an interface!), but it seems common with departures from those simple rules when people get creative. From that perspective, an initiative like this is quite understandable and convenient. Nevertheless, I don't seem to feel the need in the same way in certain other languages.
  - paulddraper 10y ago
    A little off-topic but IMO one of the more unfortunate decisions by Java was Reader/Writer and InputStream/OutputStream, instead of CharReader/CharWriter and ByteReader/ByteWriter (or similar).
wtbob 10y ago
From the description, it sounds like the necessity is the lack of a single way to do I/O in Python. If that's right, then this shows the real advantage of Go's io.Reader/Writer/&c. interfaces, which enable this sort of composability. From my Python years, I'd think that something built around mandatory read()/write() methods could do much the same.
- bdarnell 10y ago
  Python has a standard interface just like Go's `io.{Read,Writ}er`: the `send` and `recv` methods as defined by `socket.socket`. The appeal of "sans I/O" protocol implementations is primarily for asynchronous I/O, which requires a completely different style of interface. The clever thing about Go is not the standardization of a reader/writer interface, but the fact that the goroutine I/O scheduler makes these synchronous interfaces nearly as efficient as asynchronous ones, so there is no need to introduce a separate family of asynchronous interfaces.
  - jerf 10y ago
    "Python has a standard interface just like Go's `io.{Read,Writ}er`: the `send` and `recv` methods as defined by `socket.socket`." But a File has no "send" and "recv". io.Reader and Writer apply to files, sockets, byte slices, strings, compositions of other Readers or Writers, HTTP request and response bodies, and anything else that strikes my fancy to implement the correct methods. Sockets appear to deliberately not have those methods, as there's this line in the docs: "Note that there are no methods read() or write(); use recv() and send() without flags argument instead." There's nothing preventing Python from having the equivalent of Reader and Writer. It just doesn't, at least not with the standard library. In fact that's true of a lot of languages; there's nothing preventing that interface from existing, it just doesn't. This is a point in favor of getting the standard library right earlier rather than a point in favor of Go-the-language.
    - solidsnack9000 10y ago
      How would a standard interface like this support async IO?
JoshTriplett 10y ago
This nicely documents the "why", as does the linked PyCon talk. But both don't go into much detail on the "how", making it difficult to follow suit.
- Lukasa 10y ago
  Hi, I'm the linked PyCon speaker! It'd be good to know what additional information you'd like on the "how". I think at a high level I addressed that in my talk, but if it didn't make it across or you needed more information I'd love to know what you need. Ideally I'll turn this into a blog post at some point so it'd be great to have an idea of what extra info is needed.
  - JoshTriplett 10y ago
    You definitely addressed it at a high level (I saw your talk at PyCon), but I'd like to see the low-level details as well. This might just be a symptom of so few libraries following this pattern. I'd love to see some specific examples of handling various kinds of protocols (not just HTTP) with this approach, to see how it addresses various kinds of protocol components. For instance: variable-length data structures with length prefixes, variable-length data structures with "number of elements in the following array" in the middle, variable-length data structures with a terminator, text structures that require parsing tokens, and so on. Right now, the main documentation for those kinds of patterns seems to be "hope HTTP has a similar pattern and read the corresponding code in h2 or h11". I'd also love to see some reusable components that make it easier to build such protocol libraries.
    - Lukasa 10y ago
      Yeah, so that's very reasonable. In Python-land this is all pretty easy. For example, HTTP/2 is a protocol of the first kind ("variable-length data structures with length prefixes") at the framing layer, which is implemented in a Python packager called hyperframe. This uses a combination of the `struct` module and bytestring operations to achieve its results. A similar approach works for the second kind as well. Basically, in Python this is almost always much easier because struct sizing and memory allocation isn't a concern like it is in a C-like language (though even there, dynamically sized structures and pointers are your friends). But I agree, there is a lack of good discussion about "how do I actually do this?" I'd like to elaborate on that at some point for sure, because the reality is that it's remarkably simple.
clemensley 10y ago
Can someone point to a good resource for best practises in protocol design? Am currently faced with this problem and would like to avoid having to reinvent the wheel...
- jfoutz 10y ago
  Gosh, i don't know of anything off the top of my head. The only real tip i have is to byte length encode variable length requests, because scanning strings sucks. having a frame for messages for messages is nice, stick your fixed with stuff up front, and create fixed with length indicators. It sucks that you waste a little space with 0's, but avoiding the scan is pretty wonderful. GET /foo HTTP/1.1 GET HTTP 01.10 0004/foo Some things need to happen in order, you need to be authenticated before being granted access to resources, but that doesn't necessarily require the client knowing. the state at each step, Traditionally you'd have something like HELO jfoutz hunter2 OK REQEUEST 1 OK <bytes for resource 1> REQUEST 2 OK <bytes for resource 2> but really, the server knows if you're authenticated so i could send HELO jfoutz hunter2 REQUEST 1 REQUEST 1 and then get back either OK OK <bytes for resource 1> OK <bytes for resource 2> or NO NO NO or, whatever. generally it's just going to be a bunch of request/response pairs. sometimes you can't make later requests without actually looking at a response. You don't know what css or images to request until the html is parsed, for example. Usually, you can request a bunch of stuff at once, and it'll work out however it works out.
  - userbinator 10y ago
    The only real tip i have is to byte length encode variable length requests, because scanning strings sucks. This is really, really good advice to use whenever possible: it means clients can determine apriori how much data they need to read, and perhaps decide whether the length is valid and/or allocate sufficient memory. One of my annoyances with the traditional "almost-text-based" protocols like HTTP, FTP, SMTP, etc. is that parsing them is not trivial and often you need to keep reading until you hit the delimiter or reach an internal limit. In contrast, "read a length, then read length bytes" makes for simple and efficient implementation. There are certainly cases when the amount of data cannot be determined ahead of time, and in those cases I'd suggest chunked-length-prefixing; delimiters are really a method of last resort.
allan_s 10y ago
So I guess it will integrate well with uvloop http://magic.io/blog/uvloop-blazing-fast-python-networking/ http://magic.io/blog/uvloop-blazing-fast-python-networking/ ? (which is lacking an http parser :) ) May I dream that this + uvloop + asyncpg [0] and we can start to have something to build simple web services à la tornado that leverage python3.5? [0] http://magic.io/blog/asyncpg-1m-rows-from-postgres-to-python/ http://magic.io/blog/asyncpg-1m-rows-from-postgres-to-python...
- RubyPinch 10y ago
  https://github.com/MagicStack/httptools https://github.com/MagicStack/httptools magicstack already did their own http parser
voltagex_ 10y ago
Has anyone got any examples of this kind of design in C#?
nv-vn 10y ago
This idea is the basis for one of the popular HTTP libraries in OCaml, cohttp [1]. The core library (see the lib/ directory in the source code) implements the HTTP protocol, but makes no attempt to perform IO. The file lib/s.mli describes the interface that may be used to perform the IO for the library, making use of OCaml's module system to work generically over multiple interfaces. What's interesting about this approach is that it almost "generates the code" for IO once you plug it in to the implementation code (i.e. all you have to do is provide a handful of very basic functions and types and it will make use of those to construct the rest of the library). [1] - https://github.com/mirage/ocaml-cohttp https://github.com/mirage/ocaml-cohttp
knite 10y ago
Could we get "Python" in the title?
olalonde 10y ago
Node.js has pretty elegant read/write stream abstractions for this (https://nodejs.org/api/stream.html#stream_stream https://nodejs.org/api/stream.html#stream_stream).

123 points by kawera 10y ago

I used that approach for my minecraft protocol parsing library (https://github.com/dividuum/fastmc https://github.com/dividuum/fastmc). It's indeed very useful to decouple parsing a protocol from the underlying transport. I used my library both for socket communication as well as to read and write files. I'd love to see more libraries implemented that way. So thanks for posting that link. I imagine it might be difficult sometimes depending on the protocol. IIRC TLS might sometimes require writes while reading.

Lukasa 10y ago
It turns out TLS isn't that bad. OpenSSL exposes a "memory BIO", which is basically exactly this. You send in writes, and then emit data when you can, and in the meantime it buffers. It's a bit more limited than that, but generally speaking it's not too bad.
- JoshTriplett 10y ago
  I assume you handle decompression the same way? Feed in data, and the decompressor emits data when it can?
oxymoron 10y ago
On the topic of IO. I'm actually a fan of java.io.Reader/Writer and java.io.InputStream/OutputStream. It seems to me that most java protocols and parsers are quite composable by default due to these simple little abstractions. Granted, they might not be a good fit for async IO, but I'd still argue that they've held up quite well during the past fifteen years. I'll go out on a limb and claim that standard C++ IO haven't worked out quite as well, although it might just be my skip-the-oo-and-template-everything skepticism. Rust seems closer to java, although I feel like the Jury's still out. I'm not a enthusiastic of doing everything asynchronously as in Node. Haskell does seem to get the composability right. Python has some of javas thinking through a standard convention on the method names for reading and writing (an interface!), but it seems common with departures from those simple rules when people get creative. From that perspective, an initiative like this is quite understandable and convenient. Nevertheless, I don't seem to feel the need in the same way in certain other languages.
- paulddraper 10y ago
  A little off-topic but IMO one of the more unfortunate decisions by Java was Reader/Writer and InputStream/OutputStream, instead of CharReader/CharWriter and ByteReader/ByteWriter (or similar).

wtbob 10y ago

From the description, it sounds like the necessity is the lack of a single way to do I/O in Python. If that's right, then this shows the real advantage of Go's io.Reader/Writer/&c. interfaces, which enable this sort of composability. From my Python years, I'd think that something built around mandatory read()/write() methods could do much the same.

bdarnell 10y ago
Python has a standard interface just like Go's `io.{Read,Writ}er`: the `send` and `recv` methods as defined by `socket.socket`. The appeal of "sans I/O" protocol implementations is primarily for asynchronous I/O, which requires a completely different style of interface. The clever thing about Go is not the standardization of a reader/writer interface, but the fact that the goroutine I/O scheduler makes these synchronous interfaces nearly as efficient as asynchronous ones, so there is no need to introduce a separate family of asynchronous interfaces.
- jerf 10y ago
  "Python has a standard interface just like Go's `io.{Read,Writ}er`: the `send` and `recv` methods as defined by `socket.socket`." But a File has no "send" and "recv". io.Reader and Writer apply to files, sockets, byte slices, strings, compositions of other Readers or Writers, HTTP request and response bodies, and anything else that strikes my fancy to implement the correct methods. Sockets appear to deliberately not have those methods, as there's this line in the docs: "Note that there are no methods read() or write(); use recv() and send() without flags argument instead." There's nothing preventing Python from having the equivalent of Reader and Writer. It just doesn't, at least not with the standard library. In fact that's true of a lot of languages; there's nothing preventing that interface from existing, it just doesn't. This is a point in favor of getting the standard library right earlier rather than a point in favor of Go-the-language.
  - solidsnack9000 10y ago
    How would a standard interface like this support async IO?

JoshTriplett 10y ago

This nicely documents the "why", as does the linked PyCon talk. But both don't go into much detail on the "how", making it difficult to follow suit.

Lukasa 10y ago
Hi, I'm the linked PyCon speaker! It'd be good to know what additional information you'd like on the "how". I think at a high level I addressed that in my talk, but if it didn't make it across or you needed more information I'd love to know what you need. Ideally I'll turn this into a blog post at some point so it'd be great to have an idea of what extra info is needed.
- JoshTriplett 10y ago
  You definitely addressed it at a high level (I saw your talk at PyCon), but I'd like to see the low-level details as well. This might just be a symptom of so few libraries following this pattern. I'd love to see some specific examples of handling various kinds of protocols (not just HTTP) with this approach, to see how it addresses various kinds of protocol components. For instance: variable-length data structures with length prefixes, variable-length data structures with "number of elements in the following array" in the middle, variable-length data structures with a terminator, text structures that require parsing tokens, and so on. Right now, the main documentation for those kinds of patterns seems to be "hope HTTP has a similar pattern and read the corresponding code in h2 or h11". I'd also love to see some reusable components that make it easier to build such protocol libraries.
  - Lukasa 10y ago
    Yeah, so that's very reasonable. In Python-land this is all pretty easy. For example, HTTP/2 is a protocol of the first kind ("variable-length data structures with length prefixes") at the framing layer, which is implemented in a Python packager called hyperframe. This uses a combination of the `struct` module and bytestring operations to achieve its results. A similar approach works for the second kind as well. Basically, in Python this is almost always much easier because struct sizing and memory allocation isn't a concern like it is in a C-like language (though even there, dynamically sized structures and pointers are your friends). But I agree, there is a lack of good discussion about "how do I actually do this?" I'd like to elaborate on that at some point for sure, because the reality is that it's remarkably simple.

clemensley 10y ago

Can someone point to a good resource for best practises in protocol design? Am currently faced with this problem and would like to avoid having to reinvent the wheel...

jfoutz 10y ago
Gosh, i don't know of anything off the top of my head. The only real tip i have is to byte length encode variable length requests, because scanning strings sucks. having a frame for messages for messages is nice, stick your fixed with stuff up front, and create fixed with length indicators. It sucks that you waste a little space with 0's, but avoiding the scan is pretty wonderful. GET /foo HTTP/1.1 GET HTTP 01.10 0004/foo Some things need to happen in order, you need to be authenticated before being granted access to resources, but that doesn't necessarily require the client knowing. the state at each step, Traditionally you'd have something like HELO jfoutz hunter2 OK REQEUEST 1 OK <bytes for resource 1> REQUEST 2 OK <bytes for resource 2> but really, the server knows if you're authenticated so i could send HELO jfoutz hunter2 REQUEST 1 REQUEST 1 and then get back either OK OK <bytes for resource 1> OK <bytes for resource 2> or NO NO NO or, whatever. generally it's just going to be a bunch of request/response pairs. sometimes you can't make later requests without actually looking at a response. You don't know what css or images to request until the html is parsed, for example. Usually, you can request a bunch of stuff at once, and it'll work out however it works out.
- userbinator 10y ago
  The only real tip i have is to byte length encode variable length requests, because scanning strings sucks. This is really, really good advice to use whenever possible: it means clients can determine apriori how much data they need to read, and perhaps decide whether the length is valid and/or allocate sufficient memory. One of my annoyances with the traditional "almost-text-based" protocols like HTTP, FTP, SMTP, etc. is that parsing them is not trivial and often you need to keep reading until you hit the delimiter or reach an internal limit. In contrast, "read a length, then read length bytes" makes for simple and efficient implementation. There are certainly cases when the amount of data cannot be determined ahead of time, and in those cases I'd suggest chunked-length-prefixing; delimiters are really a method of last resort.

allan_s 10y ago

So I guess it will integrate well with uvloop http://magic.io/blog/uvloop-blazing-fast-python-networking/ http://magic.io/blog/uvloop-blazing-fast-python-networking/ ? (which is lacking an http parser :) ) May I dream that this + uvloop + asyncpg [0] and we can start to have something to build simple web services à la tornado that leverage python3.5? [0] http://magic.io/blog/asyncpg-1m-rows-from-postgres-to-python/ http://magic.io/blog/asyncpg-1m-rows-from-postgres-to-python...

RubyPinch 10y ago
https://github.com/MagicStack/httptools https://github.com/MagicStack/httptools magicstack already did their own http parser

voltagex_ 10y ago

Has anyone got any examples of this kind of design in C#?

nv-vn 10y ago

This idea is the basis for one of the popular HTTP libraries in OCaml, cohttp [1]. The core library (see the lib/ directory in the source code) implements the HTTP protocol, but makes no attempt to perform IO. The file lib/s.mli describes the interface that may be used to perform the IO for the library, making use of OCaml's module system to work generically over multiple interfaces. What's interesting about this approach is that it almost "generates the code" for IO once you plug it in to the implementation code (i.e. all you have to do is provide a handful of very basic functions and types and it will make use of those to construct the rest of the library). [1] - https://github.com/mirage/ocaml-cohttp https://github.com/mirage/ocaml-cohttp

knite 10y ago

Could we get "Python" in the title?

olalonde 10y ago

Node.js has pretty elegant read/write stream abstractions for this (https://nodejs.org/api/stream.html#stream_stream https://nodejs.org/api/stream.html#stream_stream).