5 ms·

> The data model for TCP is a stream of bytes. However, this is not the right data model for most datacenter applications. Datacenter applications typically exc

> The data model for TCP is a stream of bytes. However, this is not the right data model for most datacenter applications. Datacenter applications typically exchange discrete messages to implement remote procedure calls This isn't just a datacenter problem. Every single network protocol I've ever created or implemented is message based, not stream based. Every messaging system. Every video game. Every RPC transport. But, because we can't have nice things, message framing has to be re-implemented on top of TCP in a different, custom way by every single protocol. I've basically got message framing-over-TCP in muscle memory at this point, in each of the variants you commonly see. The only kinda-sorta exceptions I know about are HTTP/1.1 and telnet. But HTTP/1.1 is still a message oriented protocol; just with file-sized messages. (And even this stops being true with http2 anyway). In my opinion, the real problem is the idea that "everything is a file". Byte streams aren't a very useful abstraction. "Everything is a stream of messages" would be a far better base metaphor for computing.

langsoul-com 4y ago
What do you make to require implementing or making network protocols?
- josephg 4y ago
  Multiplayer video games, realtime updating webpages, database bindings, sync protocols (I play around around with CRDTs a lot), p2p distributed systems, whatever really! Implementing a wire protocol seems to come up about once every couple of years. And for context, I've been programming now for about 30 years.
cryptonector 4y ago
> The only kinda-sorta exceptions I know about are HTTP/1.1 and telnet. But HTTP/1.1 is still a message oriented protocol; just with file-sized messages. (And even this stops being true with http2 anyway). No, HTTP/2 and QUIC do not change the semantics of HTTP. Also, you can have endless streams with HTTP/1.1: just use chunked encoding to POST/PUT and use Range: bytes=0- and chunked encoding for GET and chunked encoding for POST response bodies. In HTTP/2 there's only the equivalent of chunked encoding -- there's no definite content length in HTTP/2.
- josephg 4y ago
  Correct me if I’m wrong, but doesn’t h2 still break up requests and responses into smaller message frames in order to do multiplexing? Those message frames are what I’m talking about - as I understand it, they are, yet again, a message oriented protocol layered on top of tcp.
  - cryptonector 4y ago
    Yes, and even HTTP/1.1 does that. I forget if telnet does something similar, but I suspect it must because it can send control data. The FTP protocol, and the BSD r-command might be the only ones that truly do no additional framing (FTP for data connections, r-commands post-login).
  - cryptonector 4y ago
    You will always need _some_ framing if you're dealing with bulk data. If it's not bulk, then you don't need framing if everything can fit into one datagram / whatever transfer unit provided by the transport, but the transport itself will need some framing, especially if it will need to support any kind of fragmentation. If it's not bulk and it doesn't fit in a datagram / whatever transfer unit provided by the transport and the transport doesn't do fragmentation, then you have to do framing yourself, and then you have a sequencing problem, and so on, and you quickly re-invent parts of TCP but at the application layer. Basically, it seems inescapable that the Internet is based on packets, and that packets are limited in size, and so application protocols have to be smeared onto packets. Things are only ever trivial when you're doing request/response protocols with always- or mostly-small requests and responses. The moment you need anything that doesn't fit in the path MTU minus overhead, you need framing. So I don't think that an octet stream abstraction is quaint and obsolete.
- amluto 4y ago
  Once chunked encoding is in the picture, even HTTP/1.1 sends messages, not streams, under the hood.
  - cryptonector 4y ago
imtringued 4y ago
I don't bother implementing that crap anymore. I just websockets which are message oriented out of the box.
bruce343434 4y ago
What’s the issue with fitting messages in streams?
- bheadmaster 4y ago
  One of the issues I can think of is head-of-line blocking [0]. If you're sending messages of different priorities over the same channel, an error in sending a low-priority message, high-priority messages will have to wait until the low-priority message is properly re-transmitted. [0] https://en.wikipedia.org/wiki/Head-of-line_blocking https://en.wikipedia.org/wiki/Head-of-line_blocking https://en.wikipedia.org/wiki/Head-of-line_blocking https://en.wikipedia.org/wiki/Head-of-line_blocking
  - fulafel 4y ago
    Yes, but this need happen only when your data center network is congested, which is hopefully rare and is relatively cheap to fix by adding capacity. And in congestion cases you need TCP's back off ability. Getting rid of head-of-line blocking also makes messages happen in order. Making messages (sometimes) happen out of order would drastically increase implementation complexity for a lot of apps.
pgorczak 4y ago
Finding the right abstraction isn’t easy in a network stack. TCP’s is less useful for application logic but it reflects the way the protocol works internally. Bunch of Bytes goes in on one side, Bytes stream out on the other side in chunks whose size depends on congestion, physical layer and other facts. A message based API hides these facts or leaks them depending on how you look at it.
ithkuil 4y ago
Even TCP itself uses discrete messages under the hood :-)
- josephg 4y ago
  Hah yes; although TCP frames can be arbitrarily refragmented and rejoined as they travel through the network before reaching your destination. If you ever play an indie game which seems unusually janky over wifi, its probably because the code isn't correctly rejoining fragmented network packets at the application level. Ethernet is remarkably well behaved in this regard. Wifi is much better at shaking out buggy code.
Aissen 4y ago
Because most protocols can handle message loss, with retransmit and proper ordering ? And we haven't started to talk about congestion yet… TCP is useful, and while I'd like to see a message get rid of one (or more) of those constraints to go with a a custom protocol, I feel like they'd be re-implementing the features in the end because these are very useful properties to have… Edit: the proposal in the article is actually quite, sensible, but requires redesigning your apps… And I'd like to see how it performs: TCP is a hugely optimized beast (when it works well), with hardware offloads, kernel optimizations, etc.
- 8371861215 4y ago
  - Kiro 4y ago
    Very peculiar spam. Does anyone have a theory what the motive is?
    - askvictor 4y ago
      Modern equivalent of a Numbers Station https://en.wikipedia.org/wiki/Numbers_station https://en.wikipedia.org/wiki/Numbers_station
yaantc 4y ago
SCTP [1] is there to provide a reliable message based protocol. And it does work inside a datacenter. The issue is outside the datacenter: it doesn't work reliably across the Internet due to middle boxes dumping anything not TCP or UDP... But inside a controlled environment like a datacenter, it works. It's been used in the telecommunication world to carry control messages in the radio access and core networks for example. So it's been tested at scale for critical applications. [1] https://en.wikipedia.org/wiki/Stream_Control_Transmission_Protocol https://en.wikipedia.org/wiki/Stream_Control_Transmission_Pr...
MisterTea 4y ago
> In my opinion, the real problem is the idea that "everything is a file". Files are just an indexed list of bytes that can represent anything. Think of them as objects. > Byte streams aren't a very useful abstraction. "Everything is a stream of messages" would be a far better base metaphor for computing. I don't understand this. A stream of messages is a stream of bytes. Its bytes all the way down.

langsoul-com 4y ago

What do you make to require implementing or making network protocols?

josephg 4y ago
Multiplayer video games, realtime updating webpages, database bindings, sync protocols (I play around around with CRDTs a lot), p2p distributed systems, whatever really! Implementing a wire protocol seems to come up about once every couple of years. And for context, I've been programming now for about 30 years.

cryptonector 4y ago

> The only kinda-sorta exceptions I know about are HTTP/1.1 and telnet. But HTTP/1.1 is still a message oriented protocol; just with file-sized messages. (And even this stops being true with http2 anyway). No, HTTP/2 and QUIC do not change the semantics of HTTP. Also, you can have endless streams with HTTP/1.1: just use chunked encoding to POST/PUT and use Range: bytes=0- and chunked encoding for GET and chunked encoding for POST response bodies. In HTTP/2 there's only the equivalent of chunked encoding -- there's no definite content length in HTTP/2.

josephg 4y ago
Correct me if I’m wrong, but doesn’t h2 still break up requests and responses into smaller message frames in order to do multiplexing? Those message frames are what I’m talking about - as I understand it, they are, yet again, a message oriented protocol layered on top of tcp.
- cryptonector 4y ago
  Yes, and even HTTP/1.1 does that. I forget if telnet does something similar, but I suspect it must because it can send control data. The FTP protocol, and the BSD r-command might be the only ones that truly do no additional framing (FTP for data connections, r-commands post-login).
- cryptonector 4y ago
  You will always need _some_ framing if you're dealing with bulk data. If it's not bulk, then you don't need framing if everything can fit into one datagram / whatever transfer unit provided by the transport, but the transport itself will need some framing, especially if it will need to support any kind of fragmentation. If it's not bulk and it doesn't fit in a datagram / whatever transfer unit provided by the transport and the transport doesn't do fragmentation, then you have to do framing yourself, and then you have a sequencing problem, and so on, and you quickly re-invent parts of TCP but at the application layer. Basically, it seems inescapable that the Internet is based on packets, and that packets are limited in size, and so application protocols have to be smeared onto packets. Things are only ever trivial when you're doing request/response protocols with always- or mostly-small requests and responses. The moment you need anything that doesn't fit in the path MTU minus overhead, you need framing. So I don't think that an octet stream abstraction is quaint and obsolete.
amluto 4y ago
Once chunked encoding is in the picture, even HTTP/1.1 sends messages, not streams, under the hood.
- cryptonector 4y ago

imtringued 4y ago

I don't bother implementing that crap anymore. I just websockets which are message oriented out of the box.

bruce343434 4y ago

What’s the issue with fitting messages in streams?

bheadmaster 4y ago
One of the issues I can think of is head-of-line blocking [0]. If you're sending messages of different priorities over the same channel, an error in sending a low-priority message, high-priority messages will have to wait until the low-priority message is properly re-transmitted. [0] https://en.wikipedia.org/wiki/Head-of-line_blocking https://en.wikipedia.org/wiki/Head-of-line_blocking https://en.wikipedia.org/wiki/Head-of-line_blocking https://en.wikipedia.org/wiki/Head-of-line_blocking
- fulafel 4y ago
  Yes, but this need happen only when your data center network is congested, which is hopefully rare and is relatively cheap to fix by adding capacity. And in congestion cases you need TCP's back off ability. Getting rid of head-of-line blocking also makes messages happen in order. Making messages (sometimes) happen out of order would drastically increase implementation complexity for a lot of apps.

pgorczak 4y ago

Finding the right abstraction isn’t easy in a network stack. TCP’s is less useful for application logic but it reflects the way the protocol works internally. Bunch of Bytes goes in on one side, Bytes stream out on the other side in chunks whose size depends on congestion, physical layer and other facts. A message based API hides these facts or leaks them depending on how you look at it.

ithkuil 4y ago

Even TCP itself uses discrete messages under the hood :-)

josephg 4y ago
Hah yes; although TCP frames can be arbitrarily refragmented and rejoined as they travel through the network before reaching your destination. If you ever play an indie game which seems unusually janky over wifi, its probably because the code isn't correctly rejoining fragmented network packets at the application level. Ethernet is remarkably well behaved in this regard. Wifi is much better at shaking out buggy code.

Aissen 4y ago

Because most protocols can handle message loss, with retransmit and proper ordering ? And we haven't started to talk about congestion yet… TCP is useful, and while I'd like to see a message get rid of one (or more) of those constraints to go with a a custom protocol, I feel like they'd be re-implementing the features in the end because these are very useful properties to have… Edit: the proposal in the article is actually quite, sensible, but requires redesigning your apps… And I'd like to see how it performs: TCP is a hugely optimized beast (when it works well), with hardware offloads, kernel optimizations, etc.

8371861215 4y ago
- Kiro 4y ago
  Very peculiar spam. Does anyone have a theory what the motive is?
  - askvictor 4y ago
    Modern equivalent of a Numbers Station https://en.wikipedia.org/wiki/Numbers_station https://en.wikipedia.org/wiki/Numbers_station

yaantc 4y ago

SCTP [1] is there to provide a reliable message based protocol. And it does work inside a datacenter. The issue is outside the datacenter: it doesn't work reliably across the Internet due to middle boxes dumping anything not TCP or UDP... But inside a controlled environment like a datacenter, it works. It's been used in the telecommunication world to carry control messages in the radio access and core networks for example. So it's been tested at scale for critical applications. [1] https://en.wikipedia.org/wiki/Stream_Control_Transmission_Protocol https://en.wikipedia.org/wiki/Stream_Control_Transmission_Pr...

MisterTea 4y ago

> In my opinion, the real problem is the idea that "everything is a file". Files are just an indexed list of bytes that can represent anything. Think of them as objects. > Byte streams aren't a very useful abstraction. "Everything is a stream of messages" would be a far better base metaphor for computing. I don't understand this. A stream of messages is a stream of bytes. Its bytes all the way down.