36 ms·

One of the big issues with TCP is a lot of communication isn't well-suited to the stream model and is message-oriented, so yes, like the author says, you have t

One of the big issues with TCP is a lot of communication isn't well-suited to the stream model and is message-oriented, so yes, like the author says, you have to go implement ACKs. And then you want multiplexing, which streams also fail at. Before you know it, you built a worse version of HTTP2.

iTokio 4y ago
And then you realize that sometimes it has become even worse due to tcp head of line blocking and you move to udp and build a worse version of http3
- hinkley 4y ago
  I think Head of Line blocking is a more concrete problem but it seems that it's often encountered when running away from a different one: If you have different conversations going on between a single client and the server, you can make several connections to do it. You'll pay a little bit more at the start of the conversation of course, so be frugal with this design, and think about ways to 'hide' the delay during bootstrapping. But know that with funky enough network conditions, it's possible for one connection to error out but the other to continue to work. The problem for me always comes back to information architecture, and the pervasive lack of it, or at least lack of investment in it. If you really have two separate conversations with the server, losing one channel shouldn't make the whole application freak out. But we all know people take the path of least resistance, and soon you have 2.5 conversations on one channel and 1.5 on the other. The advice here is some of the same advice in more sophisticated treatises on RESTful APIs (it's all distributed computing, it's all the same problem set arranged in different orders). REST calls are generally supposed to be stateless. The client making what looks like a non sequitur call to the backend should Just Work. If you manage that, then having one channel inform the client of the existence of a resource and fetching that resource out of band isn't really coupling of the two channels. That subtlety is lost on some people who defend their actual coupling by pointing out that other people have done it too, when no, they really haven't. And if anything, multiplexing lets them get away with this bad behavior for much longer, allowing the bad patterns to become idiomatic.
- touisteur 4y ago
  sctp has been available on Linux (and other bsd-socket userlands, with sctp over udp) for some time now. HTTP3 fixes the one sctp limitation, which is encryption of the whole session (and not just individual streams). At least that's what HN told me last time I said 'but sctp!' :-)
  - theamk 4y ago
    Lots of things are still limited to tcp/udp only, such as traffic analyzers, home router port forwarding and cloud firewall rules. If you want your networking software to be widely used, you probably don't want to limit yourself by using sctp.
    - touisteur 4y ago
      Yes but sctp can run over udp, that's how I ever used it. And QUIC has bitten the bullet of using udp over the Internet, so it was possible. Just feels like a missed opportunity, but now that QUIC is here, there's not much more to say. There's lots of fancy things with sctp that I always found magic and it defines the base of your architecture. Having a multi-homing, multi-path transport API alleviates many headaches.
jdthedisciple 4y ago
Naive questions but doesn't QUIC solve those problems?
- dcsommer 4y ago
  QUIC is HTTP/3 more or less and solves the same problems, yes.
  - Matthias247 4y ago
    Somewhat. Http/3 is a protocol that runs on top of QUIC. Every http/3 request uses a dedicated QUIC stream. However it is possible to use QUIC for other use cases that have nothing to do with HTTP. QUIC is more of a multiplexed TCP with mandatory encryption.
- cogman10 4y ago
  HTTP2 solves these problems. HTTP3 (QUIC) solves the head of line blocking problem of TCP. That is, a packet getting corrupted or lost causes TCP to hold up other packets from going out until that packet can be successfully delivered. So you may be multiplexed in your messages, but you end up with a slowdown and backlog of data that could have been successfully received and interpreted but for the lost packet.
- kevin_thibedeau 4y ago
  SCTP solved them. We're just rarely allowed to use it because the internet is broken.
  - lxgr 4y ago
    Yes, it's a bummer that we have to encapsulate it in UDP, but when using it like that, it works quite well in my experience.
danesparza 4y ago
If you insist on passing messages, use a well-designed message queue service and don't rebuild the wheel (but just a little shittier).
- gopalv 4y ago
  > use a well-designed message queue service There is a faq about ZeroMQ written in the Monty Pythonesque fashion of "other than that, what does ZeroMQ do for us?" I can't quite find it in my bookmarks, but it went by the "so it gives you sockets", "but also message batching" ... "but other than batching, what does it give us?". Also, the whole problem with using just TCP is that often it needs kernel level tuning - like you need to fix INET MSL on some BSD boxes to avoid port exhaustion or tweak DSACK when you have hanging connections in the network stack (like S3 connections hanging when SACK/DSACK is on). A standard library is likely to have bugs too, but hopefully someone else has found it before you run into it.
  - theossuary 4y ago
    I'm pretty hesitant to use ZeroMQ for anything anymore. I was digging into what using ZeroMQ might look like in Rust and ran across a pretty interesting issue that eventually made me decide to drop it for AMQP - https://github.com/jean-airoldie/libzmq-rs/issues/125#issuecomment-570551319 https://github.com/jean-airoldie/libzmq-rs/issues/125#issuec...
    - twic 4y ago
      I think the situation is more subtle than the poster admits. No, ZeroMQ and successors do not tell you about socket state. You can't detect disconnection or reconnection. But then if a TCP connection fails in some way that does not lead to disconnection (packets getting dropped, remote machine powers down), it can't possibly tell you about that either, but you still need to deal with it. So in any case, you need some sort of application-level error detection and recovery; you need heartbeats, and serial numbers in messages, and a protocol for explicitly restarting a connection and performing the initial handshake. And once you have that, explicit connection events from ZeroMQ are much less important. Admittedly, given that this is a TCP transport, reporting reconnections would still be useful, because TCP won't ever drop messages from the interior of a sequence itself (if it delivers 15, it has delivered 1 - 14 already), so you shouldn't need the serial numbers. And if it's really not possible to detect authentication failures, than that seems rubbish. And it seems that is indeed the case: https://github.com/zeromq/libzmq/issues/3505 https://github.com/zeromq/libzmq/issues/3505
randyrand 4y ago
I'd also recommend MQTT if you want message based networking with automatic retries, etc.

iTokio 4y ago

And then you realize that sometimes it has become even worse due to tcp head of line blocking and you move to udp and build a worse version of http3

hinkley 4y ago
I think Head of Line blocking is a more concrete problem but it seems that it's often encountered when running away from a different one: If you have different conversations going on between a single client and the server, you can make several connections to do it. You'll pay a little bit more at the start of the conversation of course, so be frugal with this design, and think about ways to 'hide' the delay during bootstrapping. But know that with funky enough network conditions, it's possible for one connection to error out but the other to continue to work. The problem for me always comes back to information architecture, and the pervasive lack of it, or at least lack of investment in it. If you really have two separate conversations with the server, losing one channel shouldn't make the whole application freak out. But we all know people take the path of least resistance, and soon you have 2.5 conversations on one channel and 1.5 on the other. The advice here is some of the same advice in more sophisticated treatises on RESTful APIs (it's all distributed computing, it's all the same problem set arranged in different orders). REST calls are generally supposed to be stateless. The client making what looks like a non sequitur call to the backend should Just Work. If you manage that, then having one channel inform the client of the existence of a resource and fetching that resource out of band isn't really coupling of the two channels. That subtlety is lost on some people who defend their actual coupling by pointing out that other people have done it too, when no, they really haven't. And if anything, multiplexing lets them get away with this bad behavior for much longer, allowing the bad patterns to become idiomatic.
touisteur 4y ago
sctp has been available on Linux (and other bsd-socket userlands, with sctp over udp) for some time now. HTTP3 fixes the one sctp limitation, which is encryption of the whole session (and not just individual streams). At least that's what HN told me last time I said 'but sctp!' :-)
- theamk 4y ago
  Lots of things are still limited to tcp/udp only, such as traffic analyzers, home router port forwarding and cloud firewall rules. If you want your networking software to be widely used, you probably don't want to limit yourself by using sctp.
  - touisteur 4y ago
    Yes but sctp can run over udp, that's how I ever used it. And QUIC has bitten the bullet of using udp over the Internet, so it was possible. Just feels like a missed opportunity, but now that QUIC is here, there's not much more to say. There's lots of fancy things with sctp that I always found magic and it defines the base of your architecture. Having a multi-homing, multi-path transport API alleviates many headaches.

jdthedisciple 4y ago

Naive questions but doesn't QUIC solve those problems?

dcsommer 4y ago
QUIC is HTTP/3 more or less and solves the same problems, yes.
- Matthias247 4y ago
  Somewhat. Http/3 is a protocol that runs on top of QUIC. Every http/3 request uses a dedicated QUIC stream. However it is possible to use QUIC for other use cases that have nothing to do with HTTP. QUIC is more of a multiplexed TCP with mandatory encryption.
cogman10 4y ago
HTTP2 solves these problems. HTTP3 (QUIC) solves the head of line blocking problem of TCP. That is, a packet getting corrupted or lost causes TCP to hold up other packets from going out until that packet can be successfully delivered. So you may be multiplexed in your messages, but you end up with a slowdown and backlog of data that could have been successfully received and interpreted but for the lost packet.
kevin_thibedeau 4y ago
SCTP solved them. We're just rarely allowed to use it because the internet is broken.
- lxgr 4y ago
  Yes, it's a bummer that we have to encapsulate it in UDP, but when using it like that, it works quite well in my experience.

danesparza 4y ago

If you insist on passing messages, use a well-designed message queue service and don't rebuild the wheel (but just a little shittier).

gopalv 4y ago
> use a well-designed message queue service There is a faq about ZeroMQ written in the Monty Pythonesque fashion of "other than that, what does ZeroMQ do for us?" I can't quite find it in my bookmarks, but it went by the "so it gives you sockets", "but also message batching" ... "but other than batching, what does it give us?". Also, the whole problem with using just TCP is that often it needs kernel level tuning - like you need to fix INET MSL on some BSD boxes to avoid port exhaustion or tweak DSACK when you have hanging connections in the network stack (like S3 connections hanging when SACK/DSACK is on). A standard library is likely to have bugs too, but hopefully someone else has found it before you run into it.
- theossuary 4y ago
  I'm pretty hesitant to use ZeroMQ for anything anymore. I was digging into what using ZeroMQ might look like in Rust and ran across a pretty interesting issue that eventually made me decide to drop it for AMQP - https://github.com/jean-airoldie/libzmq-rs/issues/125#issuecomment-570551319 https://github.com/jean-airoldie/libzmq-rs/issues/125#issuec...
  - twic 4y ago
    I think the situation is more subtle than the poster admits. No, ZeroMQ and successors do not tell you about socket state. You can't detect disconnection or reconnection. But then if a TCP connection fails in some way that does not lead to disconnection (packets getting dropped, remote machine powers down), it can't possibly tell you about that either, but you still need to deal with it. So in any case, you need some sort of application-level error detection and recovery; you need heartbeats, and serial numbers in messages, and a protocol for explicitly restarting a connection and performing the initial handshake. And once you have that, explicit connection events from ZeroMQ are much less important. Admittedly, given that this is a TCP transport, reporting reconnections would still be useful, because TCP won't ever drop messages from the interior of a sequence itself (if it delivers 15, it has delivered 1 - 14 already), so you shouldn't need the serial numbers. And if it's really not possible to detect authentication failures, than that seems rubbish. And it seems that is indeed the case: https://github.com/zeromq/libzmq/issues/3505 https://github.com/zeromq/libzmq/issues/3505

randyrand 4y ago

I'd also recommend MQTT if you want message based networking with automatic retries, etc.