5 ms·

Most of TCP woes comes from high-bandwith latency-sensitive stuff like HFT and video, but TCP isn't particularly good for low-bandwidth high-latency networks ei

Most of TCP woes comes from high-bandwith latency-sensitive stuff like HFT and video, but TCP isn't particularly good for low-bandwidth high-latency networks either (e. g. NB-IoT with 10 seconds worst case RTT): - TCP will waste roundtrips on handshakes. And then some extra on MTU discovery. - TCP will keep trying to transmit data even if it's no longer useful (same issue as with real-time multimedia). - If you move into a location with worse coverage, your latency increases, but TCP will assume packet loss due to congestion and reduce bandwidth. And in general, loss-based congestion control just doesn't work at this point. - Load balancers and middleboxes (and HTTP servers, but that's another story) may disconnect you randomly because hey, you haven't responded for four seconds, you are probably no longer there anyway, right? - You can't interpret the data you've got until you have all of it - because TCP will split packets with no regards to data structure. Which is twice as sad when all of your data would actually fit in 1200 bytes.

eru 2y ago
And TCP wants to pretend that all your data arrives as a linear stream one after the other; and keeps you from seeing package n+1, if you haven't received package n, yet. That's useful for some things, but often you can make use of package n+1, even when package n hasn't arrived, yet. For example, when you are transferring a large file, you could use erasure encoding to just automatically deal with 5% package loss. (Or you could use a fountain code to deal with variable packet loss, and the sender just keeps sending until the receiver says "I'm done" for the whole file, instead of ack-ing individual packages. Fountain codes are how deep space probes send their data back. Latency is pretty terrible out to Jupiter or Mars.)
- Aurornis 2y ago
  > And TCP wants to pretend that all your data arrives as a linear stream one after the other; That’s one of the reasons people use TCP. > For example, when you are transferring a large file, you could use erasure encoding to just automatically deal with 5% package loss. It’s never that simple. You can’t just add some erasure coding and have it automatically solve your problems. You now have to build an entire protocol around those packets to determine and track their order. You also need mechanisms to handle the case where packet loss exceeds what you can recover, which involves either restarting the transfer or a retransmission mechanism. The number of little details you have to handle quickly explodes in complexity. Even in the best case scenario, you’d be paying a price to handle erasure coding on one end, the extra bandwidth of the overhead, and then decoding on the receiving end. That’s a lot of complexity, engineering, debugging, and opportunities to introduce bugs, and for what gain? In the file transfer example, what would you actually gain by rolling your own entire transmission protocol with all this overhead? > Fountain codes are how deep space probes send their data back. Latency is pretty terrible out to Jupiter or Mars.) Fountain codes and even general erasure codes are not the right tool for this job. The loss of a digital packetized channel across the internet is very different than a noisy analog channel sent through space.
  - eru 2y ago
    > It’s never that simple. You can’t just add some erasure coding and have it automatically solve your problems. You now have to build an entire protocol around those packets to determine and track their order. Yes, and you'd want to mostly build this extra protocol only once, and stick it into a library. > You also need mechanisms to handle the case where packet loss exceeds what you can recover, which involves either restarting the transfer or a retransmission mechanism. Or you can use a fountain code, and just keep transmitting. > The number of little details you have to handle quickly explodes in complexity. Even in the best case scenario, you’d be paying a price to handle erasure coding on one end, the extra bandwidth of the overhead, and then decoding on the receiving end. Well, you can also use a simpler mechanism: you transmit all the packages from A to B once, then at the end B tells A which packages get lost, and A sends those again. Repeat until you have everything. That way needs more back-and-forth communication, but doesn't need any fancy error correcting code. > Fountain codes and even general erasure codes are not the right tool for this job. The loss of a digital packetized channel across the internet is very different than a noisy analog channel sent through space. Yes, the loss model is different. However you can eg interleave your bits to get something that close enough to work. The main reason you don't need to use error correcting codes, is that transmitting feedback to the sender is typically a lot cheaper on the internet than in outer space. Not least because even bad latencies are typically measured in seconds at most, not minutes or hours. (You could however use these codes when for some reason you have a very asymmetrical link. Eg if you have internet via geo-stationary satellite, or if one sender is broadcast to lots and lots of different receivers, and doesn't want to deal with each of them individually.)
- jakjak123 2y ago
  > And TCP wants to pretend that all your data arrives as a linear stream one after the other; This is actually very nice for an application developer. Probably a tiny bit less nice if you are running popular services taking 100k req/s.
  - eru 2y ago
    Yes, that is convenient. But it's an abstraction you don't always need, but still always pay for. That's why I would suggest that were possible the kernel should only expose something at most as high level as UDP, and we build the TCP abstractions on top of that in user level. Of course, you wouldn't want to re-write everything from scratch all the time: you'd use libraries. The nice thing about user level libraries is that you can swap them out for better versions or for different abstractions without any privileged access.
sgt 2y ago
Is NB-IoT actually being used these days? I remember it was hyped up but then the buzz disappeared.
- tliltocatl 2y ago
  It definitely does, at least in Europe, not sure how things are on the other side of the pond. For the user equipment there is no real advantage over LTE-M, actually is LTE-M better for our case (send a single packet a day and go to sleep). But then I don't see the picture from the operator side, maybe the radio savings actually matter for them. Some operators either don't have LTE-M at all or don't allow PSM on LTE-M because screw you that's why. On NB-IoT PSM is mandatory at least. And yes, LoRa/Sigfox/other non-cellular LPWAN might technically be better IF you provide the base stations. Nobody wants to do that. With cellular you can just mail the device and cross your fingers the operators haven't misconfigured roaming.
  - sgt 2y ago
    That's quite interesting. And the promised battery lifespans of NB-IoT device, did it hold some truth? In the beginning we heard about 10 year life spans for simple well designed devices.
    - jjoonathan 2y ago
      We have millions of devices in the field older than that on the same D-cell LiSOCl2 battery they shipped with. Some of them even take magnetic measurements with powered excitation!
    - tliltocatl 2y ago
      You'll have to wait 10 years to see. But on paper it does (we aim for 15) and we have devices in field for some three years. But for the "simple well-designed" - the problem is, the whole thing is a black box for application developer, and not a bugless one. So much of the "well-designed" part is to figure out when is the right time to pull modem's proverbial power cord. If you don't, it'll gladly eat up your battery in 48 hours after a cell handover goes wrong with no warning.
cma 2y ago
> If you move into a location with worse coverage, your latency increases, but TCP will assume packet loss due to congestion and reduce bandwidth. And in general, loss-based congestion control just doesn't work at this point. Don't lower levels of wifi below TCP now do often their own retransmit, even if it is just from weak/noisy signal and not a sensed collision?
saghm 2y ago
> TCP will waste roundtrips on handshakes. And then some extra on MTU discovery. This is especially true when using TLS (which is at least a major use case nowadays if not the default assumption); the TCP handshake has to complete before the TLS handshake can occur, whereas QUIC has protocol-level support for doing TLS negotiation as part of the initial handshake. Having the network protocol and the encryption be composable rather than tightly coupled feels more elegant, it's hard for me to convince myself that being able to save a round trip per connection isn't a more practical benefit in a world where we now expect pretty much everything to be encrypted in transit.
- marcosdumay 2y ago
  > feels more elegant Yet, it's not. That kind of elegance ends up in application level implementations like STARTTLS and socket data becoming unusable on the middle of a connection while both parties scramble to decide what to throw away and what to interpret. It does really look like the real thing, but it's decoupling things that are very coupled by definition.
- markhahn 2y ago
  maybe the lesson is that "composable" shouldn't mean "just nested".

eru 2y ago

And TCP wants to pretend that all your data arrives as a linear stream one after the other; and keeps you from seeing package n+1, if you haven't received package n, yet. That's useful for some things, but often you can make use of package n+1, even when package n hasn't arrived, yet. For example, when you are transferring a large file, you could use erasure encoding to just automatically deal with 5% package loss. (Or you could use a fountain code to deal with variable packet loss, and the sender just keeps sending until the receiver says "I'm done" for the whole file, instead of ack-ing individual packages. Fountain codes are how deep space probes send their data back. Latency is pretty terrible out to Jupiter or Mars.)

Aurornis 2y ago
> And TCP wants to pretend that all your data arrives as a linear stream one after the other; That’s one of the reasons people use TCP. > For example, when you are transferring a large file, you could use erasure encoding to just automatically deal with 5% package loss. It’s never that simple. You can’t just add some erasure coding and have it automatically solve your problems. You now have to build an entire protocol around those packets to determine and track their order. You also need mechanisms to handle the case where packet loss exceeds what you can recover, which involves either restarting the transfer or a retransmission mechanism. The number of little details you have to handle quickly explodes in complexity. Even in the best case scenario, you’d be paying a price to handle erasure coding on one end, the extra bandwidth of the overhead, and then decoding on the receiving end. That’s a lot of complexity, engineering, debugging, and opportunities to introduce bugs, and for what gain? In the file transfer example, what would you actually gain by rolling your own entire transmission protocol with all this overhead? > Fountain codes are how deep space probes send their data back. Latency is pretty terrible out to Jupiter or Mars.) Fountain codes and even general erasure codes are not the right tool for this job. The loss of a digital packetized channel across the internet is very different than a noisy analog channel sent through space.
- eru 2y ago
  > It’s never that simple. You can’t just add some erasure coding and have it automatically solve your problems. You now have to build an entire protocol around those packets to determine and track their order. Yes, and you'd want to mostly build this extra protocol only once, and stick it into a library. > You also need mechanisms to handle the case where packet loss exceeds what you can recover, which involves either restarting the transfer or a retransmission mechanism. Or you can use a fountain code, and just keep transmitting. > The number of little details you have to handle quickly explodes in complexity. Even in the best case scenario, you’d be paying a price to handle erasure coding on one end, the extra bandwidth of the overhead, and then decoding on the receiving end. Well, you can also use a simpler mechanism: you transmit all the packages from A to B once, then at the end B tells A which packages get lost, and A sends those again. Repeat until you have everything. That way needs more back-and-forth communication, but doesn't need any fancy error correcting code. > Fountain codes and even general erasure codes are not the right tool for this job. The loss of a digital packetized channel across the internet is very different than a noisy analog channel sent through space. Yes, the loss model is different. However you can eg interleave your bits to get something that close enough to work. The main reason you don't need to use error correcting codes, is that transmitting feedback to the sender is typically a lot cheaper on the internet than in outer space. Not least because even bad latencies are typically measured in seconds at most, not minutes or hours. (You could however use these codes when for some reason you have a very asymmetrical link. Eg if you have internet via geo-stationary satellite, or if one sender is broadcast to lots and lots of different receivers, and doesn't want to deal with each of them individually.)
jakjak123 2y ago
> And TCP wants to pretend that all your data arrives as a linear stream one after the other; This is actually very nice for an application developer. Probably a tiny bit less nice if you are running popular services taking 100k req/s.
- eru 2y ago
  Yes, that is convenient. But it's an abstraction you don't always need, but still always pay for. That's why I would suggest that were possible the kernel should only expose something at most as high level as UDP, and we build the TCP abstractions on top of that in user level. Of course, you wouldn't want to re-write everything from scratch all the time: you'd use libraries. The nice thing about user level libraries is that you can swap them out for better versions or for different abstractions without any privileged access.

sgt 2y ago

Is NB-IoT actually being used these days? I remember it was hyped up but then the buzz disappeared.

tliltocatl 2y ago
It definitely does, at least in Europe, not sure how things are on the other side of the pond. For the user equipment there is no real advantage over LTE-M, actually is LTE-M better for our case (send a single packet a day and go to sleep). But then I don't see the picture from the operator side, maybe the radio savings actually matter for them. Some operators either don't have LTE-M at all or don't allow PSM on LTE-M because screw you that's why. On NB-IoT PSM is mandatory at least. And yes, LoRa/Sigfox/other non-cellular LPWAN might technically be better IF you provide the base stations. Nobody wants to do that. With cellular you can just mail the device and cross your fingers the operators haven't misconfigured roaming.
- sgt 2y ago
  That's quite interesting. And the promised battery lifespans of NB-IoT device, did it hold some truth? In the beginning we heard about 10 year life spans for simple well designed devices.
  - jjoonathan 2y ago
    We have millions of devices in the field older than that on the same D-cell LiSOCl2 battery they shipped with. Some of them even take magnetic measurements with powered excitation!
  - tliltocatl 2y ago
    You'll have to wait 10 years to see. But on paper it does (we aim for 15) and we have devices in field for some three years. But for the "simple well-designed" - the problem is, the whole thing is a black box for application developer, and not a bugless one. So much of the "well-designed" part is to figure out when is the right time to pull modem's proverbial power cord. If you don't, it'll gladly eat up your battery in 48 hours after a cell handover goes wrong with no warning.

cma 2y ago

> If you move into a location with worse coverage, your latency increases, but TCP will assume packet loss due to congestion and reduce bandwidth. And in general, loss-based congestion control just doesn't work at this point. Don't lower levels of wifi below TCP now do often their own retransmit, even if it is just from weak/noisy signal and not a sensed collision?

saghm 2y ago

> TCP will waste roundtrips on handshakes. And then some extra on MTU discovery. This is especially true when using TLS (which is at least a major use case nowadays if not the default assumption); the TCP handshake has to complete before the TLS handshake can occur, whereas QUIC has protocol-level support for doing TLS negotiation as part of the initial handshake. Having the network protocol and the encryption be composable rather than tightly coupled feels more elegant, it's hard for me to convince myself that being able to save a round trip per connection isn't a more practical benefit in a world where we now expect pretty much everything to be encrypted in transit.

marcosdumay 2y ago
> feels more elegant Yet, it's not. That kind of elegance ends up in application level implementations like STARTTLS and socket data becoming unusable on the middle of a connection while both parties scramble to decide what to throw away and what to interpret. It does really look like the real thing, but it's decoupling things that are very coupled by definition.
markhahn 2y ago
maybe the lesson is that "composable" shouldn't mean "just nested".