3 ms·

> the endpoints need to use large buffers to hold a copy of all the unacknowledged data, as is required by the TCP protocol. It makes me wonder if anyone has t

> the endpoints need to use large buffers to hold a copy of all the unacknowledged data, as is required by the TCP protocol. It makes me wonder if anyone has tried to break down the layers to optimize this. In the fairly common case of serving a file off of long-term storage you can jut fetch the old data if needed (likely from the page cache anyways, but still better than duplicating it) and some encryption algorithms are seekable so you can redo the encryption as well. Right now the kernel doesn't really have a choice but to buffer all unacknowledged data as the UNIX socket API has no provision for requesting old data from the writer. But a smarter API could put the application in charge of making that data available as required and avoid the need for extra copies in common cases. I know that Netflix did lots of work with the FreeBSD kernel for file to socket and eventually adding in-kernel TLS to remove user space from the equation. But I don't know if they went as far to remove the socket buffers.

cyberax 2y ago
> It makes me wonder if anyone has tried to break down the layers to optimize this. Yep. There was a bunch of proxy servers that optimized HTTP for satellite service. I used Globax back in the day to speed up one-way satellite service: https://web.archive.org/web/20040602203838/http://globax.info/shorten.html https://web.archive.org/web/20040602203838/http://globax.inf... Back then traffic was around 10 cents per megabyte in my city, so satellite service was a good way to shave off these costs.
lxgr 2y ago
There aren't necessarily extra copies even when just using TCP, thanks to sendfile(2) and similar mechanisms. Buffer size isn't that much of an issue either, given the relatively low latencies involved and that you can indicate which parts exactly are missing pretty accurately these days with selective TCP acknowledgements, so you'll need at most a few round trips to identify these to the sender and eventually receive them. Practically, you'll probably not see much loss anyway, for better or worse: TCP historically interpreted packet loss as congestion, instead of actual non-congestion-induced loss on the physical layer. This is why most lower-layer protocols with higher error rates than Ethernet usually implement some sort of lower-layer ARQ to present themselves as "Ethernet-like" to upper layers in terms of error/loss rate, and this in turn has made loss-tolerant TCP (such as BBR, as described in the article) less of a research priority, I believe.
moandcompany 2y ago
This is an "old" problem that has historically been addressed through things like "Performance Enhancing Proxies (PEPs)" that are defined in RFC 3135 and RFC 3449. (https://en.wikipedia.org/wiki/Performance-enhancing_proxy https://en.wikipedia.org/wiki/Performance-enhancing_proxy) In internet-style communications, such as routing IP traffic over satellite links to low-earth-orbit or GEO, with much longer round-trip times, link latency is substantially higher than most terrestrial wired or wireless applications and acknowledgements required as part of TCP take much longer to facilitate. PEPs as an example augment the connection allowing end-user/client devices in the network with inline-PEPs to retain their normal network settings and perform the task of running or starting sessions with higher TCP-window sizes, as a method for improving overall throughput. The utility of a PEP, or PEP-acting device goes up when you imagine multiple devices, or a network of devices attached to a satellite communications terminal for WAN/backhaul connections as the link's performance can be managed at one point versus on all downstream client devices.
- lxgr 2y ago
  TIL that that's what they're called. Thank you! Do you know if they became obsolete due to modern TCP stacks handling LFNs better or for some other reason? I could imagine them being quite useful for high-loss, high-latency paths (i.e. in addition to LFNs in conjunction with poorly tuned TCP implementation), but most wireless protocols I know (802.11, GPRS and beyond etc.) just implement an ARQ process that masks the loss at the packet or lower layer. So maybe between that and LFN-aware TCPs, there wasn't much left for them to improve to justify the additional complexity?
  - moandcompany 2y ago
    It's been a long time since I've paid attention to that world, but AFAIK, PEPs are still used and essential equipment for internet-style communication (i.e. TCP over IP) via GEO satellites. It looks like in this 2022 blog post evaluating latency and throughput over Starlink, they concluded that PEPs were not being used in the Starlink network (and probably unnecessary) due to the lower latency characteristics from use of LEO satellites. They also mention that PEPs are (still) commonly employed by GEO-satcom based operators. https://blog.apnic.net/2022/11/28/fact-checking-starlinks-performance-figures/ https://blog.apnic.net/2022/11/28/fact-checking-starlinks-pe...

cyberax 2y ago

> It makes me wonder if anyone has tried to break down the layers to optimize this. Yep. There was a bunch of proxy servers that optimized HTTP for satellite service. I used Globax back in the day to speed up one-way satellite service: https://web.archive.org/web/20040602203838/http://globax.info/shorten.html https://web.archive.org/web/20040602203838/http://globax.inf... Back then traffic was around 10 cents per megabyte in my city, so satellite service was a good way to shave off these costs.

lxgr 2y ago

There aren't necessarily extra copies even when just using TCP, thanks to sendfile(2) and similar mechanisms. Buffer size isn't that much of an issue either, given the relatively low latencies involved and that you can indicate which parts exactly are missing pretty accurately these days with selective TCP acknowledgements, so you'll need at most a few round trips to identify these to the sender and eventually receive them. Practically, you'll probably not see much loss anyway, for better or worse: TCP historically interpreted packet loss as congestion, instead of actual non-congestion-induced loss on the physical layer. This is why most lower-layer protocols with higher error rates than Ethernet usually implement some sort of lower-layer ARQ to present themselves as "Ethernet-like" to upper layers in terms of error/loss rate, and this in turn has made loss-tolerant TCP (such as BBR, as described in the article) less of a research priority, I believe.

moandcompany 2y ago

This is an "old" problem that has historically been addressed through things like "Performance Enhancing Proxies (PEPs)" that are defined in RFC 3135 and RFC 3449. (https://en.wikipedia.org/wiki/Performance-enhancing_proxy https://en.wikipedia.org/wiki/Performance-enhancing_proxy) In internet-style communications, such as routing IP traffic over satellite links to low-earth-orbit or GEO, with much longer round-trip times, link latency is substantially higher than most terrestrial wired or wireless applications and acknowledgements required as part of TCP take much longer to facilitate. PEPs as an example augment the connection allowing end-user/client devices in the network with inline-PEPs to retain their normal network settings and perform the task of running or starting sessions with higher TCP-window sizes, as a method for improving overall throughput. The utility of a PEP, or PEP-acting device goes up when you imagine multiple devices, or a network of devices attached to a satellite communications terminal for WAN/backhaul connections as the link's performance can be managed at one point versus on all downstream client devices.

lxgr 2y ago
TIL that that's what they're called. Thank you! Do you know if they became obsolete due to modern TCP stacks handling LFNs better or for some other reason? I could imagine them being quite useful for high-loss, high-latency paths (i.e. in addition to LFNs in conjunction with poorly tuned TCP implementation), but most wireless protocols I know (802.11, GPRS and beyond etc.) just implement an ARQ process that masks the loss at the packet or lower layer. So maybe between that and LFN-aware TCPs, there wasn't much left for them to improve to justify the additional complexity?
- moandcompany 2y ago
  It's been a long time since I've paid attention to that world, but AFAIK, PEPs are still used and essential equipment for internet-style communication (i.e. TCP over IP) via GEO satellites. It looks like in this 2022 blog post evaluating latency and throughput over Starlink, they concluded that PEPs were not being used in the Starlink network (and probably unnecessary) due to the lower latency characteristics from use of LEO satellites. They also mention that PEPs are (still) commonly employed by GEO-satcom based operators. https://blog.apnic.net/2022/11/28/fact-checking-starlinks-performance-figures/ https://blog.apnic.net/2022/11/28/fact-checking-starlinks-pe...