8 ms·

Note there is no intrinsic reason running multiple streams should be faster than one [EDIT: "at this scale"]. It almost always indicates some bottleneck in the

by digiown 9mo ago

Note there is no intrinsic reason running multiple streams should be faster than one [EDIT: "at this scale"]. It almost always indicates some bottleneck in the application or TCP tuning. (Though, very fast links can overwhelm slow hardware, and ISPs might do some traffic shaping too, but this doesn't apply to local links). SSH was never really meant to be a high performance data transfer tool, and it shows. For example, it has a hardcoded maximum receive buffer of 2MiB (separate from the TCP one), which drastically limits transfer speed over high BDP links (even a fast local link, like the 10gbps one the author has). The encryption can also be a bottleneck. hpn-ssh [1] aims to solve this issue but I'm not so sure about running an ssh fork on important systems. 1. https://github.com/rapier1/hpn-ssh https://github.com/rapier1/hpn-ssh

mprovost 9mo ago
In general TCP just isn't great for high performance. In the film industry we used to use a commercial product Aspera (now owned by IBM) which emulated ftp or scp but used UDP with forward error correction (instead of TCP retransmission). You could configure it to use a specific amount of bandwidth and it would just push everything else off the network to achieve it.
- digiown 9mo ago
  There's an open source implementation that does something similar but for a more specific use case: https://github.com/apernet/tcp-brutal https://github.com/apernet/tcp-brutal There's gotta be a less antisocial way though. I'd say using BBR and increasing the buffer sizes to 64 MiB does the trick in most cases.
  - tclancy 9mo ago
    Have you tried searching for "tcp-kind"?
  - Onavo 9mo ago
    Looks unmaintained. Can we throw a bunch of AI agents at it? This sounds like a pretty tightly defined problem, much better than wasting tokens on re-inventing web browsers.
- pezgrande 9mo ago
  Was the torrent protocol considered at some point? Always surprised how little presence has in the industry considering how good the technology is.
  - ambicapter 9mo ago
    torrent is great for many-to-one type downloads but I assume GP is talking about single machine to single machine transfers.
  - gruez 9mo ago
    If you strip out the swarm logic (ie. downloading from multiple peers), you're just left with a protocol that transfers big files via chunks, so there's no reason that'd be faster than any other sort of download manager that supports multi-thread downloads. https://en.wikipedia.org/wiki/Download_manager https://en.wikipedia.org/wiki/Download_manager
  - KaiserPro 9mo ago
    Aspera did the chunking and encryption for you, and it looked and acted like SFTP. The cost of leaking data was/is catastrophic (as in company ending) So paying a bit of money to guarantee that your data was being sent to the right place (point to point) and couldn't leak was a worthwhile tradeoff. For Point to point transfer torrenting is a lot higher overhead than you want. plus most clients have an anti-leaching setting, so you'd need not only a custom client, but a custom protocol as well. The idea is sound though, have an index file with and then a list of chunks to pull over multiple TCP connections.
- adolph 9mo ago
  Aspera's FASP [0] is very neat. One drawback to it is that the TCP stuff not being done the traditional way must be done on CPU. Say if one packet is missing or if packets are sent out of order, the Aspera client fixes those instead of all that being done as TCP. As I understand it, this is also the approach of WEKA.io [1]. Another approach is RDMA [2] used by storage systems like Vast which pushes those order and resend tasks to NICs that support RDMA so that applications can read and write directly to the network instead of to system buffers. 0. https://en.wikipedia.org/wiki/Fast_and_Secure_Protocol https://en.wikipedia.org/wiki/Fast_and_Secure_Protocol 1. https://docs.weka.io/weka-system-overview/weka-client-and-mount-modes#the-weka-client https://docs.weka.io/weka-system-overview/weka-client-and-mo... 2. https://en.wikipedia.org/wiki/Remote_direct_memory_access https://en.wikipedia.org/wiki/Remote_direct_memory_access
  - mprovost 9mo ago
    FASP uses forward error correction instead of retransmission. So instead of waiting for something not to show up on the other end and sending it again, it calculates parity and transmits slightly more data up front, with enough redundancy that the receiving end is capable of reconstructing any missing bits. This is basically how all storage systems work, not just Weka. You calculate enough parity bits to be able to reconstruct the missing data when a drive fails. The more disks you have, the smaller the parity overhead is. Object storage like S3 does this on a massive scale. With a network transfer you typically only need a few percent, unless it's really lossy like Wifi, in which case standards like 802.11n are doing FEC for you to reduce retransmissions at the TCP layer.
    - adolph 9mo ago
      In RDMA are the NICs able to perform the reconstruction or does that use a different mechanism to avoid CPU?
      - mprovost 9mo ago
        Usually RDMA is over a network that is supposed to be lossless, but it does have checksums to detect corruption and recovers with retransmission. Infiniband NICs handle all of that.
- nh2 9mo ago
  What does "high performance" mean here? I get 40 Gbit/s over a single localhost TCP stream on my 10 years old laptop with iperf3. So the TCP does not seem to be a bottleneck if 40 Gbit/s is "high" enough, which it probably is currently for most people. I have also seen plenty situations in which TCP is faster than UDP in datacenters. For example, on Hetzner Cloud VMs, iperf3 gets me 7 Gbit/s over TCP but only 1.5 Gbit/s over UDP. On Hetzner dedicated servers with 10 Gbit links, I get 10 Gbit/s over TCP but only 4.5 Gbit/s over UDP. But this could also be due to my use of iperf3 or its implementation. I also suspect that TCP being a protocol whose state is inspectable by the network equipment between endpoints allows implementing higher performance, but I have not validated if that is done.
  - KaiserPro 9mo ago
    Aspera was/is designed for high latency links. Ie sending multi terabytes from london to new Zealand, or LA For that use case, Aspera was the best tool for the job. It's designed to be fast over links that single TCP streams couldn't You could, if you were so bold, stack up multiple TCP links and send data down those. You got the same speed, but possible not the same efficiency. It was a fucktonne cheaper to do though.
  - mprovost 9mo ago
    High performance means transferring files from NZ to a director's yacht in the Mediterranean with a 40Mbps satellite link and getting 40Mbps, to the point that the link is unusable for anyone else.
  - wtallis 9mo ago
    > I get 40 Gbit/s over a single localhost TCP stream on my 10 years old laptop with iperf3. Do you mean literally just streaming data from one process to another on the same machine, without that data ever actually transiting a real network link? There's so many caveats to that test that it's basically worthless for evaluating what could happen on a real network.
    - nh2 9mo ago
      Yes. Why? To measure other overhead of what's claimed (TCP the protocol being slow), one should exclude other things that necessarily affect alternative protocols as well (e.g. latency) as much as possible, which is what this does.
- robaato 9mo ago
  So what do you use now in film industry?
  - mprovost 9mo ago
    I suspect mostly Aspera because there are still no good alternatives.
  - magarnicle 9mo ago
    I'm in a tiny part of the film industry. Bigger clients lend us licenses to Aspera and FileCatalyst when receiving files from them, but for our own trans-oceanic transfers I dug up an ancient program called Tsunami UDP and fixed it up just enough.
dekhn 9mo ago
Single file overheads (opening millions of tiny files whose metadata is not in the OS cache and reading them) appears to be an intrinsic reason (intrinsic to the OS, at least).
- pixl97 9mo ago
  IOPs and disk read depth are common limits. Depending on what you're doing it can be faster to leave your files in a solid archive that is less likely to be fragmented and get contiguous reads.
- PunchyHamster 9mo ago
  the majority of that will be big files. And to NVMe it is VERY fast even if you run single threaded 10Gbit should be easy
yegle 9mo ago
The author tried running rsyncd demon so it's not _just_ the ssh protocol.
- ruperthair 9mo ago
  Great point, and even beyond that I think (based on the paths) it was just a command line invocation, with something like NFS handling all the networking.
Saris 9mo ago
Wouldn't lots of streams speed up transfers of thousands of small files?
- digiown 9mo ago
  If the application handles them serially, then yeah. But one can imagine the application opening files in threads, buffering them, and then finally sending it at full speed, so in that sense it is an application issue. If you truly have millions of small files, you're more likely to be bottlenecked by disk IO performance rather than application or network, though. My primary use case for ssh streams is zfs send, which is mostly bottlenecked by ssh itself.
  - catdog 9mo ago
    It's an application issue but implementation wise it's probably way more straightforward to just open a separate network connection per thread.
oceanplexian 9mo ago
Uhh.. I work with this stuff daily and there are a LOT of intrinsic reasons a single stream would be slower than running multiple: MPLS ECMP hashing you over a single path, a single loss event with a high BDP causing congestion control to kick in for a single flow, CPU IRQ affinity, probably many more I’m not thinking like the inner workings of NIC offloading queues. Source: Been in big tech for roughly ten years now trying to get servers to move packets faster
- digiown 9mo ago
  Ha, it sounds like the best way to learn something is to make a confident and incorrect claim :) > MPLS ECMP hashing you over a single path This is kinda like the traffic shaping I was talking about though, but fair enough. It's not an inherent limitation of a single stream, just a consequence of how your network is designed. > a single loss event with a high BDP I thought BBR mitigates this. Even if it doesn't, I'd still count that as a TCP stack issue. At a large enough scale I'd say you are correct that multiple streams is inherently easier to optimize throughput for. But probably not a single 1-10gb link though.
  - PunchyHamster 9mo ago
    > This is kinda like the traffic shaping I was talking about though, but fair enough. It's not an inherent limitation of a single stream, just a consequence of how your network is designed. It is. one stream gets you traffic of one path to the infrastructure. Multiple streams get you multiple and possibly also hit different servers to accelerate it even more. Just the limitation isn't hardware but "our networking device have 4 10Gbit ports instead of single 40Gbit port" Especially if link is saturated, you'd be essentially taking n-times your "fair share" of bandwidth on link.
Aurornis 9mo ago
> Note there is no intrinsic reason running multiple streams should be faster than one. The issue is the serialization of operations. There is overhead for each operation which translates into dead time between transfers. However there are issues that can cause singular streams to underperform multiple streams in the real world once you reach a certain scale or face problems like packet loss.
- wmf 9mo ago
  The ideal solution to that is pipelining but it can be complex to implement.
- nh2 9mo ago
  Is it certain that this is the reason? rsync's man page says "pipelining of file transfers to minimize latency costs" and https://rsync.samba.org/how-rsync-works.html https://rsync.samba.org/how-rsync-works.html says "Rsync is heavily pipelined". If pipelining is really in rsync, there should be no "dead time between transfers".
  - spockz 9mo ago
    I’m not sure why, but just like with scp, I’ve achieved significant speeds ups by tarring the directory first (optionally compressing it), transferring and then decompressing. Maybe because it makes the tar and submit, and the receive, untar/uncompress, happen on different threads?
    - lelandbatey 9mo ago
      It's typically a disk-latency thing, as just stat-ing the many files in a directory can have significant latency implications (especially on spinning HDDs) vs opening a single file (the tar) and read-()ing that one file in memory before writing to the network. If copying a folder with many files is slower than tarring that folder and the moving the tar (but not counting the untar) then disk latency is your bottleneck.
      - 9mo ago
softfalcon 9mo ago
> It almost always indicates some bottleneck in the application or TCP tuning. Yeah, this has been my experience with low-overhead streams as well. Interestingly, I see a ubiquity of this "open more streams to send more data" pattern all over the place for file transfer tooling. Recent ones that come to mind have been BackBlaze's CLI (B2) and taking a peek at Amazon's SDK for S3 uploads with Wireshark. (What do they know that we don't seem to think we know?) It seems like they're all doing this? Which is maybe odd, because when I analyse what Plex or Netflix is doing, it's not the same? They do what you're suggesting, tune the application + TCP/UDP stack. Though that could be due to their 1-to-1 streaming use case. There is overhead somewhere and they're trying to get past it via semi-brute-force methods (in my opinion). I wonder if there is a serialization or loss handling problem that we could be glossing over here?
- akdev1l 9mo ago
  not sure about B2 but AWS S3 SDK not assuming that people will do any tuning makes total sense cuz in my experience no one is doing that tbh
  - slightlygrilled 9mo ago
    I’ve found aws s3 it’s always been painful to get any good speed out of it unless it’s massive files you’re moving. It’s base line tuning seems to just assume large files and does no auto scaling and it’s mostly single threaded. Then even when tuning it’s still painfully slow, again seemly limited by its cpu processing and mostly on a single thread, highly annoying. Especially when you’re running it on a high core, fast storage, large internet connection machine. Just feels like there is a large amount of untapped potential in the machines…
    - odo1242 9mo ago
      It’s almost certainly also tuned to prevent excessive or “spiky” traffic to their service.
- digiown 9mo ago
yason 9mo ago
Note there is no intrinsic reason running multiple streams should be faster than one If the server side scales (as cloud services do) it might end up using different end points for the parallel connections and saturate the bandwidth better. One server instance might be serving other clients as well and can't fill one particular client's pipe entirely.
patmorgan23 9mo ago
I mean isn't a single TCP connections throughput limited by the latency? Which is why in high(er) latency WAN links you generally want to open multiple connections for large file transfers. https://wintelguy.com/wanperf.pl https://wintelguy.com/wanperf.pl
- adrian_b 9mo ago
  Only simpler transfer protocols, like TFTP, are limited by latency. The whole reason for the existence of TCP is to overcome the throughput limit determined by latency. On a network with negligible latency there is no need for TCP (you could just send each packet only after the previous is acknowledged, but the higher is the throughput of your network interface, the less likely is that the latency can be negligible). However, for latency to not matter, the TCP windows must be large enough (i.e. the amount of data that is sent before an acknowledge is received, which happens after a delay caused by latency). I use Windows very rarely today, so I do not know its current status, but until the Windows XP days it was very frequent for Windows computers to have very small default TCP window sizes, which caused low throughput on high-latency networks, so on such networks they had to be reconfigured. On high-latency networks, opening multiple connections is just a workaround for not having appropriate network settings. However, even when your own computer is configured optimally, opening multiple connections can be a workaround against various kinds of throttling implemented either by some intermediate ISP or by the destination server, though nowadays most rate limits are applied globally, to all connections from the same IP address, in order to make this workaround ineffective.
nh2 9mo ago
> has a hardcoded maximum receive buffer of 2MiB For completeness, I want to add: The 2MiB are per SSH "channel" -- the SSH protocol multiplexes multiple independent transmission channels over TCP [1], and each one has its own window size. rsync and `cat | ssh | cat` only use a single channel, so if their counterparty is an OpenSSH sshd server, their throughput is limited by the 2MiB window limit. rclone seems to be able to use multiple ssh channels over a single connection; I believe this is what the `--sftp-concurrency` setting controls. Some more discussion about the 2MiB limit and links to work for upstreaming a removal of these limits can be found in my post [3]. Looking into it just now, I found that the SSH protocol itself already supports dynamically growing per-channel window sizes with `CHANNEL_WINDOW_ADJUST`, and OpenSSH seems to generally implement that. I don't fully grasp why it doesn't just use that to extend as needed. I also found that there's an official `no-flow-control` extension with the description > channel behaves as if all window sizes are infinite. > > This extension is intended for, but not limited to, use by file transfer applications that are only going to use one channel and for which the flow control provided by SSH is an impediment, rather than a feature. So this looks exactly as designed for rsync. But no software implements this extension! I wrote those things down in [4]. It is frustrating to me that we're only a ~200 line patch away from "unlimited" instead of shitty SSH transfer speeds -- for >20 years! [1]: https://datatracker.ietf.org/doc/html/rfc4254#section-5 https://datatracker.ietf.org/doc/html/rfc4254#section-5 [2]: https://rclone.org/sftp/#sftp-concurrency https://rclone.org/sftp/#sftp-concurrency [3]: https://news.ycombinator.com/item?id=40856136 https://news.ycombinator.com/item?id=40856136 [4]: https://github.com/djmdjm/openssh-portable-wip/pull/4#issuecomment-3837136128 https://github.com/djmdjm/openssh-portable-wip/pull/4#issuec...
einpoklum 9mo ago
> Note there is no intrinsic reason running multiple streams should be faster than one Inherent reasons or no, it's been my experience across multiple protocols, applications, network connections and environments, and machines on both ends, that, _in fact_, splitting data up and operating using multiple streams is significantly faster. So, ok, it might not be because of an "inherent reason", but we still have to deal with it in real life.
bscphil 9mo ago
> TCP tuning I think a lot of file transfer issues that occur outside of the corporate intranet world involve hardware that you don't fully control on (at least) one hand. In science, for example, transferring huge amounts of data over long distances is pretty common, and I've had to do this on boxes that had poor TCP buffer configurations. Being able to multiplex your streams in situations like this is invaluable and I'd love to see more open source software that does this effectively, especially if it can punch through a firewall.
jolmg 9mo ago
> SSH was never really meant to be a high performance data transfer tool, and it shows. A practical example can be `ssh -X` vs X11 over Wireguard. The lag is obvious with the former, but X11 windows from remote clients can be indistinguishable performance-wise from those of local ones with the latter.

by digiown 9mo ago

mprovost 9mo ago

In general TCP just isn't great for high performance. In the film industry we used to use a commercial product Aspera (now owned by IBM) which emulated ftp or scp but used UDP with forward error correction (instead of TCP retransmission). You could configure it to use a specific amount of bandwidth and it would just push everything else off the network to achieve it.

digiown 9mo ago
There's an open source implementation that does something similar but for a more specific use case: https://github.com/apernet/tcp-brutal https://github.com/apernet/tcp-brutal There's gotta be a less antisocial way though. I'd say using BBR and increasing the buffer sizes to 64 MiB does the trick in most cases.
- tclancy 9mo ago
  Have you tried searching for "tcp-kind"?
- Onavo 9mo ago
  Looks unmaintained. Can we throw a bunch of AI agents at it? This sounds like a pretty tightly defined problem, much better than wasting tokens on re-inventing web browsers.
pezgrande 9mo ago
Was the torrent protocol considered at some point? Always surprised how little presence has in the industry considering how good the technology is.
- ambicapter 9mo ago
  torrent is great for many-to-one type downloads but I assume GP is talking about single machine to single machine transfers.
- gruez 9mo ago
  If you strip out the swarm logic (ie. downloading from multiple peers), you're just left with a protocol that transfers big files via chunks, so there's no reason that'd be faster than any other sort of download manager that supports multi-thread downloads. https://en.wikipedia.org/wiki/Download_manager https://en.wikipedia.org/wiki/Download_manager
- KaiserPro 9mo ago
  Aspera did the chunking and encryption for you, and it looked and acted like SFTP. The cost of leaking data was/is catastrophic (as in company ending) So paying a bit of money to guarantee that your data was being sent to the right place (point to point) and couldn't leak was a worthwhile tradeoff. For Point to point transfer torrenting is a lot higher overhead than you want. plus most clients have an anti-leaching setting, so you'd need not only a custom client, but a custom protocol as well. The idea is sound though, have an index file with and then a list of chunks to pull over multiple TCP connections.
adolph 9mo ago
Aspera's FASP [0] is very neat. One drawback to it is that the TCP stuff not being done the traditional way must be done on CPU. Say if one packet is missing or if packets are sent out of order, the Aspera client fixes those instead of all that being done as TCP. As I understand it, this is also the approach of WEKA.io [1]. Another approach is RDMA [2] used by storage systems like Vast which pushes those order and resend tasks to NICs that support RDMA so that applications can read and write directly to the network instead of to system buffers. 0. https://en.wikipedia.org/wiki/Fast_and_Secure_Protocol https://en.wikipedia.org/wiki/Fast_and_Secure_Protocol 1. https://docs.weka.io/weka-system-overview/weka-client-and-mount-modes#the-weka-client https://docs.weka.io/weka-system-overview/weka-client-and-mo... 2. https://en.wikipedia.org/wiki/Remote_direct_memory_access https://en.wikipedia.org/wiki/Remote_direct_memory_access
- mprovost 9mo ago
  FASP uses forward error correction instead of retransmission. So instead of waiting for something not to show up on the other end and sending it again, it calculates parity and transmits slightly more data up front, with enough redundancy that the receiving end is capable of reconstructing any missing bits. This is basically how all storage systems work, not just Weka. You calculate enough parity bits to be able to reconstruct the missing data when a drive fails. The more disks you have, the smaller the parity overhead is. Object storage like S3 does this on a massive scale. With a network transfer you typically only need a few percent, unless it's really lossy like Wifi, in which case standards like 802.11n are doing FEC for you to reduce retransmissions at the TCP layer.
  - adolph 9mo ago
    In RDMA are the NICs able to perform the reconstruction or does that use a different mechanism to avoid CPU?
    - mprovost 9mo ago
      Usually RDMA is over a network that is supposed to be lossless, but it does have checksums to detect corruption and recovers with retransmission. Infiniband NICs handle all of that.
nh2 9mo ago
What does "high performance" mean here? I get 40 Gbit/s over a single localhost TCP stream on my 10 years old laptop with iperf3. So the TCP does not seem to be a bottleneck if 40 Gbit/s is "high" enough, which it probably is currently for most people. I have also seen plenty situations in which TCP is faster than UDP in datacenters. For example, on Hetzner Cloud VMs, iperf3 gets me 7 Gbit/s over TCP but only 1.5 Gbit/s over UDP. On Hetzner dedicated servers with 10 Gbit links, I get 10 Gbit/s over TCP but only 4.5 Gbit/s over UDP. But this could also be due to my use of iperf3 or its implementation. I also suspect that TCP being a protocol whose state is inspectable by the network equipment between endpoints allows implementing higher performance, but I have not validated if that is done.
- KaiserPro 9mo ago
  Aspera was/is designed for high latency links. Ie sending multi terabytes from london to new Zealand, or LA For that use case, Aspera was the best tool for the job. It's designed to be fast over links that single TCP streams couldn't You could, if you were so bold, stack up multiple TCP links and send data down those. You got the same speed, but possible not the same efficiency. It was a fucktonne cheaper to do though.
- mprovost 9mo ago
  High performance means transferring files from NZ to a director's yacht in the Mediterranean with a 40Mbps satellite link and getting 40Mbps, to the point that the link is unusable for anyone else.
- wtallis 9mo ago
  > I get 40 Gbit/s over a single localhost TCP stream on my 10 years old laptop with iperf3. Do you mean literally just streaming data from one process to another on the same machine, without that data ever actually transiting a real network link? There's so many caveats to that test that it's basically worthless for evaluating what could happen on a real network.
  - nh2 9mo ago
    Yes. Why? To measure other overhead of what's claimed (TCP the protocol being slow), one should exclude other things that necessarily affect alternative protocols as well (e.g. latency) as much as possible, which is what this does.
robaato 9mo ago
So what do you use now in film industry?
- mprovost 9mo ago
  I suspect mostly Aspera because there are still no good alternatives.
- magarnicle 9mo ago
  I'm in a tiny part of the film industry. Bigger clients lend us licenses to Aspera and FileCatalyst when receiving files from them, but for our own trans-oceanic transfers I dug up an ancient program called Tsunami UDP and fixed it up just enough.

dekhn 9mo ago

Single file overheads (opening millions of tiny files whose metadata is not in the OS cache and reading them) appears to be an intrinsic reason (intrinsic to the OS, at least).

pixl97 9mo ago
IOPs and disk read depth are common limits. Depending on what you're doing it can be faster to leave your files in a solid archive that is less likely to be fragmented and get contiguous reads.
PunchyHamster 9mo ago
the majority of that will be big files. And to NVMe it is VERY fast even if you run single threaded 10Gbit should be easy

yegle 9mo ago

The author tried running rsyncd demon so it's not _just_ the ssh protocol.

ruperthair 9mo ago
Great point, and even beyond that I think (based on the paths) it was just a command line invocation, with something like NFS handling all the networking.

Saris 9mo ago

Wouldn't lots of streams speed up transfers of thousands of small files?

digiown 9mo ago
If the application handles them serially, then yeah. But one can imagine the application opening files in threads, buffering them, and then finally sending it at full speed, so in that sense it is an application issue. If you truly have millions of small files, you're more likely to be bottlenecked by disk IO performance rather than application or network, though. My primary use case for ssh streams is zfs send, which is mostly bottlenecked by ssh itself.
- catdog 9mo ago
  It's an application issue but implementation wise it's probably way more straightforward to just open a separate network connection per thread.

oceanplexian 9mo ago

Uhh.. I work with this stuff daily and there are a LOT of intrinsic reasons a single stream would be slower than running multiple: MPLS ECMP hashing you over a single path, a single loss event with a high BDP causing congestion control to kick in for a single flow, CPU IRQ affinity, probably many more I’m not thinking like the inner workings of NIC offloading queues. Source: Been in big tech for roughly ten years now trying to get servers to move packets faster

digiown 9mo ago
Ha, it sounds like the best way to learn something is to make a confident and incorrect claim :) > MPLS ECMP hashing you over a single path This is kinda like the traffic shaping I was talking about though, but fair enough. It's not an inherent limitation of a single stream, just a consequence of how your network is designed. > a single loss event with a high BDP I thought BBR mitigates this. Even if it doesn't, I'd still count that as a TCP stack issue. At a large enough scale I'd say you are correct that multiple streams is inherently easier to optimize throughput for. But probably not a single 1-10gb link though.
- PunchyHamster 9mo ago
  > This is kinda like the traffic shaping I was talking about though, but fair enough. It's not an inherent limitation of a single stream, just a consequence of how your network is designed. It is. one stream gets you traffic of one path to the infrastructure. Multiple streams get you multiple and possibly also hit different servers to accelerate it even more. Just the limitation isn't hardware but "our networking device have 4 10Gbit ports instead of single 40Gbit port" Especially if link is saturated, you'd be essentially taking n-times your "fair share" of bandwidth on link.

Aurornis 9mo ago

> Note there is no intrinsic reason running multiple streams should be faster than one. The issue is the serialization of operations. There is overhead for each operation which translates into dead time between transfers. However there are issues that can cause singular streams to underperform multiple streams in the real world once you reach a certain scale or face problems like packet loss.

wmf 9mo ago
The ideal solution to that is pipelining but it can be complex to implement.
nh2 9mo ago
Is it certain that this is the reason? rsync's man page says "pipelining of file transfers to minimize latency costs" and https://rsync.samba.org/how-rsync-works.html https://rsync.samba.org/how-rsync-works.html says "Rsync is heavily pipelined". If pipelining is really in rsync, there should be no "dead time between transfers".
- spockz 9mo ago
  I’m not sure why, but just like with scp, I’ve achieved significant speeds ups by tarring the directory first (optionally compressing it), transferring and then decompressing. Maybe because it makes the tar and submit, and the receive, untar/uncompress, happen on different threads?
  - lelandbatey 9mo ago
    It's typically a disk-latency thing, as just stat-ing the many files in a directory can have significant latency implications (especially on spinning HDDs) vs opening a single file (the tar) and read-()ing that one file in memory before writing to the network. If copying a folder with many files is slower than tarring that folder and the moving the tar (but not counting the untar) then disk latency is your bottleneck.
    - 9mo ago

softfalcon 9mo ago

> It almost always indicates some bottleneck in the application or TCP tuning. Yeah, this has been my experience with low-overhead streams as well. Interestingly, I see a ubiquity of this "open more streams to send more data" pattern all over the place for file transfer tooling. Recent ones that come to mind have been BackBlaze's CLI (B2) and taking a peek at Amazon's SDK for S3 uploads with Wireshark. (What do they know that we don't seem to think we know?) It seems like they're all doing this? Which is maybe odd, because when I analyse what Plex or Netflix is doing, it's not the same? They do what you're suggesting, tune the application + TCP/UDP stack. Though that could be due to their 1-to-1 streaming use case. There is overhead somewhere and they're trying to get past it via semi-brute-force methods (in my opinion). I wonder if there is a serialization or loss handling problem that we could be glossing over here?

akdev1l 9mo ago
not sure about B2 but AWS S3 SDK not assuming that people will do any tuning makes total sense cuz in my experience no one is doing that tbh
- slightlygrilled 9mo ago
  I’ve found aws s3 it’s always been painful to get any good speed out of it unless it’s massive files you’re moving. It’s base line tuning seems to just assume large files and does no auto scaling and it’s mostly single threaded. Then even when tuning it’s still painfully slow, again seemly limited by its cpu processing and mostly on a single thread, highly annoying. Especially when you’re running it on a high core, fast storage, large internet connection machine. Just feels like there is a large amount of untapped potential in the machines…
  - odo1242 9mo ago
    It’s almost certainly also tuned to prevent excessive or “spiky” traffic to their service.
digiown 9mo ago

yason 9mo ago

Note there is no intrinsic reason running multiple streams should be faster than one If the server side scales (as cloud services do) it might end up using different end points for the parallel connections and saturate the bandwidth better. One server instance might be serving other clients as well and can't fill one particular client's pipe entirely.

patmorgan23 9mo ago

I mean isn't a single TCP connections throughput limited by the latency? Which is why in high(er) latency WAN links you generally want to open multiple connections for large file transfers. https://wintelguy.com/wanperf.pl https://wintelguy.com/wanperf.pl

adrian_b 9mo ago
Only simpler transfer protocols, like TFTP, are limited by latency. The whole reason for the existence of TCP is to overcome the throughput limit determined by latency. On a network with negligible latency there is no need for TCP (you could just send each packet only after the previous is acknowledged, but the higher is the throughput of your network interface, the less likely is that the latency can be negligible). However, for latency to not matter, the TCP windows must be large enough (i.e. the amount of data that is sent before an acknowledge is received, which happens after a delay caused by latency). I use Windows very rarely today, so I do not know its current status, but until the Windows XP days it was very frequent for Windows computers to have very small default TCP window sizes, which caused low throughput on high-latency networks, so on such networks they had to be reconfigured. On high-latency networks, opening multiple connections is just a workaround for not having appropriate network settings. However, even when your own computer is configured optimally, opening multiple connections can be a workaround against various kinds of throttling implemented either by some intermediate ISP or by the destination server, though nowadays most rate limits are applied globally, to all connections from the same IP address, in order to make this workaround ineffective.

nh2 9mo ago

> has a hardcoded maximum receive buffer of 2MiB For completeness, I want to add: The 2MiB are per SSH "channel" -- the SSH protocol multiplexes multiple independent transmission channels over TCP [1], and each one has its own window size. rsync and `cat | ssh | cat` only use a single channel, so if their counterparty is an OpenSSH sshd server, their throughput is limited by the 2MiB window limit. rclone seems to be able to use multiple ssh channels over a single connection; I believe this is what the `--sftp-concurrency` setting controls. Some more discussion about the 2MiB limit and links to work for upstreaming a removal of these limits can be found in my post [3]. Looking into it just now, I found that the SSH protocol itself already supports dynamically growing per-channel window sizes with `CHANNEL_WINDOW_ADJUST`, and OpenSSH seems to generally implement that. I don't fully grasp why it doesn't just use that to extend as needed. I also found that there's an official `no-flow-control` extension with the description > channel behaves as if all window sizes are infinite. > > This extension is intended for, but not limited to, use by file transfer applications that are only going to use one channel and for which the flow control provided by SSH is an impediment, rather than a feature. So this looks exactly as designed for rsync. But no software implements this extension! I wrote those things down in [4]. It is frustrating to me that we're only a ~200 line patch away from "unlimited" instead of shitty SSH transfer speeds -- for >20 years! [1]: https://datatracker.ietf.org/doc/html/rfc4254#section-5 https://datatracker.ietf.org/doc/html/rfc4254#section-5 [2]: https://rclone.org/sftp/#sftp-concurrency https://rclone.org/sftp/#sftp-concurrency [3]: https://news.ycombinator.com/item?id=40856136 https://news.ycombinator.com/item?id=40856136 [4]: https://github.com/djmdjm/openssh-portable-wip/pull/4#issuecomment-3837136128 https://github.com/djmdjm/openssh-portable-wip/pull/4#issuec...

einpoklum 9mo ago

> Note there is no intrinsic reason running multiple streams should be faster than one Inherent reasons or no, it's been my experience across multiple protocols, applications, network connections and environments, and machines on both ends, that, _in fact_, splitting data up and operating using multiple streams is significantly faster. So, ok, it might not be because of an "inherent reason", but we still have to deal with it in real life.

bscphil 9mo ago

> TCP tuning I think a lot of file transfer issues that occur outside of the corporate intranet world involve hardware that you don't fully control on (at least) one hand. In science, for example, transferring huge amounts of data over long distances is pretty common, and I've had to do this on boxes that had poor TCP buffer configurations. Being able to multiplex your streams in situations like this is invaluable and I'd love to see more open source software that does this effectively, especially if it can punch through a firewall.

jolmg 9mo ago

> SSH was never really meant to be a high performance data transfer tool, and it shows. A practical example can be `ssh -X` vs X11 over Wireguard. The lag is obvious with the former, but X11 windows from remote clients can be indistinguishable performance-wise from those of local ones with the latter.