3 ms·

> Serial local reads are not a plausible cause, because the autor describes working on NVMe SSDs which have so low latency that they cannot explain that reading

> Serial local reads are not a plausible cause, because the autor describes working on NVMe SSDs which have so low latency that they cannot explain that reading 59 GB across 3000 files take 8 minutes. But the people you responded to were talking about slowdowns that exist in general, not just ones that apply directly to the post. For the post, my personal guess is that per-file overhead isn't a huge factor here, and it's mostly rsync having trouble doing >1Gbps over the network. > In fact, it is even highly likely that rsync will not / cannot pipeline reading files that appear local to it, because normal POSIX file IO does not really offer any ways to non-blocking read regular files, so the only way to do that is with threads, which rsync doesn't use. Makes sense. > it absolutely would wait for network trips between files I don't see why you're saying this. I expect it to serially read files and then put that data into a buffer that can have data from multiple files at the same time. In other words, pipelined networking. As long as the transfer queue doesn't bottom out it shouldn't have to wait for any network round trips. What leads you to think otherwise?

nh2 9mo ago
> But the people you responded to were talking about slowdowns that exist in general, not just ones that apply directly to the post. I think that's incorrect though. These slowdowns do not exist in general (see my next reply where I run rsync and it immadiately maxes out my 10 Gbit/s). I think original poster digiown is right with "Note there is no intrinsic reason running multiple streams should be faster than one [EDIT: 'at this scale']. It almost always indicates some bottleneck in the application". In this case it's the user running rsync as a serially-reading program reading from a network mount. > rsync having trouble doing >1Gbps over the network rsync copies at 10 Gbit/s without problem between my machines. Though I have to give `-e 'ssh -c aes256-gcm@openssh.com'` or aes128-gcm, otherwise encryption bottlenecks at 5 Gbit/s with the default `chacha20-poly1305@openssh.com`. > I don't see why you're saying this. Because of the part you agreed making sense: It read each file with the sequence `open()/read()/.../read()/close()`, but those files are on the network mount ("/Volumes/mercury"), so each `read()` of size `#define IO_BUFFER_SIZE (32*1024)` is a network roundtrip.
- Dylan16807 9mo ago
  I see, so you're saying the file end of rsync is forced to wait for the network because the filesystem itself waits, not the network end of rsync. That makes sense. Though I wonder what the actual delay is. The numbers in the post implied several milliseconds, enough to maybe account for 30 seconds of the 8 minutes. But maybe changing files resets the transfer speed a bunch.
  - nh2 9mo ago
    From which part do you take the "several milliseconds"? If I assume 0.2ms ping and each rsync read() is a roundtrip, I arrive at 6.4 minutes = 62955918871 B / (321024 B) 0.0002 s / (60 s/min).
    - Dylan16807 9mo ago
      To be clear, when I said "delay" in my last post I meant the per-file penalty, not the round-trip latency. Radxa Orion O6/.DS_Store, 6KB at 4.8MBps, that means it took 1.3ms to transfer Radxa Orion O6/Micro Center Visit Details.pages, 141KB at 9.8MBps, 14ms Radxa Orion O6/Radxa Orion O6.md, 19KB at 1.9MBps, 10ms We know the link can do well over 100MBps, so that time is almost all overhead. But it doesn't seem to be a simple delay. Perhaps a fresh TCP window scaling up on a per-file basis? That would be an unfortunate filesytem design. Since the two 4MB files both get up to ~100MBps, the same speed as the 250MB file, it seems like the maximum impact from switching files isn't much more than 15ms. If the average is below 10ms then we're looking at half a minute wasted over 3564 files. If the average is 20ms then switching files is responsible for 71 seconds wasted. By that estimate the file-level serialization is a real issue, but the bigger issue is whatever's preventing that 250MB file from ramping up all the way to 10Gbps.
      - nh2 9mo ago
        I guess until we know what that network share is and how it works, we cannot really progress. Taking 10 ms to fetch the 19 KB file is definitely not great, it would be ~50 LAN roundtrips. > whatever's preventing that 250MB file from ramping up all the way to 10Gbps Probably that network share does small reads if small reads are requested. Might be good if the author ran `dd` or a similar tool where the read buffer size can be controlled.

nh2 9mo ago

> But the people you responded to were talking about slowdowns that exist in general, not just ones that apply directly to the post. I think that's incorrect though. These slowdowns do not exist in general (see my next reply where I run rsync and it immadiately maxes out my 10 Gbit/s). I think original poster digiown is right with "Note there is no intrinsic reason running multiple streams should be faster than one [EDIT: 'at this scale']. It almost always indicates some bottleneck in the application". In this case it's the user running rsync as a serially-reading program reading from a network mount. > rsync having trouble doing >1Gbps over the network rsync copies at 10 Gbit/s without problem between my machines. Though I have to give `-e 'ssh -c aes256-gcm@openssh.com'` or aes128-gcm, otherwise encryption bottlenecks at 5 Gbit/s with the default `chacha20-poly1305@openssh.com`. > I don't see why you're saying this. Because of the part you agreed making sense: It read each file with the sequence `open()/read()/.../read()/close()`, but those files are on the network mount ("/Volumes/mercury"), so each `read()` of size `#define IO_BUFFER_SIZE (32*1024)` is a network roundtrip.

Dylan16807 9mo ago
I see, so you're saying the file end of rsync is forced to wait for the network because the filesystem itself waits, not the network end of rsync. That makes sense. Though I wonder what the actual delay is. The numbers in the post implied several milliseconds, enough to maybe account for 30 seconds of the 8 minutes. But maybe changing files resets the transfer speed a bunch.
- nh2 9mo ago
  From which part do you take the "several milliseconds"? If I assume 0.2ms ping and each rsync read() is a roundtrip, I arrive at 6.4 minutes = 62955918871 B / (321024 B) 0.0002 s / (60 s/min).
  - Dylan16807 9mo ago
    To be clear, when I said "delay" in my last post I meant the per-file penalty, not the round-trip latency. Radxa Orion O6/.DS_Store, 6KB at 4.8MBps, that means it took 1.3ms to transfer Radxa Orion O6/Micro Center Visit Details.pages, 141KB at 9.8MBps, 14ms Radxa Orion O6/Radxa Orion O6.md, 19KB at 1.9MBps, 10ms We know the link can do well over 100MBps, so that time is almost all overhead. But it doesn't seem to be a simple delay. Perhaps a fresh TCP window scaling up on a per-file basis? That would be an unfortunate filesytem design. Since the two 4MB files both get up to ~100MBps, the same speed as the 250MB file, it seems like the maximum impact from switching files isn't much more than 15ms. If the average is below 10ms then we're looking at half a minute wasted over 3564 files. If the average is 20ms then switching files is responsible for 71 seconds wasted. By that estimate the file-level serialization is a real issue, but the bigger issue is whatever's preventing that 250MB file from ramping up all the way to 10Gbps.
    - nh2 9mo ago
      I guess until we know what that network share is and how it works, we cannot really progress. Taking 10 ms to fetch the 19 KB file is definitely not great, it would be ~50 LAN roundtrips. > whatever's preventing that 250MB file from ramping up all the way to 10Gbps Probably that network share does small reads if small reads are requested. Might be good if the author ran `dd` or a similar tool where the read buffer size can be controlled.