3 ms·
> I wonder what could I have done better. 6 hours is roughly 10MB/s, so you likely could have gone much much quicker. Did you compress with `-z`? If you could
by maccard 3y ago
> I wonder what could I have done better.
6 hours is roughly 10MB/s, so you likely could have gone much much quicker. Did you compress with `-z`? If you could use ethernetyou probably could have done it at closer to 100MB/s on most deviceds, which would have been 35 minutes.
- dmos62 3y agoNo, I didn't use compression. Would it be useful over a high-bandwidth connection? I presume that it wasn't wifi bandwidth that was bottlenecking, though I've not really checked. One thing I could have done is found a way to track total progress, so that I could have noticed that this is going way too slow.
- skinner927 3y agoI’ve had compression noticeably speed things up even over a wired lan
- mixmastamyk 3y agoOne of the options of rsync is to print out transfer speed, --progress or verbose or similar.
- dmos62 3y agoYeah, I did that, but that shows stats per file, which is minimally useful. I would see it go up to 15 mb/s for bigger files, but turns out the aggregate speed was less than 10 mb/s.
- mixmastamyk 3y agoRight, I guess it doesn't show the total speed until the end, which is probably too late. Although 15MB/s is not very fast these days either. Elsewhere I recommended the Thunderbolt4 cable—it screamed at ~10GBs and didn't need any optimization like tar/compression.
- maccard 3y agoI wouldn't consider 10MB/s high bandwidth. My home wifi network can comfortably handle 35MB/s, while a wired connection is 110MB/s+ > No, I didn't use compression. Would it be useful over a high-bandwidth connection? Compression will be useful if you're IO bound as opposed to CPU bound, which at 10MB/s you definitely are.
- dmos62 3y agoThanks for the clarification. I presumed that it's the rsync+ssh protocol chatter that was slowing me down, not the wifi bandwidth (which I've not tested separately ). I don't have my head wrapped around the mechanics in play, to be honest.
- hooli_gan 3y agoZstd is also great for io bottlenecked transfers with its adapt flag. I don't know if you can use it with rsync though. > --adapt[=min=#,max=#]: zstd will dynamically adapt compression level to perceived I/O conditions.