5 ms·
I recently had to copy around 200gb of files over wifi. I used rsync to make sure a connection failure doesn't mean I have to start over and so that nothing is
by dmos62 3y ago
I recently had to copy around 200gb of files over wifi. I used rsync to make sure a connection failure doesn't mean I have to start over and so that nothing is lost, but it took at least 6 hours. I wonder what could I have done better.
Btw, what kinds of guarantees do you get with the dd method? Do you have to compare md5s of the resulting block level devices after?
- frankjr 3y ago> ...but it took at least 6 hours... Rsync cannot transfer more than one file at a time so if you were transferring a lot of small files that was probably the bottleneck. You could either use xargs/parallel to split the file list and run multiple instances of rsync or use something like rclone, which supports parallel transfers on its own.
- mkesper 3y agoYou could tar and zip the files into a netcat-pipe
- maccard 3y ago> I wonder what could I have done better. 6 hours is roughly 10MB/s, so you likely could have gone much much quicker. Did you compress with `-z`? If you could use ethernetyou probably could have done it at closer to 100MB/s on most deviceds, which would have been 35 minutes.
- dmos62 3y agoNo, I didn't use compression. Would it be useful over a high-bandwidth connection? I presume that it wasn't wifi bandwidth that was bottlenecking, though I've not really checked. One thing I could have done is found a way to track total progress, so that I could have noticed that this is going way too slow.
- skinner927 3y agoI’ve had compression noticeably speed things up even over a wired lan
- mixmastamyk 3y agoOne of the options of rsync is to print out transfer speed, --progress or verbose or similar.
- dmos62 3y agoYeah, I did that, but that shows stats per file, which is minimally useful. I would see it go up to 15 mb/s for bigger files, but turns out the aggregate speed was less than 10 mb/s.
- mixmastamyk 3y agoRight, I guess it doesn't show the total speed until the end, which is probably too late. Although 15MB/s is not very fast these days either. Elsewhere I recommended the Thunderbolt4 cable—it screamed at ~10GBs and didn't need any optimization like tar/compression.
- maccard 3y agoI wouldn't consider 10MB/s high bandwidth. My home wifi network can comfortably handle 35MB/s, while a wired connection is 110MB/s+ > No, I didn't use compression. Would it be useful over a high-bandwidth connection? Compression will be useful if you're IO bound as opposed to CPU bound, which at 10MB/s you definitely are.
- dmos62 3y agoThanks for the clarification. I presumed that it's the rsync+ssh protocol chatter that was slowing me down, not the wifi bandwidth (which I've not tested separately ). I don't have my head wrapped around the mechanics in play, to be honest.
- hooli_gan 3y agoZstd is also great for io bottlenecked transfers with its adapt flag. I don't know if you can use it with rsync though. > --adapt[=min=#,max=#]: zstd will dynamically adapt compression level to perceived I/O conditions.
- throwawaaarrgh 3y agoIf your transport method for rsync was ssh, that is often a bottleneck, as openssh has historically had some weird performance limits that needed obscure patches to get around. Enabling compression helps too if your CPU doesn't become a bottleneck
- dmos62 3y agoYeah, it was SSH. Thanks for the heads up.
- klabb3 3y ago> 200gb of files over wifi […] took at least 6 hours > I wonder what could I have done better. Used an Ethernet cable? That’s not an impressive throughput amount over local. WiFi has like a million more sources of perf bottlenecks. Btw, just using a cable on ONE of the device => router ~> device can help a lot.
- dmos62 3y agoYeah, I did that. One of the devices didn't have an ethernet port though.
- klabb3 3y agoIf it’s a MacBook you can get Ethernet - USB-C adapters for dirt cheap. Otherwise, you have to move closer to the router and/or get a new router/wifi card if it’s running old 2g stuff for instance. But again, WiFi is headache - especially since many parameters are outside your control (like nearby airports, neighbors, layout of your home, to name a few lol).
- vetinari 3y ago200 GB in 6 hours is too slow for wifi, too. dd doesn't skip empty blocks, like clonezilla would do.
- downrightmike 3y agoWifi shares the medium (air) with all other radios. It has a random time it waits after it stops if it sees a collision. "Carrier-sense multiple access with collision avoidance (CSMA/CA) in computer networking, is a network multiple access method in which carrier sensing is used, but nodes attempt to avoid collisions by beginning transmission only after the channel is sensed to be "idle".[1][2] When they do transmit, nodes transmit their packet data in its entirety. It is particularly important for wireless networks, where the alternative with collision detection CSMA/CD, is not possible due to wireless transmitters desensing (turning off) their receivers during packet transmission. CSMA/CA is unreliable due to the hidden node problem.[3][4] CSMA/CA is a protocol that operates in the data link layer." https://en.wikipedia.org/wiki/Carrier-sense_multiple_access_with_collision_avoidance https://en.wikipedia.org/wiki/Carrier-sense_multiple_access_...
- dmos62 3y agoI'm not sure I get your meaning. Are you implying that dd over netcat is unreliable because CSMA/CA is unreliable?
- downrightmike 3y agoSorry, just that it isn't anywhere near as efficient. There's a lot of random wait that will slow everything down.
- dmos62 3y agoOh ok, thanks for the insight!