3 ms·
For bulk transfer of many files, and especially transfer over a local/nearby network that may be hold true, but as a general practice, and especially for serial
by batbomb 12y ago
For bulk transfer of many files, and especially transfer over a local/nearby network that may be hold true, but as a general practice, and especially for serial/one off transfer of very large files, GNU parallel won't help. However, all the tools mentioned multiplex connections. They are better for transferring individual files, and you can also use them in parallel.
A combination of your network, a CPU threads, frame size, etc... are your ultimate bottleneck when you are transferring very large files.
We've transferred exabytes using bbcp and GridFTP. bbcp is very easy to use once it's set up.
At some point, you run into much different issues when trying to routinely transfer very large files across continents at speeds greater than 10Gbps.
- anon4 12y agoFor bulk transfer, the absolute fastest I've seen is piping tar through netcat and doing the reverse on the receiving end - on a 10-gigabit lan that results in transfer at the hdd speed. That was between my personal machines with consumer-grade SATA hard disks. The situation probably changes once you add hops and have multiple disks to read from at once.
- rsync 12y agobbcp is a supported protocol at rsync.net. Just saying.