6 ms·
Having transferred petabytes of data in tens of millions of files over the past months let me assure you there's only one tool that you really need: GNU paralle
by moe 12y ago
Having transferred petabytes of data in tens of millions of files over the past months let me assure you there's only one tool that you really need: GNU parallel.
Whether you copy the individual files with ftp, scp or rsync is largely irrelevant. The network is always your ultimate bottleneck. Using a slower copy-tool just means having to set a slightly higher concurrency in order to max it out.
- batbomb 12y agoFor bulk transfer of many files, and especially transfer over a local/nearby network that may be hold true, but as a general practice, and especially for serial/one off transfer of very large files, GNU parallel won't help. However, all the tools mentioned multiplex connections. They are better for transferring individual files, and you can also use them in parallel. A combination of your network, a CPU threads, frame size, etc... are your ultimate bottleneck when you are transferring very large files. We've transferred exabytes using bbcp and GridFTP. bbcp is very easy to use once it's set up. At some point, you run into much different issues when trying to routinely transfer very large files across continents at speeds greater than 10Gbps.
- anon4 12y agoFor bulk transfer, the absolute fastest I've seen is piping tar through netcat and doing the reverse on the receiving end - on a 10-gigabit lan that results in transfer at the hdd speed. That was between my personal machines with consumer-grade SATA hard disks. The situation probably changes once you add hops and have multiple disks to read from at once.
- rsync 12y agobbcp is a supported protocol at rsync.net. Just saying.
- avn2109 12y agoForgive me if I'm being ignorant, but what's stopping me from making a large torrent?
- moe 12y agoEver tried to make a torrent containing millions of files? ;-)
- avn2109 12y agoNo, but I do have tar -czf whatever.tar.gz, right?
- moe 12y agoWe usually don't have the patience nor spare disk space to spend days or weeks on creating a multi-terabyte tar-archive first. I'm also rather skeptical that the common BT clients are made to handle files in the multi-terabyte range very well. And finally, BT only makes sense when you're transferring to multiple destinations. There are better options for 1-to-1 transfers.
- avn2109 12y agoWell that makes sense. Thanks for the clarification.
- deleted 12y ago[deleted]