3 ms·
WE use 10 rsyncs in parallel to copy .5pb in less than 3 days. CPU is cheap.
by KaiserPro 12y ago
WE use 10 rsyncs in parallel to copy .5pb in less than 3 days.
CPU is cheap.
- spydum 12y agoHave done the same, found that rsync seems to not natively parallelize itself, so spread across 20 cores, it really screamed.
- rimantas 12y agoDamn, I could use some advice there. Right now I have a system which uses rsync to backup some files. It takes about 8 hours to go over 7+ million files (~3.5TB). I'd love to speed up this. I should mention that copy is done over curlftpfs :(.