3 ms·
I used rclone to migrate about 2PB of data between datacentres last year. There were some ways to make it go a bit faster, found by reading the rclone manual,
by Symbiote 2y ago
I used rclone to migrate about 2PB of data between datacentres last year.
There were some ways to make it go a bit faster, found by reading the rclone manual, but otherwise no surprises.
I wasn't sure what the maximum transfer speed could have been, but as one side was still the production system I didn't care to reach the limit anyway. Over 10Gb/s anyway.
- nodesocket 2y agoWere the files lots of smaller files or lots of large files? I imagine if lots of small files requires a lot of concurrent threads to utilize a 10Gbps connection.
- Symbiote 2y agoLots of large files, I think most would have been over 50MB. There was a folder of smaller files to move, around a million files each less than 100kB. That took longer, but not long enough that I bothered to try and speed it up.