3 ms·
Yea I’d love to know the answer to this. Of course you can just download petabytes of data but I imagine there’s gotta be more to it than just rclone. Or maybe
by roxolotl 2y ago
Yea I’d love to know the answer to this. Of course you can just download petabytes of data but I imagine there’s gotta be more to it than just rclone. Or maybe it is that simple.
- Symbiote 2y agoI used rclone to migrate about 2PB of data between datacentres last year. There were some ways to make it go a bit faster, found by reading the rclone manual, but otherwise no surprises. I wasn't sure what the maximum transfer speed could have been, but as one side was still the production system I didn't care to reach the limit anyway. Over 10Gb/s anyway.
- nodesocket 2y agoWere the files lots of smaller files or lots of large files? I imagine if lots of small files requires a lot of concurrent threads to utilize a 10Gbps connection.
- Symbiote 2y agoLots of large files, I think most would have been over 50MB. There was a folder of smaller files to move, around a million files each less than 100kB. That took longer, but not long enough that I bothered to try and speed it up.