4 ms·

Any idea what software they are using for the s3 data migration? rclone?

• roxolotl 2y ago
  Yea I’d love to know the answer to this. Of course you can just download petabytes of data but I imagine there’s gotta be more to it than just rclone. Or maybe it is that simple.
  • Symbiote 2y ago
    I used rclone to migrate about 2PB of data between datacentres last year. There were some ways to make it go a bit faster, found by reading the rclone manual, but otherwise no surprises. I wasn't sure what the maximum transfer speed could have been, but as one side was still the production system I didn't care to reach the limit anyway. Over 10Gb/s anyway.
    • nodesocket 2y ago
      Were the files lots of smaller files or lots of large files? I imagine if lots of small files requires a lot of concurrent threads to utilize a 10Gbps connection.
      • Symbiote 2y ago
        Lots of large files, I think most would have been over 50MB. There was a folder of smaller files to move, around a million files each less than 100kB. That took longer, but not long enough that I bothered to try and speed it up.