Fun problem. Without replication, here's how I would do it. A pipeline: S3 list worker(s) (see below for optiizations) -> SQS -> aws lambda (s3 sync) -> SQS ->

Fun problem. Without replication, here's how I would do it. A pipeline: S3 list worker(s) (see below for optiizations) -> SQS -> aws lambda (s3 sync) -> SQS -> aws lambda (s3 delete of original key). Now the secret: to make it really go vrrooom you need to sufficiently parallelize the list operations. For this enable S3 storage analytics on the bucket to get a manifest of all of the files a few days beforehand. Use that list to partition the prefix/key space evenly into 20 or 30 (or 100, or 200) workers each with start and end key prefix. With SQS, you can make sure the sync and delete actions succeed, and with lambda they'll run in a serverless fashion as fast as you can feed the queues. Amazon will absolutely throttle you when making a lot of requests. Reach out to them ahead of time. Make your lambda code robust, so that a failure is idempotent lean into the retries.