4 ms·
Related - I've been thinking about how to best backup my S3 buckets (some with 50k+ files) off of Amazon. Sure I can setup another bucket with that cross region
by PStamatiou 7y ago
Related - I've been thinking about how to best backup my S3 buckets (some with 50k+ files) off of Amazon. Sure I can setup another bucket with that cross region duplication feature, and I have versioning.. but would really prefer a backup off of Amazon (ie not sending manually created zips in a lightsail/ec2 or something to glacier) in case it ever gets hacked or I accidentally nuke the buckets or something like that.
Currently just doing a combination of s3cmd for a local archive (takes forever to download and then it doesnt seem like incremental syncs are any faster), as well as having Google Console clone my bucket there (but I'm not sure if it's versioned, or as easy as downloading the whole archive).
Never used duplicity -- would it be fast for something like this? Guessing I should just cron it on a remote server instead of running off a local machine frequently.
- padelt 7y agoHave you had a look at rclone? Pretty sure you can copy or even sync files from one remote storage to another. E.g. copy from S3 to B2. https://rclone.org/commands/rclone_sync/ https://rclone.org/commands/rclone_sync/
- PStamatiou 7y agoThanks, I haven't. Will take a look
- anderspitman 7y ago+1 for rclone here. It can indeed copy between remote backends. Just keep in mind that that data all has to flow through the rclone process. You could probably get much better performance by running rclone itself on an ec2 instance. Just keep an eye on your throughput usage.