3 ms·
Wow, what a coincidence, I literally just installed this yesterday because I wanted to write my own command-line S3 uploader...yes I know there's an official CL
by danso 11y ago
Wow, what a coincidence, I literally just installed this yesterday because I wanted to write my own command-line S3 uploader...yes I know there's an official CLI [1] but I wanted some customized behavior and options (such as simultaneously backing up to s3 and glacier) and thought it'd be a good chance to better understand AWS in general. I don't know what the previous boto was like but boto3 has been pretty pain-free. Many of the error messages I've run into have been very actively discussed/patched/hacked-around in the repo issues.
[1] http://aws.amazon.com/cli/ http://aws.amazon.com/cli/
- samstave 11y agohttps://github.com/bloomreach/s4cmd https://github.com/bloomreach/s4cmd
- deleted 11y ago[deleted]
- vosper 11y agoSorry to go off topic, but why would you want to simultaneously backup to S3 and Glacier? Unless you're using S3 with Reduced Redundancy then you're not going to lose any files, so Glacier seems pointless. Also, working with S3 is way easier than working with Glacier, so if you're using S3 already I would just stick with that.
- Psyonic 11y agoRedundancy is not the same as backup. You may not lose files due to machine failure, but a code error, some data scientist deleting the wrong folder, etc, could easily delete the files. Having a backup that isn't automatically tied to the s3 bucket can help prevent catastrophe in these cases.
- vosper 11y agoThat's solved by using Object Versioning in S3. That makes it easy to recover from accidental deletion. You can still permanently remove an object by deleting it's versions, but that's a separate operation and not one you'd likely do by accident. Proper permissioning can stop it even being possible. Beyond that you're probably thinking about black-swan scenarios like AWS going out of business or a hacker breaching your system and deleting everything [1]. In that case you need to be completely separated from AWS (and careful about maintaining the separation) so also using Glacier won't help anyway. [1] This happened, and put Code Spaces out of business: http://arstechnica.com/security/2014/06/aws-console-breach-leads-to-demise-of-service-with-proven-backup-plan/ http://arstechnica.com/security/2014/06/aws-console-breach-l...
- elithrar 11y ago> Proper permissioning can stop it even being possible. 100% this. A little OT, but my standard Postgres backup procedure is streaming via WAL-E to S3. The user WAL-E runs as cannot delete versions. Any attempt to overwrite base backups or WAL parts versions them. I have the versions expire after a couple of months (to reduce storage costs). I do the same for selected logs in /var/log/ via duply.
- deleted 11y ago[deleted]
- danso 11y agoYeah that's true. I think I meant more toward understanding the granularity of operations common to AWS, with the assumption that the logistics of moving data onto S3 to Glacier would be the same. Since you can control expire-times on S3 files, there might be situations in which you want files on S3 for a short period, maybe 30 days, because you want a window for quick access just in case you were prematurely archiving, while making sure they're also on Glacier...I frequently find myself hoarding files, keeping them on S3 (or worse, USB keys) for longer than I had intended...
- teej 11y agoS3 Lifecycle Management does this for you! http://docs.aws.amazon.com/AmazonS3/latest/dev/object-lifecycle-mgmt.html http://docs.aws.amazon.com/AmazonS3/latest/dev/object-lifecy...
- pdelgallego 11y ago> such as simultaneously backing up to s3 and glacier I you want that for every object in the bucket (or specific files with a prefix), you are be able achieve that by adding a lifecycle rule: 1 Select the bucket 2 Add a new lifecycle rule 3 Action = Archive only 4 Archive to the Glacier Storage Class 0 after the object's creation date. (Enter '0' for same-day archival) 5 Save rule You can automate that using aws cli