3 ms·

EBS-SnapShooter – Python script to snapshot EBS volumes

28 points by smile0x90 10y ago

smile0x90 10y ago
This too make AWS - EBS snapshot periodically and more easy. Docker image can also be found in quay . It's an OpenSource project so feel free to contribute :)
- rad_gruchalski 10y ago
  Why not a Lambda function?
  - smile0x90 10y ago
    The purpose of this repo is to prepare a Kubernetes Periodic Job to make ebs snapshots. It can be done with different manner ;)
    - eropple 10y ago
      What is the benefit of assuming additional risk by inserting this into your infrastructure instead of using AWS's existing stuff? (I will admit: am being a little hard on you because I think this is a bad idea. On the scale of bad ideas, it's only a bad idea and not a really bad idea, but it's a bad idea.)
robmoorman 10y ago
What should be the advantage of this script, why just not use the AWS Scheduled backups service
- smile0x90 10y ago
  It can be done within k8s ecosystem using periodic job. That is the pupose of this repo.
eropple 10y ago
CloudWatch Events can already do this for you. I wish it were exposed through automation; that's still coming, but in the interim it's still a solution that minimizes your own failure surface. Do AWS tasks with AWS's tools whenever you can--it minimizes points of failure that you have to manage. Check it: http://docs.aws.amazon.com/AmazonCloudWatch/latest/events/TakeScheduledSnapshot.html http://docs.aws.amazon.com/AmazonCloudWatch/latest/events/Ta... I've also seen people use Lambda, which is halfway to "do AWS tasks with AWS's tools"; it's only a few lines of code and can be flexibly triggered without this much bulk. (Same code can usually be tossed in cron.) If you want to go the persistent-daemon approach, there's something like automated-ebs-snapshots (which I used before CloudWatch Events came along), which is reasonably bulletproofed and is already used in anger: https://github.com/skymill/automated-ebs-snapshots https://github.com/skymill/automated-ebs-snapshots Writing stuff like this is decent practice to get comfortable with the AWS APIs, but these are generally solved problems. I'd be real uncomfortable with k8s's "secrets? what's that? is that something I store unencrypted?" approach to anything remotely sensitive when I'm throwing AWS credentials around; AWS already provides a better ACL method for this sort of thing in IAM. (If you have insisted on a k8s cluster and you aren't able to leverage IAM, this is a problem on your end that should be rectified sooner than soon.)
- foxylion 10y ago
  A good blog post about snapshots with CloudWatch and Lambdas can be found here: Part 1: create snapshots https://serverlesscode.com/post/lambda-schedule-ebs-snapshot-backups/ https://serverlesscode.com/post/lambda-schedule-ebs-snapshot... Part 2: expire old snapshots https://serverlesscode.com/post/lambda-schedule-ebs-snapshot-backups-2/ https://serverlesscode.com/post/lambda-schedule-ebs-snapshot...
- manojlds 10y ago
  Here's my lambda implementation that snapshots and purges - https://github.com/manojlds/ebs-snapshot-lambda https://github.com/manojlds/ebs-snapshot-lambda
- paulddraper 10y ago
  Don't your solutions give you potentially corrupted snapshots? For this reason, for my backups, I run a cronjob on the EC2 instance and call fsfreeze (https://linux.die.net/man/8/fsfreeze https://linux.die.net/man/8/fsfreeze). Is this unnecessary?
tschellenbach 10y ago
Similar to: https://github.com/tschellenbach/Snaptastic https://github.com/tschellenbach/Snaptastic
manojlds 10y ago
EBS Snapshot (and purge) is like building the todo app for the Devops world :) Here's my AWS Lambda implementation that snapshots and purges - https://github.com/manojlds/ebs-snapshot-lambda https://github.com/manojlds/ebs-snapshot-lambda
- machbio 10y ago
  Yes, the next step from Todo App is to do Incremental backups, and delete the old ones beyond certain version.. :)
  - paulddraper 10y ago
    EBS snapshots are incremental already....
psypher246 10y ago
AWS has released a best practice guide and pre-built automated template solution for this on their AWS Answers site: https://aws.amazon.com/answers/infrastructure-management/ebs-snapshot-scheduler/ https://aws.amazon.com/answers/infrastructure-management/ebs...

28 points by smile0x90 10y ago

This too make AWS - EBS snapshot periodically and more easy. Docker image can also be found in quay . It's an OpenSource project so feel free to contribute :)

rad_gruchalski 10y ago
Why not a Lambda function?
- smile0x90 10y ago
  The purpose of this repo is to prepare a Kubernetes Periodic Job to make ebs snapshots. It can be done with different manner ;)
  - eropple 10y ago
    What is the benefit of assuming additional risk by inserting this into your infrastructure instead of using AWS's existing stuff? (I will admit: am being a little hard on you because I think this is a bad idea. On the scale of bad ideas, it's only a bad idea and not a really bad idea, but it's a bad idea.)

robmoorman 10y ago

What should be the advantage of this script, why just not use the AWS Scheduled backups service

smile0x90 10y ago
It can be done within k8s ecosystem using periodic job. That is the pupose of this repo.

eropple 10y ago

CloudWatch Events can already do this for you. I wish it were exposed through automation; that's still coming, but in the interim it's still a solution that minimizes your own failure surface. Do AWS tasks with AWS's tools whenever you can--it minimizes points of failure that you have to manage. Check it: http://docs.aws.amazon.com/AmazonCloudWatch/latest/events/TakeScheduledSnapshot.html http://docs.aws.amazon.com/AmazonCloudWatch/latest/events/Ta... I've also seen people use Lambda, which is halfway to "do AWS tasks with AWS's tools"; it's only a few lines of code and can be flexibly triggered without this much bulk. (Same code can usually be tossed in cron.) If you want to go the persistent-daemon approach, there's something like automated-ebs-snapshots (which I used before CloudWatch Events came along), which is reasonably bulletproofed and is already used in anger: https://github.com/skymill/automated-ebs-snapshots https://github.com/skymill/automated-ebs-snapshots Writing stuff like this is decent practice to get comfortable with the AWS APIs, but these are generally solved problems. I'd be real uncomfortable with k8s's "secrets? what's that? is that something I store unencrypted?" approach to anything remotely sensitive when I'm throwing AWS credentials around; AWS already provides a better ACL method for this sort of thing in IAM. (If you have insisted on a k8s cluster and you aren't able to leverage IAM, this is a problem on your end that should be rectified sooner than soon.)

foxylion 10y ago
A good blog post about snapshots with CloudWatch and Lambdas can be found here: Part 1: create snapshots https://serverlesscode.com/post/lambda-schedule-ebs-snapshot-backups/ https://serverlesscode.com/post/lambda-schedule-ebs-snapshot... Part 2: expire old snapshots https://serverlesscode.com/post/lambda-schedule-ebs-snapshot-backups-2/ https://serverlesscode.com/post/lambda-schedule-ebs-snapshot...
manojlds 10y ago
Here's my lambda implementation that snapshots and purges - https://github.com/manojlds/ebs-snapshot-lambda https://github.com/manojlds/ebs-snapshot-lambda
paulddraper 10y ago
Don't your solutions give you potentially corrupted snapshots? For this reason, for my backups, I run a cronjob on the EC2 instance and call fsfreeze (https://linux.die.net/man/8/fsfreeze https://linux.die.net/man/8/fsfreeze). Is this unnecessary?

tschellenbach 10y ago

Similar to: https://github.com/tschellenbach/Snaptastic https://github.com/tschellenbach/Snaptastic

manojlds 10y ago

EBS Snapshot (and purge) is like building the todo app for the Devops world :) Here's my AWS Lambda implementation that snapshots and purges - https://github.com/manojlds/ebs-snapshot-lambda https://github.com/manojlds/ebs-snapshot-lambda

machbio 10y ago
Yes, the next step from Todo App is to do Incremental backups, and delete the old ones beyond certain version.. :)
- paulddraper 10y ago
  EBS snapshots are incremental already....

psypher246 10y ago

AWS has released a best practice guide and pre-built automated template solution for this on their AWS Answers site: https://aws.amazon.com/answers/infrastructure-management/ebs-snapshot-scheduler/ https://aws.amazon.com/answers/infrastructure-management/ebs...