5 ms·

Lost an EBS snapshot on AWS once. The only way to provide some assurance in this space is to make sure your data is stored in more than one physically and comme

by uuyi 5y ago

Lost an EBS snapshot on AWS once. The only way to provide some assurance in this space is to make sure your data is stored in more than one physically and commercially separate location.

Aachen 5y ago
This is the way. I get that not doing small-scale things yourself saves costs and that's why "cloud" exists, but if something is important to you or your organization, I cannot stress enough how useful having just a simple old data copy is. I've lost data to bad drives too often myself. Microsoft can erroneously close your paid-for account without recourse, or ransomware can try to extort you, but your data will simply not be lost if you have an offline (or more advanced: append-only) system with an hourly/daily/... backup on it. There are various options here but the one I happened to have read up on (and then contributed documentation to make it easier to use safely) is Restic with Rest-server in append-only mode. It's not perfect (e.g. no compression yet, only dedup) but it's pretty good. If you're an individual or small–medium business, put your favorite raspberry pi equivalent in a closet and invest a hundred bucks in a hard drive big enough for the important documents, pictures of loved ones, etc. A single drive shouldn't be your only backup, just don't put 100% of your trust in magic cloud. See also Atlassian this week: it may be down, but if you need to access the data of ticket xyz for a lead this week and you have a local data export, you can find the needed data.
- mahastore 5y ago
  you do get what you paid for
  - Aachen 5y ago
    Not always. Pay for various games on Xbox, paid onedrive storage, pay for office licenses for the whole family, trip some false positive and still get your account banned with everything in it. You can talk to a wall or ask a judge to force Microsoft to give a reason, but then they can't actually show the reason because that's against some law in another country or something. It's fantastic. (Dutch, unfortunately) https://tweakers.net/nieuws/194930/microsoft-moet-blokkade-nederlands-onedrive-account-opheffen-van-rechter.html https://tweakers.net/nieuws/194930/microsoft-moet-blokkade-n... https://tweakers.net/reviews/9094/account-geblokkeerd-wat-nu.html https://tweakers.net/reviews/9094/account-geblokkeerd-wat-nu...
sidewndr46 5y ago
That works until you find out the commercially separate location was using Amazon for their storage solution as well.
herpderperator 5y ago
3-2-1 Backup: https://www.backblaze.com/blog/the-3-2-1-backup-strategy/ https://www.backblaze.com/blog/the-3-2-1-backup-strategy/
ckozlowski 5y ago
I would argue that EBS snapshots should not be used as a backup solution. EBS is reliable, but isn't engineered for the same durability as say, S3. These also become more unwieldy and less useful the older they get. Use EBS snapshots for rollback prior to making changes to your environment. For durable backup I'd recommend storing objects in S3/Glacier. If you need to image an entire volume, make an AMI, which is stored in S3. Much more durable and targeted than a snapshot, and in the case of S3 objects, gives you benefits such as logging and version control.
- forty 5y ago
  Aren't EBS snapshots stored on S3? :)
  - ckozlowski 5y ago
    They are (and in going to look that up I see that they're being pitched as more of a backup solution than they were previously) but given they're incremental and based on the existing EBS volume, I'd still recommend using something without dependencies. I'm curious now as to how the EBS Snapshot was lost. If I had to guess, I suspect it had something to with that linkage back to the original volume. I maintain that the best backup has no dependencies on another resource existing.
    - uuyi 5y ago
      We never got a straight answer from AWS. We now run an instance in another region and account and rdiff-backup to it.
    - forty 5y ago
      It's a good point. There is a relatively new feature to archive EBS snapshots [1] which stores non incremental snapshots [1] https://aws.amazon.com/fr/blogs/aws/new-amazon-ebs-snapshots-archive/ https://aws.amazon.com/fr/blogs/aws/new-amazon-ebs-snapshots...
- imtringued 5y ago
  For most people Glacier is a terrible solution.

by uuyi 5y ago

Lost an EBS snapshot on AWS once. The only way to provide some assurance in this space is to make sure your data is stored in more than one physically and commercially separate location.

Aachen 5y ago

This is the way. I get that not doing small-scale things yourself saves costs and that's why "cloud" exists, but if something is important to you or your organization, I cannot stress enough how useful having just a simple old data copy is. I've lost data to bad drives too often myself. Microsoft can erroneously close your paid-for account without recourse, or ransomware can try to extort you, but your data will simply not be lost if you have an offline (or more advanced: append-only) system with an hourly/daily/... backup on it. There are various options here but the one I happened to have read up on (and then contributed documentation to make it easier to use safely) is Restic with Rest-server in append-only mode. It's not perfect (e.g. no compression yet, only dedup) but it's pretty good. If you're an individual or small–medium business, put your favorite raspberry pi equivalent in a closet and invest a hundred bucks in a hard drive big enough for the important documents, pictures of loved ones, etc. A single drive shouldn't be your only backup, just don't put 100% of your trust in magic cloud. See also Atlassian this week: it may be down, but if you need to access the data of ticket xyz for a lead this week and you have a local data export, you can find the needed data.

mahastore 5y ago
you do get what you paid for
- Aachen 5y ago
  Not always. Pay for various games on Xbox, paid onedrive storage, pay for office licenses for the whole family, trip some false positive and still get your account banned with everything in it. You can talk to a wall or ask a judge to force Microsoft to give a reason, but then they can't actually show the reason because that's against some law in another country or something. It's fantastic. (Dutch, unfortunately) https://tweakers.net/nieuws/194930/microsoft-moet-blokkade-nederlands-onedrive-account-opheffen-van-rechter.html https://tweakers.net/nieuws/194930/microsoft-moet-blokkade-n... https://tweakers.net/reviews/9094/account-geblokkeerd-wat-nu.html https://tweakers.net/reviews/9094/account-geblokkeerd-wat-nu...

sidewndr46 5y ago

That works until you find out the commercially separate location was using Amazon for their storage solution as well.

herpderperator 5y ago

3-2-1 Backup: https://www.backblaze.com/blog/the-3-2-1-backup-strategy/ https://www.backblaze.com/blog/the-3-2-1-backup-strategy/

ckozlowski 5y ago

I would argue that EBS snapshots should not be used as a backup solution. EBS is reliable, but isn't engineered for the same durability as say, S3. These also become more unwieldy and less useful the older they get. Use EBS snapshots for rollback prior to making changes to your environment. For durable backup I'd recommend storing objects in S3/Glacier. If you need to image an entire volume, make an AMI, which is stored in S3. Much more durable and targeted than a snapshot, and in the case of S3 objects, gives you benefits such as logging and version control.

forty 5y ago
Aren't EBS snapshots stored on S3? :)
- ckozlowski 5y ago
  They are (and in going to look that up I see that they're being pitched as more of a backup solution than they were previously) but given they're incremental and based on the existing EBS volume, I'd still recommend using something without dependencies. I'm curious now as to how the EBS Snapshot was lost. If I had to guess, I suspect it had something to with that linkage back to the original volume. I maintain that the best backup has no dependencies on another resource existing.
  - uuyi 5y ago
    We never got a straight answer from AWS. We now run an instance in another region and account and rdiff-backup to it.
  - forty 5y ago
    It's a good point. There is a relatively new feature to archive EBS snapshots [1] which stores non incremental snapshots [1] https://aws.amazon.com/fr/blogs/aws/new-amazon-ebs-snapshots-archive/ https://aws.amazon.com/fr/blogs/aws/new-amazon-ebs-snapshots...
imtringued 5y ago
For most people Glacier is a terrible solution.