5 ms·

Two separate S3 buckets with different credentials. The server pushes its own backups to the "quarantine" bucket and later a process moves the backup to the lon

Two separate S3 buckets with different credentials. The server pushes its own backups to the "quarantine" bucket and later a process moves the backup to the long term storage bucket.

dev_dull 7y ago
I’m surprised this answer is so far down. Production machines write to an S3 bucket with a lambda trigger. As soon as the upload is finished, the lambda either lifts the file to another location or changes the acl so the production machines can no longer access it.
luhn 7y ago
I use a single bucket with versioning enabled. Production server has access to write as it pleases, but isn't allowed to touch past versions of a file.
- eropple 7y ago
  S3 now allows permanent object locking, too, which makes this even harder to screw up.
- brazzledazzle 7y ago
  I think having the bucket in a different account with different root/MFA might make sense here too. Your production AWS account is more likely to get owned than your account dedicated to backups.
  - luhn 7y ago
    Yes! The bucket is located in a different account with separate billing and very limited on who has root/admin access.
reaperducer 7y ago
Two separate S3 buckets with different credentials This gives me the heebie-jeebies. It seems better to me to have one on Amazon, and one on another cloud provider entirely. Plus regular sneakernet archives, if your data is of a size that would permit that sort of thing. I'm fortunate that this kind of thing is handled by other people where I work. It's my understanding that we have production data on one service on the east coast, a mirror with a different company in the south, and weekly backup archives stored offline on the west coast.
- PowerfulWizard 7y ago
  One thing I've considered doing but haven't done, is building an "archive" box just to sit in the office and mirror cloud data. It wouldn't be all that complicated just mirror some cloud storage onto local encrypted storage -- pretty easy until somewhere over 10TB. There are plenty of ways that is could fail but they would be much more uncorrelated than a backup on the same cloud provider.
  - chx 7y ago
    > pretty easy until somewhere over 10TB. 10TB is not even the largest single HDD money can buy... and I wouldn't use a single drive anyways. What I'd do... There's the new Silverstone CS381 chsasis with is <30L (so it easily fits anywhere) and houses eight 3.5" hot swap disks plus 4 2.5" for boot and a micro ATX PC... Looking at https://diskprices.com/ https://diskprices.com/ I'd go with the Seagate 8TB drives at 120 bucks by shucking them from external and then you need a cheap uATX board and an LSI-9207-8i card.
    - SturgeonsLaw 7y ago
      Thank you, I've been looking for a good home NAS solution, this looks like it could be it
    - PowerfulWizard 7y ago
      Yeah I was vague on the size because it depends how you build it ... If I were to do it I'd most likely want a 2x replication factor locally, a rebuild time under 12 hours for a drive loss, and set a threshold around 80% storage use where I'd need to expand. Staying within those parameters I would expect maintenance to be easier and unrushed. I think that would send me towards smaller drives, but haven't properly analyzed it. I've been meaning to figure out how to set all that up with a ZFS system. Ironically it would be so much easier to build and test on cloud servers so I would probably do that first. That case does look very nice for a storage box though.

Two separate S3 buckets with different credentials. The server pushes its own backups to the "quarantine" bucket and later a process moves the backup to the long term storage bucket.

dev_dull 7y ago

I’m surprised this answer is so far down. Production machines write to an S3 bucket with a lambda trigger. As soon as the upload is finished, the lambda either lifts the file to another location or changes the acl so the production machines can no longer access it.

luhn 7y ago

I use a single bucket with versioning enabled. Production server has access to write as it pleases, but isn't allowed to touch past versions of a file.

eropple 7y ago
S3 now allows permanent object locking, too, which makes this even harder to screw up.
brazzledazzle 7y ago
I think having the bucket in a different account with different root/MFA might make sense here too. Your production AWS account is more likely to get owned than your account dedicated to backups.
- luhn 7y ago
  Yes! The bucket is located in a different account with separate billing and very limited on who has root/admin access.

reaperducer 7y ago

Two separate S3 buckets with different credentials This gives me the heebie-jeebies. It seems better to me to have one on Amazon, and one on another cloud provider entirely. Plus regular sneakernet archives, if your data is of a size that would permit that sort of thing. I'm fortunate that this kind of thing is handled by other people where I work. It's my understanding that we have production data on one service on the east coast, a mirror with a different company in the south, and weekly backup archives stored offline on the west coast.

PowerfulWizard 7y ago
One thing I've considered doing but haven't done, is building an "archive" box just to sit in the office and mirror cloud data. It wouldn't be all that complicated just mirror some cloud storage onto local encrypted storage -- pretty easy until somewhere over 10TB. There are plenty of ways that is could fail but they would be much more uncorrelated than a backup on the same cloud provider.
- chx 7y ago
  > pretty easy until somewhere over 10TB. 10TB is not even the largest single HDD money can buy... and I wouldn't use a single drive anyways. What I'd do... There's the new Silverstone CS381 chsasis with is <30L (so it easily fits anywhere) and houses eight 3.5" hot swap disks plus 4 2.5" for boot and a micro ATX PC... Looking at https://diskprices.com/ https://diskprices.com/ I'd go with the Seagate 8TB drives at 120 bucks by shucking them from external and then you need a cheap uATX board and an LSI-9207-8i card.
  - SturgeonsLaw 7y ago
    Thank you, I've been looking for a good home NAS solution, this looks like it could be it
  - PowerfulWizard 7y ago
    Yeah I was vague on the size because it depends how you build it ... If I were to do it I'd most likely want a 2x replication factor locally, a rebuild time under 12 hours for a drive loss, and set a threshold around 80% storage use where I'd need to expand. Staying within those parameters I would expect maintenance to be easier and unrushed. I think that would send me towards smaller drives, but haven't properly analyzed it. I've been meaning to figure out how to set all that up with a ZFS system. Ironically it would be so much easier to build and test on cloud servers so I would probably do that first. That case does look very nice for a storage box though.