6 ms·

I sometimes do infrastructure consulting. One of the first questions I ask is if they have at least one fully independent, full/incremental off-site backup tha

I sometimes do infrastructure consulting. One of the first questions I ask is if they have at least one fully independent, full/incremental off-site backup that can't be corrupted from the main infrastructure, and if they have ever checked if they actually work and are restorable. I'm continuously surprised how often the answer turns out to be no after dinner digging, even in larger companies with otherwise well-run IT. No, the automatic 7 day RDS snapshots or turning on S3 versioning is not a sufficient backup. Neither is mirroring to a S3 Glacier bucket in the same org, or rsyncing to a a backup server in the same datacenter. Backups are annoying and unglamorous. Nobody wants to do them, or do the tedious work of validating them or setting up something like an automated restore test. Until the day you lose your data.

nicbou 6y ago
This applies to average people too. I wonder who among us can say they meet your (reasonable) standard. Like you said, backups are annoying and unglamorous. Yet, the data on my laptop is the only thing I could not replace. It's more important to me than my passport or my birth certificate. Its preservation is certainly worth a bit of thought.
- disabled 6y ago
  > Like you said, backups are annoying and unglamorous. It’s called having a network attached storage (NAS) device. I have a Synology NAS, which I backup to, continuously at 5 minute intervals. Warning: Microsoft image and file backups sometimes do not work. I recommend Acronis True Image instead, which comes with antivirus. It pretty much always works, never falter never fail. Get the version that allows you to back up to the cloud with blockchain features. You will be happy you did.
  - tenebrisalietum 6y ago
    Can ransomware leak on to your always-connected NAS?
    - disabled 6y ago
      Yes, of course. This is one of the reasons why I have 3 copies of everything I use (local/computer, cloud, NAS). This is particularly why I have a cloud backup, with blockchain data authentication. But, this is a basic overview of how to prevent NAS ransomware attacks: https://www.howtogeek.com/435452/how-to-secure-your-synology-nas-from-ransomware/ https://www.howtogeek.com/435452/how-to-secure-your-synology... There is better advice elsewhere but this is a good start. This may be a good comment to look at: https://news.ycombinator.com/item?id=25618346 https://news.ycombinator.com/item?id=25618346 This may be another good comment to take a look at: https://news.ycombinator.com/item?id=24860863 https://news.ycombinator.com/item?id=24860863
      - spideymans 6y ago
        Can you elaborate some more on your cloud backup solution? More specifically, I'm curious about the privacy/security aspect. It's the main reason why I've been somewhat hesitant to adopt cloud backups for my own computers. \
        disabled 6y ago
        I encrypt all of my information before I back it up. It is a risk that I am willing to take. Governments can always decrypt my information. Bruce Schneier, a world-renowned security expert who wrote "Click Here to Kill Everybody", states that he does not recommend storing information in foreign servers, in countries that you are not a citizen of. I suggest taking that advice. Acronis is Swiss-based and has to comply with the GDPR, due to its direct ties with the European Union. I am a dual US|EU (Croatian) citizen, with legal rights to work/live/retire in Switzerland, so I do not have to worry about storing my information "abroad". The EU has strict regulations, and they are only going to get stricter. There have even been talks about the EU being allowed to legally break encryption recently. If you are a third-country national (not an EU/EEA/Swiss citizen), then I do not recommend Acronis cloud backup or any other EU/EEA/Swiss service for storing your personal data. America may have its problems, and privacy may be a joke, but at least you have rights and sovereignty there.
  - nicbou 6y ago
    I use rsync and hard links to create incremental backups, and Google Drive as a convenience backup. First to a collection point, then to remote locations. It's actually managed by a side project of mine: https://github.com/nicbou/timeline https://github.com/nicbou/timeline
- AnIdiotOnTheNet 6y ago
  ...really? What are you keeping on your laptop that is so important to you? I can't think of anything on my computer that I couldn't lose and pretty much just shrug off.
  - nicbou 6y ago
    Photos, journals and other personal data.
tomrod 6y ago
> No, the automatic 7 day RDS snapshots or turning on S3 versioning is not a sufficient backup. Neither is mirroring to a S3 Glacier bucket in the same org, or rsyncing to a a backup server in the same datacenter. Why?
- danielheath 6y ago
  A few reasons come to mind. If you lose control of an aws root account it can take weeks to get it back. That’s probably enough time for the hackers to clean out the backups. Billing issues can lead to aws wiping out an account. For $work the backups are in AWS but using a different payment method, account owner etc to prevent cross contamination. Honestly, they should be outside aws entirely, but separate accounts is a good start.
  - viraptor 6y ago
    > If you lose control of an aws root account it can take weeks to get it back. > Billing issues can lead to aws wiping out an account. These are small-scale related issues though. When you're at the level where it's hard to have a reasonable backup outside of AWS, you can also resolve the root account issues fairly quickly by calling your TAM directly.
    - deleted 6y ago
      [deleted]
- tooltower 6y ago
  Because they share the failure domain. In all of those cases, there's a single point of failure.
m3nu 6y ago
> Backups are annoying and unglamorous Three years ago, after doing YC Startup School, I built https://www.borgbase.com https://www.borgbase.com to offer the simple, but secure backup service I wanted myself. Today it’s a viable business and my customers are all great and value backups as essential part of their own business. Wouldn’t want to be in any other “more glamorous” corner of the industry. Also kudos to anyone - partner or competitor - working in this “unglamorous” space. You’re all doing great work.
- novok 6y ago
  Do you do offsite offline backups too with verification? What if your infra gets really hacked and they wipe out all of your customer's backup data everywhere? Just because borg clients have append only modes, it doesn't stop them from deleting the raw files on your drives.
  - m3nu 6y ago
    If a storage server gets p0wned, the raw data could be deleted. That's true for every cloud provider. What's important, they still can't read the backup, since it's encrypted on the client. Storage servers are also isolated from each other and in different DCs, cities and regions. Additional offline backups aren't really feasible past a certain data volume and daily change/velocity. I'd still encourage everyone to have them for their own essential data in addition to a cloud backup. E.g. by burning it to BluRay or tape (3-2-1 rule). You can see find my own, more philosophical discussion, of the topic here: https://docs.borgbase.com/strategy/ https://docs.borgbase.com/strategy/. There I also distinguish between operational backups and archives. BorgBase is focused on the former. Offline backups are more suited for the latter.
    - citrin_ru 6y ago
      > If a storage server gets p0wned, the raw data could be deleted. That's true for every cloud provider. It's not hard to create a cloud backup service where delete requires separate credentials which are not used in day-to-day operations (and so can be kept secure). And without these credentials a backup is kept N days and cannot be deleted or overwritten. Don't know if anyone do this, though.
SMAAART 6y ago
> Backups are annoying and unglamorous It gets better (as in worse) someone can easily cut down back-up expenses, and become a hero by "balancing the budget with no disruptions to operations", get a fat bonus, and then after a year or so, leave. Their successors won't get any bonuses by increasing the budget for something that has no ROI. And randsomware is booming!
- bitexploder 6y ago
  This is a somewhat cynical take, but there are kernels of truth in there. Often, organizations not focused on quality will allow such things to happen. More than one startup has also been out out of business by not having a backup. But it’s often negligence / ignorance and not budget. When you put all the cards on the table for an organization and give them the information they need, they make much better security choices.
  - SMAAART 6y ago
    It’s negligence / ignorance that leas to stupid budget cuts.
  - bryanrasmussen 6y ago
    I think backups come up some time after MVP and launch for most organizations, this means that you don't know when it actually has to come, so someone with technical skills says hey at some point we need to put in the effort and money to get good backups and ability to restore of all this data, and someone in charge of how the business grows has to say yes we better do it. Unless you do not open for any sort of business without having all backup system set up then you run the risk, and with thousands of companies running the risk so as to not be out-competed by faster, riskier companies someone will suffer a bad outcome of their risk-taking.
- chalst 6y ago
  > Their successors won't get any bonuses by increasing the budget for something that has no ROI. Possibly, but not if the business has a culture of recognising downside risk. Single events that are not extremely unlikely and that could cripple the business are things that every company should be looking out for.
AnIdiotOnTheNet 6y ago
> I'm continuously surprised how often the answer turns out to be no after dinner digging, even in larger companies with otherwise well-run IT. I am not even remotely surprised, because most companies are not IT oriented companies, and most of those either have managers that can't be made to understand the importance of DR, or (sadly) IT people who can't. I myself am currently in a multi-year long political battle to justify a mere $600mo to get our DR env (built with old equipment that is on the verge of irrevocable failure) moved from a closet at a branch location to an actual co-lo that won't lose power and internet 3 times a week. At least we finally got that backup generator approved for the corporate office after the third time a goose committed suicide on our power lines. Ugh...
jiux 6y ago
Thank you for bringing this to light. I have been heads down on my own side hustle as a one man show. Do you have any suggestions on how to properly approach backups for smaller groups like myself working with limited funds? For context, some of the tech on my plate includes a few DO ubuntu droplets (hosting docker containerized services), Postgres DB (prod DO managed w/ auto-snaps, pretty much same scenario you mentioned), S3 storage for user-created assets (DO spaces w/ no backup strategy yet; no, not launched, yet), GitHub for source code, and physical MacBooks (iCloud + manual backups to a single physical external SSD). Realistically speaking I’m bootstrapped. More importantly, I’m interested in learning the right ways in doing backups.
WrtCdEvrydy 6y ago
The horror stories I had shared with my boss are as relevant today as when they happened. His is from the early 90s, the RAID card on the backup server was bad so it was just randomly flipping bits on the files being stored... nothing was ever tested until they needed one of those backups. Mine was about a tape setup that followed the correct 3-2-1 setup... one of the CLI tools had an update and the agreement was waiting for user input so it would just drop and the rest of the script continued... no one had ever tested a backup so the tapes being shuffled around were empty.

nicbou 6y ago

This applies to average people too. I wonder who among us can say they meet your (reasonable) standard. Like you said, backups are annoying and unglamorous. Yet, the data on my laptop is the only thing I could not replace. It's more important to me than my passport or my birth certificate. Its preservation is certainly worth a bit of thought.

disabled 6y ago
> Like you said, backups are annoying and unglamorous. It’s called having a network attached storage (NAS) device. I have a Synology NAS, which I backup to, continuously at 5 minute intervals. Warning: Microsoft image and file backups sometimes do not work. I recommend Acronis True Image instead, which comes with antivirus. It pretty much always works, never falter never fail. Get the version that allows you to back up to the cloud with blockchain features. You will be happy you did.
- tenebrisalietum 6y ago
  Can ransomware leak on to your always-connected NAS?
  - disabled 6y ago
    Yes, of course. This is one of the reasons why I have 3 copies of everything I use (local/computer, cloud, NAS). This is particularly why I have a cloud backup, with blockchain data authentication. But, this is a basic overview of how to prevent NAS ransomware attacks: https://www.howtogeek.com/435452/how-to-secure-your-synology-nas-from-ransomware/ https://www.howtogeek.com/435452/how-to-secure-your-synology... There is better advice elsewhere but this is a good start. This may be a good comment to look at: https://news.ycombinator.com/item?id=25618346 https://news.ycombinator.com/item?id=25618346 This may be another good comment to take a look at: https://news.ycombinator.com/item?id=24860863 https://news.ycombinator.com/item?id=24860863
    - spideymans 6y ago
      Can you elaborate some more on your cloud backup solution? More specifically, I'm curious about the privacy/security aspect. It's the main reason why I've been somewhat hesitant to adopt cloud backups for my own computers. \
      - disabled 6y ago
        I encrypt all of my information before I back it up. It is a risk that I am willing to take. Governments can always decrypt my information. Bruce Schneier, a world-renowned security expert who wrote "Click Here to Kill Everybody", states that he does not recommend storing information in foreign servers, in countries that you are not a citizen of. I suggest taking that advice. Acronis is Swiss-based and has to comply with the GDPR, due to its direct ties with the European Union. I am a dual US|EU (Croatian) citizen, with legal rights to work/live/retire in Switzerland, so I do not have to worry about storing my information "abroad". The EU has strict regulations, and they are only going to get stricter. There have even been talks about the EU being allowed to legally break encryption recently. If you are a third-country national (not an EU/EEA/Swiss citizen), then I do not recommend Acronis cloud backup or any other EU/EEA/Swiss service for storing your personal data. America may have its problems, and privacy may be a joke, but at least you have rights and sovereignty there.
- nicbou 6y ago
  I use rsync and hard links to create incremental backups, and Google Drive as a convenience backup. First to a collection point, then to remote locations. It's actually managed by a side project of mine: https://github.com/nicbou/timeline https://github.com/nicbou/timeline
AnIdiotOnTheNet 6y ago
...really? What are you keeping on your laptop that is so important to you? I can't think of anything on my computer that I couldn't lose and pretty much just shrug off.
- nicbou 6y ago
  Photos, journals and other personal data.

tomrod 6y ago

> No, the automatic 7 day RDS snapshots or turning on S3 versioning is not a sufficient backup. Neither is mirroring to a S3 Glacier bucket in the same org, or rsyncing to a a backup server in the same datacenter. Why?

danielheath 6y ago
A few reasons come to mind. If you lose control of an aws root account it can take weeks to get it back. That’s probably enough time for the hackers to clean out the backups. Billing issues can lead to aws wiping out an account. For $work the backups are in AWS but using a different payment method, account owner etc to prevent cross contamination. Honestly, they should be outside aws entirely, but separate accounts is a good start.
- viraptor 6y ago
  > If you lose control of an aws root account it can take weeks to get it back. > Billing issues can lead to aws wiping out an account. These are small-scale related issues though. When you're at the level where it's hard to have a reasonable backup outside of AWS, you can also resolve the root account issues fairly quickly by calling your TAM directly.
  - deleted 6y ago
    [deleted]
tooltower 6y ago
Because they share the failure domain. In all of those cases, there's a single point of failure.

m3nu 6y ago

> Backups are annoying and unglamorous Three years ago, after doing YC Startup School, I built https://www.borgbase.com https://www.borgbase.com to offer the simple, but secure backup service I wanted myself. Today it’s a viable business and my customers are all great and value backups as essential part of their own business. Wouldn’t want to be in any other “more glamorous” corner of the industry. Also kudos to anyone - partner or competitor - working in this “unglamorous” space. You’re all doing great work.

novok 6y ago
Do you do offsite offline backups too with verification? What if your infra gets really hacked and they wipe out all of your customer's backup data everywhere? Just because borg clients have append only modes, it doesn't stop them from deleting the raw files on your drives.
- m3nu 6y ago
  If a storage server gets p0wned, the raw data could be deleted. That's true for every cloud provider. What's important, they still can't read the backup, since it's encrypted on the client. Storage servers are also isolated from each other and in different DCs, cities and regions. Additional offline backups aren't really feasible past a certain data volume and daily change/velocity. I'd still encourage everyone to have them for their own essential data in addition to a cloud backup. E.g. by burning it to BluRay or tape (3-2-1 rule). You can see find my own, more philosophical discussion, of the topic here: https://docs.borgbase.com/strategy/ https://docs.borgbase.com/strategy/. There I also distinguish between operational backups and archives. BorgBase is focused on the former. Offline backups are more suited for the latter.
  - citrin_ru 6y ago
    > If a storage server gets p0wned, the raw data could be deleted. That's true for every cloud provider. It's not hard to create a cloud backup service where delete requires separate credentials which are not used in day-to-day operations (and so can be kept secure). And without these credentials a backup is kept N days and cannot be deleted or overwritten. Don't know if anyone do this, though.

SMAAART 6y ago

> Backups are annoying and unglamorous It gets better (as in worse) someone can easily cut down back-up expenses, and become a hero by "balancing the budget with no disruptions to operations", get a fat bonus, and then after a year or so, leave. Their successors won't get any bonuses by increasing the budget for something that has no ROI. And randsomware is booming!

bitexploder 6y ago
This is a somewhat cynical take, but there are kernels of truth in there. Often, organizations not focused on quality will allow such things to happen. More than one startup has also been out out of business by not having a backup. But it’s often negligence / ignorance and not budget. When you put all the cards on the table for an organization and give them the information they need, they make much better security choices.
- SMAAART 6y ago
  It’s negligence / ignorance that leas to stupid budget cuts.
- bryanrasmussen 6y ago
  I think backups come up some time after MVP and launch for most organizations, this means that you don't know when it actually has to come, so someone with technical skills says hey at some point we need to put in the effort and money to get good backups and ability to restore of all this data, and someone in charge of how the business grows has to say yes we better do it. Unless you do not open for any sort of business without having all backup system set up then you run the risk, and with thousands of companies running the risk so as to not be out-competed by faster, riskier companies someone will suffer a bad outcome of their risk-taking.
chalst 6y ago
> Their successors won't get any bonuses by increasing the budget for something that has no ROI. Possibly, but not if the business has a culture of recognising downside risk. Single events that are not extremely unlikely and that could cripple the business are things that every company should be looking out for.

AnIdiotOnTheNet 6y ago

> I'm continuously surprised how often the answer turns out to be no after dinner digging, even in larger companies with otherwise well-run IT. I am not even remotely surprised, because most companies are not IT oriented companies, and most of those either have managers that can't be made to understand the importance of DR, or (sadly) IT people who can't. I myself am currently in a multi-year long political battle to justify a mere $600mo to get our DR env (built with old equipment that is on the verge of irrevocable failure) moved from a closet at a branch location to an actual co-lo that won't lose power and internet 3 times a week. At least we finally got that backup generator approved for the corporate office after the third time a goose committed suicide on our power lines. Ugh...

jiux 6y ago

Thank you for bringing this to light. I have been heads down on my own side hustle as a one man show. Do you have any suggestions on how to properly approach backups for smaller groups like myself working with limited funds? For context, some of the tech on my plate includes a few DO ubuntu droplets (hosting docker containerized services), Postgres DB (prod DO managed w/ auto-snaps, pretty much same scenario you mentioned), S3 storage for user-created assets (DO spaces w/ no backup strategy yet; no, not launched, yet), GitHub for source code, and physical MacBooks (iCloud + manual backups to a single physical external SSD). Realistically speaking I’m bootstrapped. More importantly, I’m interested in learning the right ways in doing backups.

WrtCdEvrydy 6y ago

The horror stories I had shared with my boss are as relevant today as when they happened. His is from the early 90s, the RAID card on the backup server was bad so it was just randomly flipping bits on the files being stored... nothing was ever tested until they needed one of those backups. Mine was about a tape setup that followed the correct 3-2-1 setup... one of the CLI tools had an update and the agreement was waiting for user input so it would just drop and the rest of the script continued... no one had ever tested a backup so the tapes being shuffled around were empty.