5 ms·

backups are a one-way hash unless you've tested your restoration

by dd82 5y ago

Spivak 5y ago
While it's always good to state the importance of testing your backups the fear that HN seems to have surrounding backups seems really out-of-touch. You would believe that everyone's backups are teetering on the edge of failure constantly. If your backup process is so brittle and complex that you assume that it will be broken when you need it you should probably do something about that. It's not that hard to have the things that store your data spit it out on a schedule to be transferred somewhere write-only and durable.
- kuratkull 5y ago
  Getting it back into a fresh live server without warnings and errors is the tricky part.
  - davidgh 5y ago
    But if you have a backup of the data, at least it’s possible.
  - bluGill 5y ago
    Oh sure, but if you have the data you can hire people to get it done. The important part is having the data, if the data is there (meaning not encrypted) money and time can get it working. I'd much prefer to have a good backup media vs having to find someone who knows how to read faded data from a way overused and well out of date tape, but there is technology to get back data from such bad backups. The important part is having the backup in some form. Having a well tested restore ability is a great idea, but not nearly as important as having the backups in the first place. Most backup programs are designed for restore, even if you screwed up, odds are you can get the data back later.
    - marcosdumay 5y ago
      If you can't automatically restore, it's because you've lost data. There is a huge variance on the data value and difficulty or recreating it, but a backup should let you clearly and unambiguously recreate whatever system you are backing up.
      - bluGill 5y ago
        Not exactly. There are a lot of possible scenarios where the data isn't last, but can't be automatically restored. A broken tape for example isn't lost data, but it will be a big pain to recover from. If the data isn't properly indexed you will have a big problem figuring out what file goes to what machine, but the data can still be there. You will have a hard time to find a backup program that doesn't have a good and tested restore procedure. However that doesn't mean it works in your particular edge cases. Even if the backups would work perfectly, this forced downtime might the best time to apply some change that your admins have known should be done for a while but couldn't afford the downtime. (you couldn't do a schema update, but there are some smaller config changes that still require taking the master database done for a bit)
    - kuratkull 5y ago
      Having a downed database server for eg. 24h (while you are out trying to hire people to fix the failing restore for you) will probably mean you lose many of your B2B service users. Your clients might also try out that shiny new SLA you promised them. Backups also ensure business continuity - which might be more important than past data for some workflows.
      - bluGill 5y ago
        Of course, time is a factor here. Even in the best case you can't be back up and running in less than an hour, and most likely the best case is already looking at days to get everything back to where it was. Regardless, backups are the first priority. Then a tested restore procedure.
  - happymellon 5y ago
    Maybe, but it's cheaper and easier to have a bunch of techs on overtime than paying the ransom and hoping that they don't just take the money and run.
- tolbish 5y ago
  There is a difference between verifying that your data is backed up and verifying that you can do a complete operational restore on a moment's notice.
  - Spivak 5y ago
    Very few shops can do what you're describing. It's not something that is often worthwhile enough to get budget and engineering time. If you're reaching into your backups for a full restore then you're going to have data unavailability due to the nature of how cold backups work. Having delayed replicas and warm backups are a godsend when you have them but lots of shops don't want to pay at least 2x for their production db cluster and know the tradeoffs they're making.
- marcosdumay 5y ago
  > You would believe that everyone's backups are teetering on the edge of failure constantly. Hum... Backups aren't normally "at the edge of failure", the procedure either works or doesn't work. One must test to ensure the procedure works and continue working after all the environment changes done today. That is, except for proprietary formats, like Exchange. Those can fail at any time, retroactively.

backups are a one-way hash unless you've tested your restoration

by dd82 5y ago

backups are a one-way hash unless you've tested your restoration

While it's always good to state the importance of testing your backups the fear that HN seems to have surrounding backups seems really out-of-touch. You would believe that everyone's backups are teetering on the edge of failure constantly. If your backup process is so brittle and complex that you assume that it will be broken when you need it you should probably do something about that. It's not that hard to have the things that store your data spit it out on a schedule to be transferred somewhere write-only and durable.

kuratkull 5y ago
Getting it back into a fresh live server without warnings and errors is the tricky part.
- davidgh 5y ago
  But if you have a backup of the data, at least it’s possible.
- bluGill 5y ago
  Oh sure, but if you have the data you can hire people to get it done. The important part is having the data, if the data is there (meaning not encrypted) money and time can get it working. I'd much prefer to have a good backup media vs having to find someone who knows how to read faded data from a way overused and well out of date tape, but there is technology to get back data from such bad backups. The important part is having the backup in some form. Having a well tested restore ability is a great idea, but not nearly as important as having the backups in the first place. Most backup programs are designed for restore, even if you screwed up, odds are you can get the data back later.
  - marcosdumay 5y ago
    If you can't automatically restore, it's because you've lost data. There is a huge variance on the data value and difficulty or recreating it, but a backup should let you clearly and unambiguously recreate whatever system you are backing up.
    - bluGill 5y ago
      Not exactly. There are a lot of possible scenarios where the data isn't last, but can't be automatically restored. A broken tape for example isn't lost data, but it will be a big pain to recover from. If the data isn't properly indexed you will have a big problem figuring out what file goes to what machine, but the data can still be there. You will have a hard time to find a backup program that doesn't have a good and tested restore procedure. However that doesn't mean it works in your particular edge cases. Even if the backups would work perfectly, this forced downtime might the best time to apply some change that your admins have known should be done for a while but couldn't afford the downtime. (you couldn't do a schema update, but there are some smaller config changes that still require taking the master database done for a bit)
  - kuratkull 5y ago
    Having a downed database server for eg. 24h (while you are out trying to hire people to fix the failing restore for you) will probably mean you lose many of your B2B service users. Your clients might also try out that shiny new SLA you promised them. Backups also ensure business continuity - which might be more important than past data for some workflows.
    - bluGill 5y ago
      Of course, time is a factor here. Even in the best case you can't be back up and running in less than an hour, and most likely the best case is already looking at days to get everything back to where it was. Regardless, backups are the first priority. Then a tested restore procedure.
- happymellon 5y ago
  Maybe, but it's cheaper and easier to have a bunch of techs on overtime than paying the ransom and hoping that they don't just take the money and run.
tolbish 5y ago
There is a difference between verifying that your data is backed up and verifying that you can do a complete operational restore on a moment's notice.
- Spivak 5y ago
  Very few shops can do what you're describing. It's not something that is often worthwhile enough to get budget and engineering time. If you're reaching into your backups for a full restore then you're going to have data unavailability due to the nature of how cold backups work. Having delayed replicas and warm backups are a godsend when you have them but lots of shops don't want to pay at least 2x for their production db cluster and know the tradeoffs they're making.
marcosdumay 5y ago
> You would believe that everyone's backups are teetering on the edge of failure constantly. Hum... Backups aren't normally "at the edge of failure", the procedure either works or doesn't work. One must test to ensure the procedure works and continue working after all the environment changes done today. That is, except for proprietary formats, like Exchange. Those can fail at any time, retroactively.