6 ms·

To each of you guys having those extensive backup solutions (like NAS + cloud sync, second nas, etc)... .. do you actually TEST those backups? This questions

by zytek 10y ago

To each of you guys having those extensive backup solutions (like NAS + cloud sync, second nas, etc)... .. do you actually TEST those backups? This questions comes from my experience as a system engineeer who found a critical bug in our MySQL backup solution that prevented them from restoring (inconsistent filesystem). Also, a friend of mine learned the hard way that his Backblaze backup was unrestorable.

majewsky 10y ago
Very true. I overheard a similar conversation last week at work: "We have set up the backup procedure for our new production databases." - "Have you tested restore?" - "Well, uhm..." - sound of JIRA ticket being opened By the way, I misread your username and, for a second, thought you were sytse.
- vassilevsky 10y ago
  I'd love to hear that sound :)
chadcmulligan 10y ago
it's not a backup till you test it - just complicated wishful thinking.
- chopin 10y ago
  As a colleague of mine says: You don't want a backup, you want a restore.
atmosx 10y ago
I have restored files from BB multiple times. It is a great solution for non technical ppl or offices that have at least 24Mbps connections up/down.
zapu 10y ago
I'm using CrashPlan and I have recovered multiple files over past couple of years, that I either mistakenly deleted or overwritten. I haven't tried any full-scale restore, yet, though.
- ValentineC 10y ago
  CrashPlan lost some data of mine in 2013 from querying a corrupted Volume Shadow Copy Services database on Windows. (At least, that was their explanation. I'm surprised that their client did not independently verify the data after it was uploaded.) I moved off CrashPlan in 2016 because their upload speed continues to be embarrassingly slow outside the US even with deduplication and compression turned off (they have a datacentre where I'm at, but it's for Enterprise customers only). They also highly recommend having 1GB of RAM for every 1TB backed up, which sounded a bit unreasonable to me.
  - wikibob 10y ago
    What did you switch to? Problem with BackBlaze and others is that they delete backups if not connected for 30 days, particularly external drives.
    - ValentineC 10y ago
      I moved to Acronis True Image when they offered unlimited cloud backups with their 2016 version. They probably couldn't sustain it, because I had to pay a lot more for backups when I wanted to renew in 2017. Now, I'm using both Arq and Synology's Hyper Backup with Amazon Cloud Drive. One of the problems I foresee is that while Amazon doesn't care how much data one stores in Cloud Drive, they have suspended users for downloading past an arbitrary limit in a certain period of time — so full restores might not be possible.
mironathetin 10y ago
".. do you actually TEST those backups?" yes (of course), see my post above. I have used time machine repeatedly to restore lost or damaged files. I also replaced harddrives several times and played back my carbon copy clone. It boots and I have never missed a file in years.
sharkoz 10y ago
Do you work at Gitlab ? :)
- m_samuel_l 10y ago
  too soon
Juliate 10y ago
Excellent point, and that begs another question: how do you actually test your backups? Of course, each case is specific, but is there a "best practice" checklist, or some general points to check for basic restoration?
- RossM 10y ago
  I've been thinking about this as well - it seems like it would fit in nicely with other CI jobs. With database backups, for example, you should be able to script the restore procedure and apply some assertions to check it worked. Bonus with this is that you now have a script when you actually need to restore.
- trcollinson 10y ago
  Excellent question. I do test my backups and restores on a rather constant basis. Each environment within my infrastructure takes a bit of a different approach. Application This is by far the easiest for me to test. We have a CI/CD jon which literally makes a new environment, from scratch, and deploys our application to it in a production configuration. It runs a test suite which tests functionality across the application. Finally, it destroys the environment. It reports on each portion of the process. In this way we know exactly how long it would take to redeploy the entire application from scratch on a new infrastructure and get it up and running. This morning it took about about 6 minutes total before tests ran. Database We are running an RDBMS. We use a combination of daily full backup, incremental transaction log like backup, and point in time backup. Again, in our CI/CD when a full backup is taken it is pulled, loaded, and a test routine is run against it to check integrity. At this time, the recovery from the day before is destroyed. When a transaction log backup is made, CI/CD picks up this change and applies it to the full backup restore and runs a set of tests for integrity check. This leaves us with a warm standby ready to be switched over to in case of the main database server going down. We have never had to use the warm standby in an emergency but we have a test to make sure we can cut that over as well. For point in time backup testing this goes back to our application test above. The application test will spin up with a point in time recovery of the database backup. It will test the integrity of that recovery and then test the application against it. Finally, it will swap from the point in time recovered database to the warm backup. It runs the test suite against that for integrity as well. File Store People often forget this but those buckets that get hold all of your file storage in the cloud can be destroyed so easily (sad, sad experience taught me this). We test those as well. I am sure you can guess at this point how we do that? CI/CD. It's a rather simple process with a ton of gain. A few notes People always ask me this, so I will answer it first. Yes this costs money. It's not as bad as running a second production environment. But it will cost you a bit. My follow up question is, how much does downtime cost you? My CI/CD is always Gitlab CI at this point. I've used Jenkins. I've used Travis. I like Gitlab CI. You can do all of this with any of those. We script literally everything. Computers are so good at repetitive tasks. Why would you EVER do anything manually? Really. If it has to do with your infrastructure, script it. If anyone has any questions about these ideas, feel free to reach out.
marcosdumay 10y ago
I'm a big fan of setting up testing and dev environments from the production backups. For personal backup of files, I just verify the results are in place. I've checked them once or twice, but honestly, I'm more concerned about my scripts stopping running than they running and not being correct.
- Daviey 10y ago
  That is fine, providing you don't operate in a confidential or regulated environment. :/
  - marcosdumay 10y ago
    Yes, many warnings do apply.
RubyPinch 10y ago
what happened to the backblaze backup?
- CodeWriter23 10y ago
  Interesting question. I wonder if it was prior to when BB moved to the "direct wire" architecture in Storage Pod 4.0.
BatFastard 10y ago
I had a company that I was doing some work for come to me to ask for a copy of the database. Their backs were corrupt, and it was not until they tried to restore it did they find out. But they have 5 years of bad backups
jmathai 10y ago
I have a script that checks for data rot[1]. It's part of a more comprehensive backup system[2]. [1] https://medium.com/vantage/how-to-protect-your-photos-from-bit-rot-9d0c6998121f#.xito0zp4r https://medium.com/vantage/how-to-protect-your-photos-from-b... [2] https://medium.com/swlh/my-automated-photo-workflow-using-google-photos-and-elodie-afb753b8c724#.1qwnqkmr8 https://medium.com/swlh/my-automated-photo-workflow-using-go...
notheguyouthink 10y ago
This is one thing i like about doing content addressed storage. I've been toying with my own implementation, quite similar to Camlistore. The net result is it's super simple to verify an entire datastore as being valid or not.

by zytek 10y ago

majewsky 10y ago

Very true. I overheard a similar conversation last week at work: "We have set up the backup procedure for our new production databases." - "Have you tested restore?" - "Well, uhm..." - sound of JIRA ticket being opened By the way, I misread your username and, for a second, thought you were sytse.

vassilevsky 10y ago
I'd love to hear that sound :)

chadcmulligan 10y ago

it's not a backup till you test it - just complicated wishful thinking.

chopin 10y ago
As a colleague of mine says: You don't want a backup, you want a restore.

atmosx 10y ago

I have restored files from BB multiple times. It is a great solution for non technical ppl or offices that have at least 24Mbps connections up/down.

zapu 10y ago

I'm using CrashPlan and I have recovered multiple files over past couple of years, that I either mistakenly deleted or overwritten. I haven't tried any full-scale restore, yet, though.

ValentineC 10y ago
CrashPlan lost some data of mine in 2013 from querying a corrupted Volume Shadow Copy Services database on Windows. (At least, that was their explanation. I'm surprised that their client did not independently verify the data after it was uploaded.) I moved off CrashPlan in 2016 because their upload speed continues to be embarrassingly slow outside the US even with deduplication and compression turned off (they have a datacentre where I'm at, but it's for Enterprise customers only). They also highly recommend having 1GB of RAM for every 1TB backed up, which sounded a bit unreasonable to me.
- wikibob 10y ago
  What did you switch to? Problem with BackBlaze and others is that they delete backups if not connected for 30 days, particularly external drives.
  - ValentineC 10y ago
    I moved to Acronis True Image when they offered unlimited cloud backups with their 2016 version. They probably couldn't sustain it, because I had to pay a lot more for backups when I wanted to renew in 2017. Now, I'm using both Arq and Synology's Hyper Backup with Amazon Cloud Drive. One of the problems I foresee is that while Amazon doesn't care how much data one stores in Cloud Drive, they have suspended users for downloading past an arbitrary limit in a certain period of time — so full restores might not be possible.

mironathetin 10y ago

".. do you actually TEST those backups?" yes (of course), see my post above. I have used time machine repeatedly to restore lost or damaged files. I also replaced harddrives several times and played back my carbon copy clone. It boots and I have never missed a file in years.

sharkoz 10y ago

Do you work at Gitlab ? :)

m_samuel_l 10y ago
too soon

Juliate 10y ago

Excellent point, and that begs another question: how do you actually test your backups? Of course, each case is specific, but is there a "best practice" checklist, or some general points to check for basic restoration?

RossM 10y ago
I've been thinking about this as well - it seems like it would fit in nicely with other CI jobs. With database backups, for example, you should be able to script the restore procedure and apply some assertions to check it worked. Bonus with this is that you now have a script when you actually need to restore.
trcollinson 10y ago
Excellent question. I do test my backups and restores on a rather constant basis. Each environment within my infrastructure takes a bit of a different approach. Application This is by far the easiest for me to test. We have a CI/CD jon which literally makes a new environment, from scratch, and deploys our application to it in a production configuration. It runs a test suite which tests functionality across the application. Finally, it destroys the environment. It reports on each portion of the process. In this way we know exactly how long it would take to redeploy the entire application from scratch on a new infrastructure and get it up and running. This morning it took about about 6 minutes total before tests ran. Database We are running an RDBMS. We use a combination of daily full backup, incremental transaction log like backup, and point in time backup. Again, in our CI/CD when a full backup is taken it is pulled, loaded, and a test routine is run against it to check integrity. At this time, the recovery from the day before is destroyed. When a transaction log backup is made, CI/CD picks up this change and applies it to the full backup restore and runs a set of tests for integrity check. This leaves us with a warm standby ready to be switched over to in case of the main database server going down. We have never had to use the warm standby in an emergency but we have a test to make sure we can cut that over as well. For point in time backup testing this goes back to our application test above. The application test will spin up with a point in time recovery of the database backup. It will test the integrity of that recovery and then test the application against it. Finally, it will swap from the point in time recovered database to the warm backup. It runs the test suite against that for integrity as well. File Store People often forget this but those buckets that get hold all of your file storage in the cloud can be destroyed so easily (sad, sad experience taught me this). We test those as well. I am sure you can guess at this point how we do that? CI/CD. It's a rather simple process with a ton of gain. A few notes People always ask me this, so I will answer it first. Yes this costs money. It's not as bad as running a second production environment. But it will cost you a bit. My follow up question is, how much does downtime cost you? My CI/CD is always Gitlab CI at this point. I've used Jenkins. I've used Travis. I like Gitlab CI. You can do all of this with any of those. We script literally everything. Computers are so good at repetitive tasks. Why would you EVER do anything manually? Really. If it has to do with your infrastructure, script it. If anyone has any questions about these ideas, feel free to reach out.

marcosdumay 10y ago

I'm a big fan of setting up testing and dev environments from the production backups. For personal backup of files, I just verify the results are in place. I've checked them once or twice, but honestly, I'm more concerned about my scripts stopping running than they running and not being correct.

Daviey 10y ago
That is fine, providing you don't operate in a confidential or regulated environment. :/
- marcosdumay 10y ago
  Yes, many warnings do apply.

RubyPinch 10y ago

what happened to the backblaze backup?

CodeWriter23 10y ago
Interesting question. I wonder if it was prior to when BB moved to the "direct wire" architecture in Storage Pod 4.0.

BatFastard 10y ago

I had a company that I was doing some work for come to me to ask for a copy of the database. Their backs were corrupt, and it was not until they tried to restore it did they find out. But they have 5 years of bad backups

jmathai 10y ago

I have a script that checks for data rot[1]. It's part of a more comprehensive backup system[2]. [1] https://medium.com/vantage/how-to-protect-your-photos-from-bit-rot-9d0c6998121f#.xito0zp4r https://medium.com/vantage/how-to-protect-your-photos-from-b... [2] https://medium.com/swlh/my-automated-photo-workflow-using-google-photos-and-elodie-afb753b8c724#.1qwnqkmr8 https://medium.com/swlh/my-automated-photo-workflow-using-go...

notheguyouthink 10y ago

This is one thing i like about doing content addressed storage. I've been toying with my own implementation, quite similar to Camlistore. The net result is it's super simple to verify an entire datastore as being valid or not.