7 ms·

World Backup Day

105 points by unnervingduck 2y ago

ahazred8ta 2y ago
A gentle reminder to actually TEST your restore workflow. WBD: since 2011! https://hn.algolia.com/?q=worldbackupday https://hn.algolia.com/?q=worldbackupday
- dewey 2y ago
  People always say that but what do they actually mean? Restore one file, check if the backup is bootable in case it should?
  - 369548684892826 2y ago
    For static files like photos I hash check drives against each other to check for bit rot. But yeah, for tape based cloud storage I can't think what else to do except restore one file to check I still have access. It's too expensive to restore the whole archive.
    - lloydatkinson 2y ago
      Do you have any automation or scripts for that, or is it usually adhoc?
      - dmd 2y ago
        You absolutely should NOT automate this. If you do, you then need to manually check that automation with the same frequency- so you haven’t gained anything!
        369548684892826 2y ago
        I do have this automated, but the checksum output is posted to a kind of watchdog service. Every day I get an email that says everything is as expected, or not.
        lloydatkinson 2y ago
        I need to google every time to find a good process for hashing files and comparing across disks? Not only that, but remember to do it frequently enough?
      - 369548684892826 2y ago
        Yeah, I use rsync with `-cavin`, if the output includes `>` or `<` then there are differences in checksums. #!/bin/bash hostname=$(hostname) checksumFile=/tmp/checksum-file.txt rsync -cavin --info=name2 --no-perms --no-owner --no-group /srv/data/photos/ user@remote:/media/data/rsnapshot/daily.0/srv/data/photos/ > $checksumFile minimumSize=5000 actualSize=$(wc -l <"$checksumFile") if [ $actualSize -ge $minimumSize ]; then fileDiffs=$(cat $checksumFile | grep -e '<' -e '>' || echo 'all checksums match') else fileDiffs=$(echo $checksumFile is too small) fi echo $fileDiffs
  - rzzzt 2y ago
    You can also restore the entire backup to /dev/null (this device has lots of free space) or pipe the output into a checksum generator. Borg also has a consistency checking command.
  - tomatocracy 2y ago
    I'm sure there will be plenty of advice for testing the backups themselves, but for online backups (either to server or to a service), one thing which is easy to forget to do is to check you can still get access from "scratch" - ie if you have lost all local data. Make sure you don't need eg passwords or keys for the online backup service which you have no other copy of (doesn't help if they are in the backup itself!) or indeed for email/cloud accounts which you might need to regain access to to get to your backups.
  - wjnc 2y ago
    You’d actually want to think a bit more. Where is your paper decryption key? Is it off premises? Test a, some or all of the files in the archive. Boot into an ISO. Try it again. And that for all of your backups. It’s not a backup until it is tested.
crossroadsguy 2y ago
I was happy with my restic, borg (vorta), and tarsnap backup setup until one day I had to retrieve just one file from the backups and I realised there was no straight forward way to do that in any of these. Now I think I must look for backup which lets be “easily” get my files back and search whether something is there by a certain approximate name among my backups without having me deal with mounts (that too one version at a time) and often fail. It would be nice to just know that file/dir “abc xyz” or with similar names were backed up in snapshots m, n,…, z. Then I can just fetch the version I want.
- ehecatl42 2y ago
  Not sure that I understand. What's hard about `restic -r /media/ehecatl42/t14g3-backup/t14g3-restic-repo restore latest --target /home/ehecatl42/Desktop/nvim-restore/ --include /home/ehecatl42/.config/nvim/`* and just `cp`ing your missing files from that. * From my recent .bash_history
  - dewey 2y ago
    If you want to make a point about something being “not hard” maybe using a less messy command would bring that point across better ;)
    - viraptor 2y ago
      It may look messy but this is really the minimal amount of info. You need the location of backup, operation, time for restoration, pattern to restore and the destination. If you already set up the backup part of restic, then this command shouldn't be hard to understand / reproduce at all. You could maybe default to latest and default to restoring to ".", but that carries some risks. This is only as complex as necessary.
      - dewey 2y ago
        > this command shouldn't be hard to understand I understand the command, I'm just questioning the example if they want to do some positive marketing for the tool. Something simple like the following gives the same information but is much more understandable than unnecessarily convoluted directory names. restic -r /backups/restic restore latest --target /home/dewey --include /home/dewey/myfile.txt
submeta 2y ago
Very happy with Time Machine and Carbon Copy Cloner on my Mac. I have a dozen hard drives and SSDs that I rotate and a NAS, but I don’t use that for backup. Never lost a file. But a reminder to myself: I need off site backups.
Flimm 2y ago
The website should also encourage users to download a copy of their online accounts (Google Takeout, Facebook data, etc). That's a pretty big omission in 2024. I know plenty of people who have been locked out of their own accounts for one reason or another.
- Simon_ORourke 2y ago
  That's probably a pretty massive download for most people, but I guess well worth it.
- ndsipa_pomu 2y ago
  Nice one - you reminded me to restart my Thunderbird backup of GMail. I've got it setup to use imap to connect to GMail (a couple of different accounts as it happens) so it's got a copy of all my emails and keeps itself up to date efficiently. As a bonus, it's far more capable at deleting old messages if I need to clear some space to keep under Google's limit.
  - racked 2y ago
    Try running 'getmail' with a daily cron job. It's an old piece of software, and (at least for me) required an initial `for` loop to catch up with old mail, but it's the best low-maintenance way to backup Gmail AFAIK. Deleting old mail is another story though.
cjk2 2y ago
Coincidentally this landed on my weekly backup day. Using Time Machine weekly to an external SSD. Also a separate flat monthly archive to another SSD. And just in case that is not enough, rclone my documents and pictures library (everything) to S3 monthly too. This is restored once a quarter into my spare Mac mini.
- hammyhavoc 2y ago
  After having so many sudden failures on SSDs (usually the controller), you may want to consider a spinning rust backup too as they're a little more forgiving in many regards, even if they're more susceptible to shock damage. In the same breath, lost a lot of spinning rusts over the years, and been at the mercy of data recovery services. Multiple copies, both cold and hot ones backed by parity are important. I've lost drives I'm restoring backups from multiple times too, as restoration operations tend to be intensive and extended. I also don't recommend the average external spinning rust in a caddy as they're usually not very well ventilated, so they can get very hot and suffer from diminished longevity. Not to mention that a lot of off-the-shelf external drives already in an enclosure are frequently B-grade, or refurbished drives. Yes, even from big manufacturers. Storage is fairly woeful.
dewey 2y ago
Went through many iterations over the years. Arq, restic, borg, tarsnap setting up encryption, incremental backups with cronjobs and all the good stuff. Some years ago I realized that I value the possibility to restore also for people that are not me higher than the nerd factor and security so I just settled with Backblaze as my main backup and Time Machine for local convenience. Carbon Copy Cloned to just clone my attached drives to my NAS if they are attached. For me backups are not the place to fiddle around with obscure solutions that nobody in my family would be able to use.
- 10729287 2y ago
  Definitely. And write things !
- crossroadsguy 2y ago
  I am also doing borg, restic, tarsnap currently (evaluating Kopia) but of late a fatigue has been setting in. My best backup (and in fact recovery) experience has been with CrashPlan (Personal) really. Nothing comes close to it somehow. Tinkering, deduplication, compression all are hunky dory but at the end of the day I just want to be able to easily visualise what I do and what I need and what I am searching et cetera.
- PorterBHall 2y ago
  I have made my system pretty basic, relying on cron and rsync, because I figure that’s the most futureproof and secure way. But you helped me realize that if something happened to me and our NAS, my family would not be able to restore from backup. I’ll have to work on that.
1vuio0pswjnm7 2y ago
https://web.archive.org/web/20140325225131if_/http://www.worldbackupday.com/en/ https://web.archive.org/web/20140325225131if_/http://www.wor...
hkt 2y ago
Really, world backup day should've been on Good Friday so that world restore day could be on Easter Monday. Jesus Saves! (and takes daily snapshots which he uploads to a secure offsite location)
- lifestyleguru 2y ago
  Jesus would restore too.
  - hagbard_c 2y ago
    He did, just today: https://www.britannica.com/topic/Easter-holiday https://www.britannica.com/topic/Easter-holiday
NKosmatos 2y ago
I would add a couple of important things: - mention of the 3-2-1 backup rule - include emails and other social accounts into the backup strategy - validating and restoring backups is as important as creating them
implements 2y ago
In case no-one’s aware, iPhone / iPad Files app has a “Connect to Server” (…) option which can connect to an SMB share, making it significantly easier to back up any downloaded or created files to a PC without having to install and use iTunes. And there’s plugging in the device via USB, which mounts the DCIM files (photos and videos) as a mass storage device, and allows to you to back up those, too.
- PorterBHall 2y ago
  Thanks for the tip! I missed that.
- Lammy 2y ago
  TIL — apparently since iOS 13 !
jenny91 2y ago
Just wait until this becomes so big that like backblaze starts crashing on March 31st because of the flood of uploads.
- atYevP 2y ago
  Yev from Backblaze here -> we welcome the challenge!
yakkomajuri 2y ago
This is very much about personal backups but while we're here - what open source backup tools do people recommend for very simple filesystem backups to a remote server?
- deleted 2y ago
  [deleted]
- 0cVlTeIATBs 2y ago
  FreeFileSync It's what you'd like if your idea of simplicity is dragging and dropping one folder to another, but want to more closely see what files have changed before the copy.
- cr125rider 2y ago
  rsync is great
  - yakkomajuri 2y ago
    Well the current system is a shell script with rsync that I wrote but I'd like something slightly more sophisticated without having to build logic myself for incremental backups, potentially encryption, etc, etc
    - afruitpie 2y ago
      They might have meant rclone: https://rclone.org/ https://rclone.org/
      - yakkomajuri 2y ago
        This looks interesting, thanks!
fragmede 2y ago
Anyone have a good way of backing up their HN writings?
n0n0n4t0r 2y ago
I work for a company that handles backup at the scale of the biggest companies in the world. And I promise you, what's hard is not the backup, it's it's security (to not lose it, to not let it get corrupt, to avoid any unintended usage, ...) and its restauration.

105 points by unnervingduck 2y ago

A gentle reminder to actually TEST your restore workflow. WBD: since 2011! https://hn.algolia.com/?q=worldbackupday https://hn.algolia.com/?q=worldbackupday

dewey 2y ago
People always say that but what do they actually mean? Restore one file, check if the backup is bootable in case it should?
- 369548684892826 2y ago
  For static files like photos I hash check drives against each other to check for bit rot. But yeah, for tape based cloud storage I can't think what else to do except restore one file to check I still have access. It's too expensive to restore the whole archive.
  - lloydatkinson 2y ago
    Do you have any automation or scripts for that, or is it usually adhoc?
    - dmd 2y ago
      You absolutely should NOT automate this. If you do, you then need to manually check that automation with the same frequency- so you haven’t gained anything!
      - 369548684892826 2y ago
        I do have this automated, but the checksum output is posted to a kind of watchdog service. Every day I get an email that says everything is as expected, or not.
      - lloydatkinson 2y ago
        I need to google every time to find a good process for hashing files and comparing across disks? Not only that, but remember to do it frequently enough?
    - 369548684892826 2y ago
      Yeah, I use rsync with `-cavin`, if the output includes `>` or `<` then there are differences in checksums. #!/bin/bash hostname=$(hostname) checksumFile=/tmp/checksum-file.txt rsync -cavin --info=name2 --no-perms --no-owner --no-group /srv/data/photos/ user@remote:/media/data/rsnapshot/daily.0/srv/data/photos/ > $checksumFile minimumSize=5000 actualSize=$(wc -l <"$checksumFile") if [ $actualSize -ge $minimumSize ]; then fileDiffs=$(cat $checksumFile | grep -e '<' -e '>' || echo 'all checksums match') else fileDiffs=$(echo $checksumFile is too small) fi echo $fileDiffs
- rzzzt 2y ago
  You can also restore the entire backup to /dev/null (this device has lots of free space) or pipe the output into a checksum generator. Borg also has a consistency checking command.
- tomatocracy 2y ago
  I'm sure there will be plenty of advice for testing the backups themselves, but for online backups (either to server or to a service), one thing which is easy to forget to do is to check you can still get access from "scratch" - ie if you have lost all local data. Make sure you don't need eg passwords or keys for the online backup service which you have no other copy of (doesn't help if they are in the backup itself!) or indeed for email/cloud accounts which you might need to regain access to to get to your backups.
- wjnc 2y ago
  You’d actually want to think a bit more. Where is your paper decryption key? Is it off premises? Test a, some or all of the files in the archive. Boot into an ISO. Try it again. And that for all of your backups. It’s not a backup until it is tested.

crossroadsguy 2y ago

I was happy with my restic, borg (vorta), and tarsnap backup setup until one day I had to retrieve just one file from the backups and I realised there was no straight forward way to do that in any of these. Now I think I must look for backup which lets be “easily” get my files back and search whether something is there by a certain approximate name among my backups without having me deal with mounts (that too one version at a time) and often fail. It would be nice to just know that file/dir “abc xyz” or with similar names were backed up in snapshots m, n,…, z. Then I can just fetch the version I want.

ehecatl42 2y ago
Not sure that I understand. What's hard about `restic -r /media/ehecatl42/t14g3-backup/t14g3-restic-repo restore latest --target /home/ehecatl42/Desktop/nvim-restore/ --include /home/ehecatl42/.config/nvim/`* and just `cp`ing your missing files from that. * From my recent .bash_history
- dewey 2y ago
  If you want to make a point about something being “not hard” maybe using a less messy command would bring that point across better ;)
  - viraptor 2y ago
    It may look messy but this is really the minimal amount of info. You need the location of backup, operation, time for restoration, pattern to restore and the destination. If you already set up the backup part of restic, then this command shouldn't be hard to understand / reproduce at all. You could maybe default to latest and default to restoring to ".", but that carries some risks. This is only as complex as necessary.
    - dewey 2y ago
      > this command shouldn't be hard to understand I understand the command, I'm just questioning the example if they want to do some positive marketing for the tool. Something simple like the following gives the same information but is much more understandable than unnecessarily convoluted directory names. restic -r /backups/restic restore latest --target /home/dewey --include /home/dewey/myfile.txt

submeta 2y ago

Very happy with Time Machine and Carbon Copy Cloner on my Mac. I have a dozen hard drives and SSDs that I rotate and a NAS, but I don’t use that for backup. Never lost a file. But a reminder to myself: I need off site backups.

Flimm 2y ago

The website should also encourage users to download a copy of their online accounts (Google Takeout, Facebook data, etc). That's a pretty big omission in 2024. I know plenty of people who have been locked out of their own accounts for one reason or another.

Simon_ORourke 2y ago
That's probably a pretty massive download for most people, but I guess well worth it.
ndsipa_pomu 2y ago
Nice one - you reminded me to restart my Thunderbird backup of GMail. I've got it setup to use imap to connect to GMail (a couple of different accounts as it happens) so it's got a copy of all my emails and keeps itself up to date efficiently. As a bonus, it's far more capable at deleting old messages if I need to clear some space to keep under Google's limit.
- racked 2y ago
  Try running 'getmail' with a daily cron job. It's an old piece of software, and (at least for me) required an initial `for` loop to catch up with old mail, but it's the best low-maintenance way to backup Gmail AFAIK. Deleting old mail is another story though.

cjk2 2y ago

Coincidentally this landed on my weekly backup day. Using Time Machine weekly to an external SSD. Also a separate flat monthly archive to another SSD. And just in case that is not enough, rclone my documents and pictures library (everything) to S3 monthly too. This is restored once a quarter into my spare Mac mini.

hammyhavoc 2y ago
After having so many sudden failures on SSDs (usually the controller), you may want to consider a spinning rust backup too as they're a little more forgiving in many regards, even if they're more susceptible to shock damage. In the same breath, lost a lot of spinning rusts over the years, and been at the mercy of data recovery services. Multiple copies, both cold and hot ones backed by parity are important. I've lost drives I'm restoring backups from multiple times too, as restoration operations tend to be intensive and extended. I also don't recommend the average external spinning rust in a caddy as they're usually not very well ventilated, so they can get very hot and suffer from diminished longevity. Not to mention that a lot of off-the-shelf external drives already in an enclosure are frequently B-grade, or refurbished drives. Yes, even from big manufacturers. Storage is fairly woeful.

dewey 2y ago

Went through many iterations over the years. Arq, restic, borg, tarsnap setting up encryption, incremental backups with cronjobs and all the good stuff. Some years ago I realized that I value the possibility to restore also for people that are not me higher than the nerd factor and security so I just settled with Backblaze as my main backup and Time Machine for local convenience. Carbon Copy Cloned to just clone my attached drives to my NAS if they are attached. For me backups are not the place to fiddle around with obscure solutions that nobody in my family would be able to use.

10729287 2y ago
Definitely. And write things !
crossroadsguy 2y ago
I am also doing borg, restic, tarsnap currently (evaluating Kopia) but of late a fatigue has been setting in. My best backup (and in fact recovery) experience has been with CrashPlan (Personal) really. Nothing comes close to it somehow. Tinkering, deduplication, compression all are hunky dory but at the end of the day I just want to be able to easily visualise what I do and what I need and what I am searching et cetera.
PorterBHall 2y ago
I have made my system pretty basic, relying on cron and rsync, because I figure that’s the most futureproof and secure way. But you helped me realize that if something happened to me and our NAS, my family would not be able to restore from backup. I’ll have to work on that.

1vuio0pswjnm7 2y ago

https://web.archive.org/web/20140325225131if_/http://www.worldbackupday.com/en/ https://web.archive.org/web/20140325225131if_/http://www.wor...

hkt 2y ago

Really, world backup day should've been on Good Friday so that world restore day could be on Easter Monday. Jesus Saves! (and takes daily snapshots which he uploads to a secure offsite location)

lifestyleguru 2y ago
Jesus would restore too.
- hagbard_c 2y ago
  He did, just today: https://www.britannica.com/topic/Easter-holiday https://www.britannica.com/topic/Easter-holiday

NKosmatos 2y ago

I would add a couple of important things: - mention of the 3-2-1 backup rule - include emails and other social accounts into the backup strategy - validating and restoring backups is as important as creating them

implements 2y ago

In case no-one’s aware, iPhone / iPad Files app has a “Connect to Server” (…) option which can connect to an SMB share, making it significantly easier to back up any downloaded or created files to a PC without having to install and use iTunes. And there’s plugging in the device via USB, which mounts the DCIM files (photos and videos) as a mass storage device, and allows to you to back up those, too.

PorterBHall 2y ago
Thanks for the tip! I missed that.
Lammy 2y ago
TIL — apparently since iOS 13 !

jenny91 2y ago

Just wait until this becomes so big that like backblaze starts crashing on March 31st because of the flood of uploads.

atYevP 2y ago
Yev from Backblaze here -> we welcome the challenge!

yakkomajuri 2y ago

This is very much about personal backups but while we're here - what open source backup tools do people recommend for very simple filesystem backups to a remote server?

deleted 2y ago
[deleted]
0cVlTeIATBs 2y ago
FreeFileSync It's what you'd like if your idea of simplicity is dragging and dropping one folder to another, but want to more closely see what files have changed before the copy.
cr125rider 2y ago
rsync is great
- yakkomajuri 2y ago
  Well the current system is a shell script with rsync that I wrote but I'd like something slightly more sophisticated without having to build logic myself for incremental backups, potentially encryption, etc, etc
  - afruitpie 2y ago
    They might have meant rclone: https://rclone.org/ https://rclone.org/
    - yakkomajuri 2y ago
      This looks interesting, thanks!

fragmede 2y ago

Anyone have a good way of backing up their HN writings?

n0n0n4t0r 2y ago

I work for a company that handles backup at the scale of the biggest companies in the world. And I promise you, what's hard is not the backup, it's it's security (to not lose it, to not let it get corrupt, to avoid any unintended usage, ...) and its restauration.