4 ms·

How I backup my servers

29 points by NicoJuicy 3y ago

NoPicklez 3y ago
Over my years of reviewing companies backup & disaster recovery capabilities, there is one key part of all of this which is how you can recover the backups if your stack fails or part thereof. Have you tested your ability to recover these backups at all levels? e.g. OS, DBMS and data. If so, it would be worth including that in your article. Appreciating that not everyone has the same use case or the same risk appetite but it is important.
- NicoJuicy 3y ago
  Perhaps it wasn't clear enough. But I've mentioned most of this in my post: > By default, I already use the backup option on my VPS that is configurable per Cloud provider ( in my case, my Windows Servers run on Vultr and Linux servers on Hetzner). So I won’t go into detail on this. Which basically handles backup and restore from systems within the cloud provider I'm using. My data and db's are stored 'externally'. And I restore them regularely (locally) to test things ( eg. before I deploy to prod and/or repro a bug). So I test my backups ~every 2 weeks since I've added them in my dev workflow. > Being able to reproduce a production issue in dev under 2 minutes from any device that I own, has been a time-saver An extra precaution would be to simple sync the BackBlaze bucket to my local NAS, but that doesn't seem to be required atm. Tldr: yes, I test my backups since I use them in my dev workflow.
  - remram 3y ago
    Hetzner seems to allow 7 snapshots total [1]. That means you can keep 1 week of daily backups, or less than two months of weekly backups. Either way, it's useful for the "server burned down" situation, but not really for the "can't find this file", "I want my previous config", or "this file got corrupted somehow" situations. Obligatory Tao of Backup: http://taobackup.com/history.html http://taobackup.com/history.html Also 20% of your server's total cost seems pricey. [1]: https://docs.hetzner.com/cloud/servers/getting-started/enabling-backups/ https://docs.hetzner.com/cloud/servers/getting-started/enabl...
    - NicoJuicy 3y ago
      20% of a 7,74€/m. vps ( currently) is not a big deal. As far as I'm aware, 20% seems to be the default ( reference: digital ocean). It's very rare that I would need a server backup older than 7 days in practice. Because I backup my data a month. It could be better if it remained 1 backup per month though ( eg. In Sunday)
      - remram 3y ago
        It's very rare that you'll need a backup at all, when you do something bad and unknown has happened, and you can't be sure that it happened within 7 days of when you noticed. 7 days is also within the period of time people can be on vacation. It's way too short.
        NicoJuicy 3y ago
        I mostly use backup and restore functionally outside of office hours as a precaution with a limited amount of time. Take a backup, migrate, test and restore, if required. For the rare cases that I needed it, it's been worth it.
PeterStuer 3y ago
Bit of an adjacent issue, but why is backup of a typical home/family network still not perfected in 2023? What is needed (of the top of my head): - runs on a nas, so it can coordinate the backups of the different machines - nice browser based gui, with adequate security. - can accomodate storage extensions for when space runs low - backs up arbitrarily large files. - uses VSS on Windows for snapshot consistency and to back up (permanently) open files - uses global deduplication to save up significant space - can do disk image as well as arbitrary file/folder selection backups - offers advanced retention policies with good out of the box defaults - allows both server and user side throtteling and pauses - offers bandwith limiter - offers archiving - auto client agent updates - bootable usb smart restore to changed hardware - no 'call us for getting a price quotation' - no complicated extras license needs for PC/Server/VM/DB/... , just a simple one site family non commercial use price. Synology active backup for business, which I use now, goes there about 70% of the way. Any suggestions for alternatives?
- 8fingerlouie 3y ago
  > runs on a nas, so it can coordinate the backups of the different machines Do we really assume that the "typical home/family" is using a NAS ? I think it far more common that they use "the cloud" like OneDrive, Google Drive, Apple iCloud, etc. That is not a bad thing (privacy issues aside) as data resilience is much much better in the cloud, than almost any NAS you can setup at home. A paid OneDrive subscription gives you dual datacenter redundance (erasure coding) with unlimited versions for 30 days of all files, copy on write, and much more. And regarding privacy, that is also not an issue that the "typical" user is concerned about. All the big vendors pretty much swear to never look at your data _unless you share it_, and for the "nerds", things like Cryptomator exists. In my book, and one i use myself, a modern setup will have the cloud as primary storage, and "all you need" is a local box that can synchronize cloud data locally with versions, or make a versioned backup locally (the sync is useless if upstream gets corrupted). In theory a "real time" sync of data and scheduled BTRFS/ZFS/APFS snapshots every N minutes would be a versioned backup. On top of that you then add a cloud backup of the synchronized data to a different cloud provider, and you have 3-2-1 backup in place. This will, for most users, be cheaper than buying and running a NAS, as the average family probably has less than 2TB data, and comes with less risk of data loss due to hardware failure. The biggest threat to your data in the cloud is loss of access, which you can mitigate with the local mirror and the remote backup. You can get Microsoft Family 365 for around €65/year (Microsoft HUP program and other deals), which includes 6 x 1TB OneDrive storage. Add to that a Backblaze Backup subscription at €65/yeaer and you have both primary cloud storage and cloud backup at €130/year. Where i live, just powering a 4 bay NAS for a year costs around €200. > Synology active backup for business, which I use now, goes there about 70% of the way. Any suggestions for alternatives? I personally use Arq for my client backups, but it does not support global deduplication, only per marchine. For global deduplication, you might want to look at Restic (open source, stable), Duplicacy (commercial license) or Kopia (open source, beta state)
twooclock 3y ago
Nice setup, congrats! The part of verifying backups via dev resonates with me. Only I'm doing it manualy from time to time. Do you mind sharing your scrips? I'd love to "upgrade" my backups for my mssql.
- NicoJuicy 3y ago
  I don't mind, but I'm currently on weekend with friends ( still travelling ) and I'll only be on my laptop when it's required. I'll check next week and reply here.
  - twooclock 3y ago
    Thank you! Will wait...
ozim 3y ago
For anyone being afraid of ransomware keep in mind you want to pull backups instead of writing them. So backup server has rights only to read your prod server and prod server should not have write rights on backup server or at least “create only” if other option is too much hassle.
- viraptor 3y ago
  Letting anything access your production servers from outside is not amazing. There are multiple ways to do an append-only backup instead. Borg has a special mode for that, with the docs: https://borgbackup.readthedocs.io/en/stable/usage/notes.html#append-only-mode-forbid-compaction https://borgbackup.readthedocs.io/en/stable/usage/notes.html...
  - remram 3y ago
    No one said anything about "letting *anything* access"... use IP whitelists and keypairs as needed. The feature you linked will only work with smart backends (e.g. VM running borg), so if I'm using S3 etc to store my backups I can't use that. That's a big caveat given the pricing difference between object storage (e.g. S3) and block storage (where I can run `borg serve --append-only`). edit: Actually it seems borg doesn't support object storage at all.
    - viraptor 3y ago
      Letting anything, as in letting any one whitelisted service is still not great. The moment you do pull, you're accepting commands from outside instead of deciding locally what gets sent. This makes the scope for possible vulnerabilities larger. Borg was just one example. And you can still run it effectively append-only by mounting the s3 endpoint locally and setting the right IAM permissions on the bucket.
Borg3 3y ago
Backups arent that hard really. A lot of interesting tools. Imo handling day to day docs/data/mutables is harder. Hence im happy to use simple DVFS repo manager. No need to do backups of those :) Stuff is stored usualy on desktop, laptop and server.

29 points by NicoJuicy 3y ago

Over my years of reviewing companies backup & disaster recovery capabilities, there is one key part of all of this which is how you can recover the backups if your stack fails or part thereof. Have you tested your ability to recover these backups at all levels? e.g. OS, DBMS and data. If so, it would be worth including that in your article. Appreciating that not everyone has the same use case or the same risk appetite but it is important.

NicoJuicy 3y ago
Perhaps it wasn't clear enough. But I've mentioned most of this in my post: > By default, I already use the backup option on my VPS that is configurable per Cloud provider ( in my case, my Windows Servers run on Vultr and Linux servers on Hetzner). So I won’t go into detail on this. Which basically handles backup and restore from systems within the cloud provider I'm using. My data and db's are stored 'externally'. And I restore them regularely (locally) to test things ( eg. before I deploy to prod and/or repro a bug). So I test my backups ~every 2 weeks since I've added them in my dev workflow. > Being able to reproduce a production issue in dev under 2 minutes from any device that I own, has been a time-saver An extra precaution would be to simple sync the BackBlaze bucket to my local NAS, but that doesn't seem to be required atm. Tldr: yes, I test my backups since I use them in my dev workflow.
- remram 3y ago
  Hetzner seems to allow 7 snapshots total [1]. That means you can keep 1 week of daily backups, or less than two months of weekly backups. Either way, it's useful for the "server burned down" situation, but not really for the "can't find this file", "I want my previous config", or "this file got corrupted somehow" situations. Obligatory Tao of Backup: http://taobackup.com/history.html http://taobackup.com/history.html Also 20% of your server's total cost seems pricey. [1]: https://docs.hetzner.com/cloud/servers/getting-started/enabling-backups/ https://docs.hetzner.com/cloud/servers/getting-started/enabl...
  - NicoJuicy 3y ago
    20% of a 7,74€/m. vps ( currently) is not a big deal. As far as I'm aware, 20% seems to be the default ( reference: digital ocean). It's very rare that I would need a server backup older than 7 days in practice. Because I backup my data a month. It could be better if it remained 1 backup per month though ( eg. In Sunday)
    - remram 3y ago
      It's very rare that you'll need a backup at all, when you do something bad and unknown has happened, and you can't be sure that it happened within 7 days of when you noticed. 7 days is also within the period of time people can be on vacation. It's way too short.
      - NicoJuicy 3y ago
        I mostly use backup and restore functionally outside of office hours as a precaution with a limited amount of time. Take a backup, migrate, test and restore, if required. For the rare cases that I needed it, it's been worth it.

PeterStuer 3y ago

Bit of an adjacent issue, but why is backup of a typical home/family network still not perfected in 2023? What is needed (of the top of my head): - runs on a nas, so it can coordinate the backups of the different machines - nice browser based gui, with adequate security. - can accomodate storage extensions for when space runs low - backs up arbitrarily large files. - uses VSS on Windows for snapshot consistency and to back up (permanently) open files - uses global deduplication to save up significant space - can do disk image as well as arbitrary file/folder selection backups - offers advanced retention policies with good out of the box defaults - allows both server and user side throtteling and pauses - offers bandwith limiter - offers archiving - auto client agent updates - bootable usb smart restore to changed hardware - no 'call us for getting a price quotation' - no complicated extras license needs for PC/Server/VM/DB/... , just a simple one site family non commercial use price. Synology active backup for business, which I use now, goes there about 70% of the way. Any suggestions for alternatives?

8fingerlouie 3y ago
> runs on a nas, so it can coordinate the backups of the different machines Do we really assume that the "typical home/family" is using a NAS ? I think it far more common that they use "the cloud" like OneDrive, Google Drive, Apple iCloud, etc. That is not a bad thing (privacy issues aside) as data resilience is much much better in the cloud, than almost any NAS you can setup at home. A paid OneDrive subscription gives you dual datacenter redundance (erasure coding) with unlimited versions for 30 days of all files, copy on write, and much more. And regarding privacy, that is also not an issue that the "typical" user is concerned about. All the big vendors pretty much swear to never look at your data _unless you share it_, and for the "nerds", things like Cryptomator exists. In my book, and one i use myself, a modern setup will have the cloud as primary storage, and "all you need" is a local box that can synchronize cloud data locally with versions, or make a versioned backup locally (the sync is useless if upstream gets corrupted). In theory a "real time" sync of data and scheduled BTRFS/ZFS/APFS snapshots every N minutes would be a versioned backup. On top of that you then add a cloud backup of the synchronized data to a different cloud provider, and you have 3-2-1 backup in place. This will, for most users, be cheaper than buying and running a NAS, as the average family probably has less than 2TB data, and comes with less risk of data loss due to hardware failure. The biggest threat to your data in the cloud is loss of access, which you can mitigate with the local mirror and the remote backup. You can get Microsoft Family 365 for around €65/year (Microsoft HUP program and other deals), which includes 6 x 1TB OneDrive storage. Add to that a Backblaze Backup subscription at €65/yeaer and you have both primary cloud storage and cloud backup at €130/year. Where i live, just powering a 4 bay NAS for a year costs around €200. > Synology active backup for business, which I use now, goes there about 70% of the way. Any suggestions for alternatives? I personally use Arq for my client backups, but it does not support global deduplication, only per marchine. For global deduplication, you might want to look at Restic (open source, stable), Duplicacy (commercial license) or Kopia (open source, beta state)

twooclock 3y ago

Nice setup, congrats! The part of verifying backups via dev resonates with me. Only I'm doing it manualy from time to time. Do you mind sharing your scrips? I'd love to "upgrade" my backups for my mssql.

NicoJuicy 3y ago
I don't mind, but I'm currently on weekend with friends ( still travelling ) and I'll only be on my laptop when it's required. I'll check next week and reply here.
- twooclock 3y ago
  Thank you! Will wait...

ozim 3y ago

For anyone being afraid of ransomware keep in mind you want to pull backups instead of writing them. So backup server has rights only to read your prod server and prod server should not have write rights on backup server or at least “create only” if other option is too much hassle.

viraptor 3y ago
Letting anything access your production servers from outside is not amazing. There are multiple ways to do an append-only backup instead. Borg has a special mode for that, with the docs: https://borgbackup.readthedocs.io/en/stable/usage/notes.html#append-only-mode-forbid-compaction https://borgbackup.readthedocs.io/en/stable/usage/notes.html...
- remram 3y ago
  No one said anything about "letting *anything* access"... use IP whitelists and keypairs as needed. The feature you linked will only work with smart backends (e.g. VM running borg), so if I'm using S3 etc to store my backups I can't use that. That's a big caveat given the pricing difference between object storage (e.g. S3) and block storage (where I can run `borg serve --append-only`). edit: Actually it seems borg doesn't support object storage at all.
  - viraptor 3y ago
    Letting anything, as in letting any one whitelisted service is still not great. The moment you do pull, you're accepting commands from outside instead of deciding locally what gets sent. This makes the scope for possible vulnerabilities larger. Borg was just one example. And you can still run it effectively append-only by mounting the s3 endpoint locally and setting the right IAM permissions on the bucket.

Borg3 3y ago

Backups arent that hard really. A lot of interesting tools. Imo handling day to day docs/data/mutables is harder. Hence im happy to use simple DVFS repo manager. No need to do backups of those :) Stuff is stored usualy on desktop, laptop and server.