7 ms·

Backups Aren't Simple

350 points by afilipovski 17d ago

readthenotes1 16d ago
"Obviously, none of this is worth anything if you don’t actually test restores." One of my teachers claimed that contrary to common behavior, no one actually valued backups; it was restores that were worth paying for.
- TheCondor 16d ago
  It is a special sensation when you go to your hand rolled backup "solution" to make some simple recoveries and start finding problems. It is a complex mix of not dread but disgust, sick to your stomach, self-anger, and loss. I can only image what that's like with like all of your children's baby pictures being the lost material. Borg and restic are fantastic, it still seems like some sort of dummy proof mode might be desirable, all the features in the world and they have some some reasonably sane policy or something that includes verifications. On the flip side of that though, really and truly, I've restored a laptop from a timemachine backup and it came back damn nearly identical and that was magical.
  - innocent_name 16d ago
    >Borg ... fantastic I needed to temporarily move off 20TB from my NAS and bought a Hetzner storage box. I wanted to encrypt the data before sending it so i went with the borg ssh mount. Long story short, borg failed a couple of times due to https://github.com/borgbackup/borg/issues/7672 https://github.com/borgbackup/borg/issues/7672, i resumed it per documentation and got 2gb of silently corrupted data. There was no way to do data integrity checks due to broken pipes and there was no option to resume the job from the previous point either. I couldn't afford to go with the s3, backblaze wasn't an option due to placing trust into their proprietary client doing the encryption and i got majorly screwed in the end.
    - Dylan16807 16d ago
      That issue happens when the repo is corrupted? That's not the cause of the failure then. Also you must have been unaware of Backblaze B2? It doesn't have a proprietary client, and it launched 11 years ago only a couple months after attic was forked to create borg.
- gerdesj 16d ago
  Try running a business on that basis - where do the restores come from? 8) We charge our customers for backup storage by volume and that includes an annual demonstration of recovery. We fire up some or all of their systems on our gear in isolation and show that they are reasonably functional. Your teacher's comment is riffing on the well trod lines of: "heights don't kill people, it's depths that kill people", which is all about perspective. This is is not related to "guns don't kill people ..." which is about agency.
  - bitwize 16d ago
    "It's not the fall that kills you, it's the sudden stop at the end."
  - OkayPhysicist 16d ago
    Running a business on that basis would look like the white(ish)-hat version of ransomware. "Oh, your data is gone? How much is it really worth to you?"
  - al_borland 16d ago
    It is true that very few people proactively care about backups. It’s like flossing. People know they’re supposed to do it, but in the moment it feels like a waste of time and energy, when there are more important things to do. When data is lost, or decay takes over, the value can skyrocket. Depending on what is lost, someone may be willing to pay 100x when they would have spent on backup in order to get their data back, this is why data recovery services can basically charge whatever they want. My assumption is the Venn diagram between people who pass the marshmallow test, and those who proactively backup their systems, has a lot of overlap.
PunchyHamster 16d ago
So is apparently making readable site. Software like https://restic.net/ https://restic.net/ does a good job. Few choices of backend Accepts stuff from pipe too so you can just pipe mysqldump or pg_dumpall without intermediate files Very decent options for checking repo integrity, personally (well, at work too, we use it on few hundred user machines and servers) I also added "roll a dice for backup and try to restore it" test script to make sure it is working Decent deduplication too so "store year's worth of weekly snapshot" is very sensible strategy once you exclude the "constantly changing and useless" (caches etc.) files out of it can mount FUSE directory with all backups on sensible OSes. https://kopia.io/ https://kopia.io/ does the same +GUI but a bit worse on CLI front (it insists on keeping local config which makes scripting a bit more involved, but not much)
- bt1a 16d ago
  importantly, does your die have a side that corresponds to a multi-continental cataclysmic natural disaster of magnitude only known unknown to pre-recorded-history
- LilBytes 16d ago
  I'm using Restic & Backrest (https://garethgeorge.github.io/backrest/ https://garethgeorge.github.io/backrest/) everywhere I can, they're both exceptional OSS. IMHO, the easiest way to perform a restore test is to use your production backups to refresh a lesser environment on a weekly cadence. Naturally this has to be appropriate to the type of data you're restoring, e.g., our E2E (end to end) testing environment has been built to the same risk tolerances of production so it's suitable for production data and the PII (Personally Identifiable Information) it hosts. If this isn't in your own risk tolerances, you can still perform the same test but ensure the data from prod is anonymised or alternatively destroyed, and perform a second restore after production restore with the dataset that preceeded it that has no PII data or similar. With this the E2E tests confirm the backups are expected and you can tick a box that when shit does eventually hit the fan, data corruption isn't a problem. Normal caveats apply, you must make sure your backups are immutable so they cannot be modified or changed _after_ the tests have been ran.
  - 16d ago
bloomingeek 16d ago
"encrypted, chunk-level deduplicated, GFS-rotated, point-in-time archived, cloud, 3-2-1 backup solution" is now my newest password, no commas. (Don't tell anyone!)
- rileymat2 16d ago
  Change the o in rotated to a 0 and you might have a secure password.
- b3lvedere 16d ago
  Person entering coffee shop: "So what is the WiFI password here?" "You sure would to like to know that eh you effing hipster" "Does that start with a capital Y?" "Yes. No spaces."
  - voiper1 16d ago
    Rocketjump! The wife password is four words all uppercase. https://www.youtube.com/watch?v=bLE7zsJk4AI https://www.youtube.com/watch?v=bLE7zsJk4AI
- ChrisRR 16d ago
  Really? It's showing up as ***** for me
  - GTP 16d ago
    I think it's Hunter2
pizzafeelsright 16d ago
I don't want backups. I want synced duplicated data in geographically separated by shared credentials.
- offmycloud 16d ago
  > I don't want backups. I want synced duplicated data in geographically separated by shared credentials. If you accidentally delete something, how long does it take for the delete to propagate to all of your geographically separated datacenters? You need some sort of point-in-time snapshots to be able to recover from accidents and silent corruption.
  - bob1029 16d ago
    > You need some sort of point-in-time snapshots to be able to recover from accidents and silent corruption. https://docs.aws.amazon.com/AmazonS3/latest/userguide/versioning-workflows.html https://docs.aws.amazon.com/AmazonS3/latest/userguide/versio...
  - tim333 16d ago
    Dropbox is quite good like that - you can get back deleted or changed stuff up to 30 days old.
- layer8 16d ago
  Sounds like you are the second type of people mentioned in the article.
- keithnz 16d ago
  that's not really the same thing, what you are describing is redundancy so failure in one (or more) place doesn't bring the system down. Backups are about preventing data loss. Data loss can happen in many different ways, not just a system crashing and burning. It can be deliberately/accidently done such that it would be sync'd everywhere.
innocent_name 16d ago
>Motivated by our success in developing this solution, we try to use it to backup the homelab with its 10 Docker containers. But later we find out from logs on the individual machines that backups are failing. The reason being that many Docker containers like to create root-owned files, and if you’re not careful you can create a cronjob running as the default user. and a privesc also.
- cvhc 16d ago
  Yeah it feels it can be risky to "make the backup also dump the databases, and give it full filesystem permissions on our Docker volumes", depending on what binaries get the root privileges. What I do is having a minimal systemd timer under root that only calls restic for backing up files (with some additional systemd seucurity restrictions). App-dependent backup logics, like dumping databases, are done by user/container-level cronjobs separately.
- afilipovski 16d ago
  The ideal setup is rootless Podman and having all your containers not even be able to create root-owned files. It would also be great if services did their own database dumps. Immich does this and it's such a relief to just be able to copy over its volume for backup purposes. Unfortunately we don't live in an ideal world and so much of the software both doesn't dump it's own databases, and is incapable of running in a rootless container. If you weigh the probability of a privesc from your backup with full access to the volumes, against the probability that the software you run has a CVE, which one is worse? This is one of those things I consciously chose to ignore in my setup. Maybe it will come back to bite me in the future, sure. Or maybe I will learn a better approach that pushes my frontier of convenience-security forward.
gerdesj 16d ago
One of our customers decided to go for off site backups with us - they had been a customer for 15 odd years for other services. Backups are not our core business, its one of the services and I got around to setting them up sooner rather than later. Two weeks later they had a fire in one of their huge printing machines which caused quite a lot of direct damage due to heat and rendered a lot of equipment broken due to smoke/fumes. It was also, as it turns out, a week before the first COVID 19 lockdown in the UK. Their backups landed on a XFS file system with reflinks enabled. I cloned their repo and fired up NFS with their VMs running on our gear. It took me another hour to spin up another OpenVPN server (CA etc) for them to use and sort out a few other details (coffee doesn't brew itself). They ran via VPN out of our data centre for most of the pandemic. That was an utter triumph but I also have some rather less triumphant stories about backups and lack thereof. Let's skip over those 8)
senectus1 16d ago
Amazon are finding this out atm :-P https://archive.is/Ay7RJ https://archive.is/Ay7RJ
publlus_enigma 16d ago
There are four times in my life I have suffered regrettable data loss incidents. The first was when the telephone pole outside our house was struck directly by lightning. Not only was it the loudest thing I have ever heard, the current surged through the telephone line, into the internal fax modem, and fries everything within its vicinity. I was 10. I did have backuos, but only only floppy and they didn't cover everything. The second was storing data in OneDrive - a change to their terms surrounding "lifetime" unlikely noted storage, combined with a client that was unusably slow to download and a deadline for data retrieval meant that I lost most of my files. The third was SD card failure in digital camera on holiday, the controller chip died catastrophically, leaving the card completely unrecognised. It was a brand new Sony 128GB card, manufactured by Toshiba, and it seemed to be a common issue. I now shoot to two cards simultaneously. And the fourth time was ... Performing a backup. An errant script deleted the source content, but I'd also deleted the existing backup to free up space for the new backup. I've been weary of using rewritable media for some time now as a consequence, but I think backups themselves are high risk activities.
- Forgeties79 16d ago
  >I now shoot 2 cards simultaneously A very lovely change over the last 8 years or so lol. I came up in film during the DSLR revolution. 5D2’s/7D’s/Rebels (i series) years. 8bit 420 nasty aliasing recording on single cards and praying baby. Magic lantern booted on those same SD’s!
  - jeffbee 16d ago
    Mainstream DSLRs with dual card slots came out 22 years ago.
    - mrheosuper 16d ago
      Sadly after 22 years, it's still considered a "high-end" feature. Even on $1600 Fujifilm x100vi, there is still single SD slot.
      - 16d ago
ebrahimh 16d ago
I’m setting up 3-2-1-ish backups for my infra of 3 hosts, and definitely leaning towards Restic + Backrest. All my hosts run the same CoreOS setup (https://github.com/ebrahim37/infra-template https://github.com/ebrahim37/infra-template), where container volumes are placed in one central volumes/ folder and that is the only thing I have to backup. I plan to implement it like this: vps1: - restic container with custom sh entrypoint that will backup volumes/ to homelab every 24 hours homelab: - backrest container, to back up volumes/, do prune/check, replicate repo to offsite - rest-server container, will store backups from vps1, homelab, offsite offsite: - restic container, backs up volumes/ to homelab every 24 hours - rest-server container, store copy of backups from homelab Only caveat is backing up databases, will either have to do: stop container, backup volume/database-data, start container; or use pg dump etc. The deduplication is nice, you can have a snapshot for each week of the past year without crazy storage cost
- codys 16d ago
  If your system has a way to take consistent snapshots in the filesystem (btrfs, zfs) or volume manager (lvm2, perhaps with a bit of filesystem support to obtain fs consistency), that can be used to avoid database downtime, if desired. https://www.postgresql.org/docs/current/backup-file.html https://www.postgresql.org/docs/current/backup-file.html
- ray_v 16d ago
  [flagged]
- zenoprax 16d ago
  I've been trying to find a solution for this too! I was considering using Rclone but too many things are using SQLite for me to trust rsync. I was also going to go with CoreOS but I'm leaning towards Fedora Cloud now in case I need to manage things a bit more (and "auto updating" is not something I want as that suggests auto rebooting). Your secrets.yaml makes me nervous though - too easy to miss a key and leave something exposed. Why not just add the whole file to the vault?
  - ebrahimh 16d ago
Devin3162 16d ago
[dead]
AdieuToLogic 16d ago
A friend of mine used to work at Veritas[0] making enterprise data retention solutions. When I spoke about their product as being "making backups", he corrected me by saying: We are not in the backup business. We are in the restoration business. 0 - https://en.wikipedia.org/wiki/Backup_Exec https://en.wikipedia.org/wiki/Backup_Exec
- voiceofunreason 16d ago
  Backups are boring. Restores, on the other hand, are often exciting.
  - BLKNSLVR 16d ago
    Backups can be automated. Restores are a manual, heightened-attention, deep-focus process.
  - wallst07 16d ago
    If Seinfeld told a backup joke: "You know how to take the backup, you just don't know how to restore the backup. And that's really the most important part of the backup, the restoring. Anybody can just copy a file!"
  - GarnetFloride 16d ago
    Restores can be Very exciting. I worked tech support and got trained on an email server backup program. I was alone for my first call on the program. Before I could finish the greeting I knew it was going to be a wild call as I could hear in the background someone shouting and someone sobbing, the guy actually on the phone sounded like he wasn't planning on making it home that night. Turned out they were testing out some sync software and after a test they deleted the sync without turning off the app which then did its thing and now nobody could log in except admin who was the only user left. I was able to find and confirm that the backups were in place and accessible. Which was good, I could hear them breathing again. But I had no way to get the restore going because the source server was so hosed. Just then the dev walking by and I got him on the call and he did some outright hacker magic to get things to the point that a restore could function. I thought about writing my notes into a troubleshooting guide but then realized it would be better to skip the hacking and make a setup guide that showed them how to set it up properly in the first place.
snizovtsev 16d ago
I'm polishing my backup system right now, and found that `tar | zstd | gpg` plus a small vibe-coded wrapper is pretty much all I need. GNU tar has its own incremental index via `--listed-incremental=FILE`. Unlike Borg or Restic, which have their own more complicated repository formats, this leaves me with just one additional file (the `.snar`) alongside a dumb, portable full-disk tar archive. The nice part is that, unlike repositories that require both read/write access patterns, tar can compute deltas using only the small `.snar` file, while the main `full-disk.tar` can remain buried in write-only Glacier storage. This makes it a much better fit for Glacier's write-once model and 180-day minimum retention. My current plan is to upload a full-disk tar to an external HDD + Glacier on a weekly cadence, and the `.snar`-based deltas daily. That gives me a pretty simple cloud backup solution for a few bucks a month (after burning through $100+ of free credits).
- nine_k 16d ago
  How well did it work for restoring? For me, the ability to mount a backup and look at a particular file proved to be important. Also, I restored from my backups four times; two of them was moving between machines, pretty quickly.
  - snizovtsev 16d ago
    [dead]
- cortesoft 16d ago
  The problem with using glacier for backups is it's hard to run restore exercises, and without restore exercises, a backup is pretty dangerous
  - snizovtsev 16d ago
    You can exercise it against regular, non-Glacier S3. Same flow; the only difference is that you skip one attribute on upload and one request/wait step on download.
  - pmw 16d ago
chasil 16d ago
For my Oracle Linux systems, I rely on the vendor ReaR RPM package for the baseline backup. This is similar to the "Ignite" utility that I used in my HP-UX days. https://relax-and-recover.org/ https://relax-and-recover.org/ This is two-fold useful, first as it enables boot from backup media for total recovery, second in that it creates a backup.tar.gz that holds everything that was not explitictly ignored in the /etc/rear/local.conf file. I'm running Oracle databases on these systems (and the local.conf is configured to ignore the datafile directories). I have standby databases (not Dataguard in that I don't have online redolog replication) that allow me to recover them to the primary in a disaster, with some loss of committed transactions. I finally have rsync configured for some scratch temporary files. I have dallied with btrfs snapshot replication for home directories (in a loopback mount). My vendor support (via a CSI number) has been on and off, so I don't use btrfs in areas that we need it most. My replacements do not like the complexity (and I am retiring).
darvo31 16d ago
Yep, backup is a verb, restore is a noun. Testing restores is where the real work begins.
smallerfish 16d ago
On my home PC I use restic, and (shockingly) I run it manually (via a script). I have a terminal banner that nags me when backup hasn't run in 24 hours, or displays spanish verb tables when it has (so I will notice if the banner doesn't show). I've been burned in the past losing a backup cron, and dead man's snitch seemed overkill for my needs here. And I've also forgotten what backup system was in place on a prior install, and it took me way longer than I liked figuring out how it was setup when I needed to do a restore. Obviously for production systems automation & monitoring is the way. I kind of like this setup on my own machine though.
- qiuguoli2 16d ago
  [flagged]
mrheosuper 16d ago
As an average-techie person, the only piece of data that i truly practice 3-2-1 is photos and videos. 1 on my phone, 1 on my small server running Immich, and 1 is on Google Photo(sadly, the quality is loss, but i can live with that). Most of other data i only have 2 backup, usually 1 at my local machine(code) and 1 online(github). Yeah it's not ideal, but another question to ask yourself, is it truly worth it. I trust the engineers at MS doing much better job at backing up their data than me.
- dsego 16d ago
  I tried to set up something similar, photos and videos on local NAS + cloud storage. Also with data loss, eg. I don't sync any large RAW camera files to cloud. I clone my local machine with CCC onto an external drive, that way it should be easy to just restore the last state onto a new machine. So I think it's archival storage for media vs snapshots of your working setup so you can restore quickly.
bakinator 16d ago
On the topic of backups, 7zip recently changed its handling of the FILE_ATTRIBUTE_ARCHIVE bit on unzipped files which caused an issue with some old software I was using. This is from the old DOS days with the BACKUP/RESTORE commands - the bit was set if any backup was made of the file. It's comically naive because it only assumes a single backup and this bit was forgotten to the sands of time until it was disturbed by some meddling kids!
m463 16d ago
I've given out jwz's link to several people: https://www.jwz.org/doc/backups.html https://www.jwz.org/doc/backups.html (maybe copy/paste this into browser instead of following link, since referrer from hn apparently does something)
- NetOpWibby 16d ago
  Oh that's fantastic (HN redirect)
- everybodyknows 16d ago
  > have this happen every morning at 5AM by creating a temporary text file containing this line: 0 5 * * * rsync -vax --delete --ignore-errors / /Volumes/Backup/ This leaves you needlessly exposed to this failure mode: Commit some "braino" that wipes out some recent work. Go to bed without noticing the loss. 5am rsync run wipes your backup as well. The fix is to back up to not just one but to a rotation of images. Use --link-dest to reduce storage overhead to the size of the directory hierarchy only.
- cranberryjoe 15d ago
  Clicks on the link…yeah that’s accurate
justsomehnguy 16d ago
> So our first principle is to have a backup, i.e. a copy of your files someplace else. So far so good. So who are we? A household with a localhost administrator?[0] Then you are surely doesn't need encrypted, chunk-level corporate level bla-bla-bla. You need a Syncthing copy to some other device not at home - and an additional backup procedure to maintain the history and protect against PEBKAC errors - which can run on your local Syncthing copy or/and the other one. And no, nobody needs your 555GB of RAWs of sunsets/flowers/precious_family_moments you shot - not even you.[1] [0] well considering the tone of TFA and most of the comments here [1] that's Instagram/Google Photos/iCloud for nowadays, totally with "Remember this day N years ago?"
- afilipovski 16d ago
  > So who are we? A household with a localhost administrator? In my case, yes. I've gotten into the trap of building a homelab that over time became pretty important, more like a small prod environment. We take for granted the "stability" that cloud services provide (until the terms of service change and you are screwed over by either loss of privacy, loss of access, etc). Self-hosting these things yourself exposes you to this complexity. Photos aren't that important, how about your passwords and documents and the whole setup around all this? I also do a part of this just for the love of the game.
crossroadsguy 16d ago
If you don't control your data size, it will end up controlling you and your backup choices, which will eventually lead to many avoidable, disastrous outcomes. Somehow, this gets missed in data storage and backup planning.
- branon 16d ago
  Agreed - information always expands to fill the available space, and placing a cap on your available space is an underrated strategy
jongjong 16d ago
When it comes to software, it's almost never simple. What people think of as simple is rarely so. For example: Requirement: "Just copy a file from folder A to folder B." Minimal translation: "Implement the capability to copy any file, in any format and of any size from SharePoint, located at a configurable path, with appropriate authentication and access control checks, then stream it in chunks to a different, configurable path inside an S3 bucket, also with appropriate authentication and access controls in place. Ensure that any disruption in either service which may occur while the file is streaming can be recovered from at the point it failed instead of having to restart from the beginning. Ensure that the retry mechanism is built-in and that the retry window is configurable; if the file cannot be copied within the specific time window, then an error should be sent via email to a configured email address. Ensure that the entire transfer is encrypted in transit... If the file happens to be a folder, then you must copy across all of its contents recursively up to a certain configured MAX_DEPTH to avoid DoS and ensuring that the system does not get caught in an infinite loop due to symlinks pointing to a parent folder... In this case, send an email to the configured address... Etc... Etc..." And the thing is; if you tell AI "Just copy a file from folder A to folder B." - It will not meet your 'basic' needs because even if it does a great job at filling the gaps in your requirements, it will still take shortcuts. In order for an AI to avoid taking shortcuts, it would have to make you fill out a questionnaire and make you sign up for and configure services; it would not be a pleasant user experience. The user experience cannot be pleasant, because the AI cannot read your mind and it cannot know your intent.
- TeMPOraL 16d ago
  Here's a thing: my intent and requirements actually are: "Just copy a file from folder A to folder B." All that access control and authentication and symlink nonsense? That's a self-inflicted problem that exists only in enterprise, and shouldn't be assumed - much less created - for regular users.
  - jongjong 15d ago
    This is your intent until I finish implementing and then you complain that it doesn't also work with folders or large files and it doesn't work with any file system. Or you will complain if your colleague accesses your file without your permission. Reality of software development is that the user would expect every single one of those things but they simply don't understand what they want up front. There are three different aspects to consider in my experience; what the user says they want, what the user thinks they want and what the user actually wants. The three can have substantial differences.
dsego 16d ago
Something is better than nothing. I accidentally spilled liquid on my macbook just the other day. It's still in the shop. Luckily, I periodically clone my drive with CCC onto a single external SSD. Don't wait to set up the perfect backup solution, just have a copy of your data in more than one place, that's a good start imho.
ryuuseijin 16d ago
A bit of a tangent, but the linked rsnapshot tool is about 7k lines of perl code. The COPYING file is 20 years old and it's still maintained, the last change was 6 months ago. I'm feeling really old. [1]: https://github.com/rsnapshot/rsnapshot/blob/master/rsnapshot-program.pl https://github.com/rsnapshot/rsnapshot/blob/master/rsnapshot...
- geye1234 16d ago
  Yes, rsnapshot is excellent, I'm glad the article mentioned it. I use it for my weekly backups. So wonderfully simple. I wish the config file weren't quite so strict about syntax, but I only need to change it about twice a decade so no big deal.
wolfi1 16d ago
concerning backups, when it comes to disaster recovery you need a strategy for restoring as well, sometimes it's not only the data you would need but the systems as well
bnegreve 16d ago
I have a cron script that rsync my entire home directory every day with a remote server. Since rsync doesn't remove remote files, I re-create a backup from scratch the first day of each month and store a copy of the previous one. Using old standardized open source tools means it survives system updates without having to fix anything for years. I got my laptop stolen/lost 3 times over the past 15 years, and I have always been able to restore everything the next day on a new laptop, in the time it takes to transfer the files over the network. I haven't given much thought, I'm sure there is a realistic scenario where this strategy would fails but I haven't found it yet.
- graemep 16d ago
  rsync can delete remote files with --delete or --delete-after What is missing is recovering accidentally deleted or corrupted data.
  - bnegreve 16d ago
    > rsync can delete remote files with --delete or --delete-after Yes but I don't use it precisely because that lets me recover files that I have accidentally deleted. I assume it's the intended default behavior of rsync.
    - graemep 16d ago
      What if that happens after your monthly creation of a new backup.
      - bnegreve 16d ago
        I keep the previous backup for one month and then delete it. So eventually the file is lost yes, but this is inevitable if you assume finite amount of storage.
        graemep 16d ago
        In that case I think its a good system. Its very similar to an incremental backup in what it achieves.
nirmeetimthebes 16d ago
[dead]
QuiEgo 16d ago
Time Machine (to NAS) + iCloud (w/ ADP) + local copy is the lowest friction path I have found.
- nmacias 16d ago
  I had an synology NAS power blip which caused a long term time machine to become completely corrupt, like, i’m not confident its quite so compatible with non apple disk formats
KPGv2 16d ago
Truenas, Crashplan gives me a 3-2-1 with rolling, incremental backups at a pretty low price
Helmut10001 16d ago
I really like ZFS snapshots with offsite pull-mode sync using Jim Salter's sanoid/syncoid [1]. ZFS is the base for all OS/filesystems on top of it. If you have a good system for organizing ZFS datasets, and separating ephemeral from persistent data (e.g. [2]), then this is 90% of the backup requirements already fullfilled. [1]: https://github.com/jimsalterjrs/sanoid [2]: https://du.nkel.dev/blog/2026-05-16_rootless_docker_virtiofs_proxmox/
- aborsy 16d ago
  Agree!
- hovering_nox 16d ago
  I like to use ZFS snapshots of running systems and then create and upload the backup from them via the excellent Borgmatic. (I know you can't depend on snapshots for databases and similar, but it's worked until now™.)
- zenoprax 16d ago
  Thanks for the second link - looks interesting. I have an old and loud 16-bay server that boots every 7 days or so if no one is home, `syncoid there here`, and shuts off. I'll get pinged by uptimerobot once in a while if it's overdue and I'll get a notification if a pool is unhealthy or reaching capacity (`sanoid --health` I think). Otherwise I forget I even have it set up. Automated cold backups are great peace-of-mind.
  - Helmut10001 16d ago
    Exactly! My offsite Backup box boots using a Shelly Plug S set on timer; it doesn't need to unlock the enrypted ZFS datasets because I use zfs sync snapshots in raw mode (-R). After the zfs scrupt, the box sends an email to me and shuts down. Next week: Repeat.
- Aldipower 16d ago
  Using this in production for 5 years now. We have a replicated MongoDB and one replica lives on a ZFS doing snapshot backups via syncoid. Love it.
overtone1000 16d ago
+1 for restic. ZFS snapshots + restic backups to backblaze for my homeserver. My secret sauce is a healtchecks.io instance that blows up my phone if ZFS scrubs throw any errors, when local snapshots fail, or when restic checks or backups fail.
janci 16d ago
My setup: rsync to Synology NAS, snapshotting there, sync to remote Synology NAS over VPN. (The other NAS is at my parent's and they have similar setup syncing to me) The best part: it is just files, not some proprietary archive. I have high confidence in this. I can restore with the most rudimentary tools. This setup has worked for me well on linux, windows and now mac. I migrated to new computer&os by restoring the backup mostly.
Gud 16d ago
I use ZFS and do off site backups using rsync.net.
999900000999 16d ago
They are simple. The issue is companies love to fire QA people who should be the ones testing these. Microsoft says we don’t need no SDETs, our products can ship full of issues. What are the hostages, I mean customers going to do about it? I have no sympathy for any major corp who suffers data loss. Now individuals who lost grandmas wedding photos, that sucks. But my recovery strategy for that is simple. After scanning I emailed it out to one cousin who emailed them out to a few more. Plus I have a cloud backup ( I have no idea where the the original files are). Data is meant to be shared after all
l1am0 16d ago
Huge fan of borg + vorta for backups and time Maschine like „Jump back in time“. Wrote about it years ago: https://simon-frey.com/blog/borgvorta-is-finally-a-usable-backup-solution-for-linux-desktop/ https://simon-frey.com/blog/borgvorta-is-finally-a-usable-ba...
flippingheck 16d ago
> There are two types of people: those who have suffered a catastrophic loss of data, and those who will. This is an empirical claim, but is it grounded in reality? 99% of people don't backup, and that's probably the right choice because the risk is low, they can't meaningfully improve their restoration rate themselves and they don't care enough about their data to classify its loss as catastrophic. When a person hears someone talk about the importance of backups, but also haven't heard friends/family suffer this date, they will rightfully ignore this warning. Or do we all follow the best practices as it relates to backups, exercise, sleep, nutrition, accounting, house maintenance, ... Etc. I do not. Backups aren't near the top of that list. This stuff just isn't very important to most people and that's okay.
- grumbelbart2 16d ago
  > 99% of people don't backup This depends on your viewpoint. Using gmail for example is an implicit backup of your e-mails. Uploading your pictures to Facebook the same. The move from local, private compute and storage to the cloud solved this issue for many to a degree that it is now less important.
- ygouzerh 16d ago
  I think this is more a quote for engineers
- afilipovski 16d ago
  Let's say SSDs have an annualized failure rate of 1%. (The real figures are from 0.5% for more reliable ones, 1.5% for less reliable ones). The annualized failure rate for HDDs is a bit higher, but let's give it 1% anyway. Over 40 years (even with changing disks, upgrading PCs, etc.), the chances of you ending up with your data intact is 0.99^40=0.67. May not seem too bad, but this is just a hardware failure of your main drive we're talking about. When you factor in all the other types of screwups that happen, the annualized failure rate will skyrocket. So a more conservative version of this quote is that a 20-year old today is very likely to experience data loss over his lifetime. Thankfully with cloud being ubiquitous at least photos are given one copy in the cloud, but IMO even that is risky. People have different appetites for risk, and that's OK. But over the last 20 years the all of our data has become pretty important. I don't know how many people could stomach the loss of important documents, photos, etc.
jpthewall 16d ago
[dead]
bibelo 16d ago
> as long as you run restores every 6 months uh no, in a company, as a sysadmin, you try your backup files at least every month
BetterThanSober 16d ago
Backups isn't real until it's restored don't be like me folks
mamonoleechi 16d ago
3 rules of backup (don't remember where i read this) - do several backups (not using the same hardware brand or host if you do it online) - check your backup (if something was wrong, you have another backup from the first rule) - do not store your backups at the same place (if your place get flooded or burn, it will be pretty handy)
mattbillenstein 16d ago
Ran duplicity - it's really a blunt hammer, lots of duplication, rotation, etc. Then switched to Restic - so much better - highly recommend this.
ethanprk 16d ago
[flagged]
dirkc 16d ago
> “There are two types of people: those who have suffered a catastrophic loss of data, and those who will.” When I was a teenager, I was the reason for data loss for my dad, twice. Both times it was because I was re-partitioning a hard drive to install linux. You would think that taught me a lesson about backups, instead it reminds me every now and again to be grateful for an awesome dad and aspire to handle situations with my kid similarly :)
- mesrik 16d ago
  >> “There are two types of people: those who have suffered a catastrophic loss of data, and those who will.” Also "There are just two types of equipment; those which are already broken and those which will be broken". I've found a good way to point out anyone thinking right direction not to trust their valuable data in any single entity, product, location, etc. When I heard it decades ago IIRC 80's Nokia Data Unix courses I went. The distinction made Backups is just 1/3 of the triad Redundancy, Backups, Archives. 1) Redundancy is what you get using RAID, multiple network connectivity etc. ie. good for avoiding single point of failures and avoiding loss service or product availability. 2) Backups are for recovering lost data needed returning lost running known state right before fault or some time state before what backup schedule and used rotation cycle can provide. 3) Archives are meant to saving valuable data not expected to change or active use and but there is need saving much longer periods. Not seeing and understanding usefulness of difference between these seem hard some people I met over decades. Which then led to confusion, not knowing what they are up to with their data and also often later disappointment when they found out that for example backup programs are not great if you expected archiving instead.
- wallst07 16d ago
  Reminds me of a story: I was sitting at a restaurant waiting for my food. I casually notice two young children in the next booth playing with a phone, but pay little attention. Moments later their dad (presumably) comes back and asks for his phone back and sits down with their food. The kids are eating and the dad is fumbling with his phone with odd looks and I can see his frustration growing. Then out I hear something like "What did you do to my phone?" repeated loudly over and over. "You erased my phone, it's gone." The small children (guessing) had figured out how to factory reset his phone. My thought at the time was something like : How many times did I do something as a toddler I don't remember to piss of my dad like that and that he never got mad or brought it up later (Lots of times knowing myself).
abejora 16d ago
I agree with this sentiment! For our Abejora timesheet SaaS, setting up backups was one of the harder parts. We had to change direction a few times while implementing it. We finally got a nice setup with CloudNativePG + Barman. This allows for point-in-time restores, but there were a lot of lessons to learn along the way. - The various types of (database) backups (logical, binary, onsite, offsite, snapshots, write-ahead log...) in combination with the various types of data (database, files, cluster configuration...) - In our earlier approaches, we tried to preserve the old database volume if it was not corrupt, and use that in our restore. This caused so many complications, because you are fighting the recommended approach. So now, when we need to restore, we always restore from backups and the 'live volume' is dropped. - For a restore, we just spin up a completely new Kubernetes cluster, instead of trying to restore in-cluster. This is a lot easier. - Many object stores allow for retention periods, which you can put to good use to prevent malicious or accidental removal of backups. HOWEVER, not all of them are really 'locked'. In some services, you can still override the lock with a forced delete; in others, you can still remove the project holding the storage buckets, which will delete the buckets, and so on... so test those things, instead of just blindly depending on a 'retention period' claim. - We now automatically run a scheduled restore with verifications on a weekly basis. This requirement does shape your environment, so keep that in mind! There is also the question of how you can reliably and automatically verify that the restore restored the latest data (of a live prod environment). Various solutions exist here, but most are not very elegant! Honestly, this is only worth it if you are already handling sufficient volume. If you are just starting out, then the easier approach is to just go with a hosted database, which will handle backups and point-in-time restores for you.
cuvert 16d ago
For my small library photos, keepass passwords, etc around 30gb, I use syncthing, which will backup on my pc, on my parents pc (they live in another country), on my sisters pc and on my phone. They also use this triangular mode to backup their stuff. Syncthing is setup to only sync through tailscale or home routers network.
- Oxodao 16d ago
  sync is not backup Also why you sync through LAN only?
  - cl3misch 16d ago
    It's tailscale or local LAN. syncthing allows you to specify multiple addresses for the same peer > tailscale or home routers network
  - cuvert 16d ago
    I was in a vacation and I couldn't connect to a remote pc. And since then I set up wireguard in all routers and I also changed syncthing LAN only.
    - Oxodao 16d ago
      Ok, i do use it through their relay but i should do the same
- cm2187 16d ago
  So you delete a file by mistake and it replicates within seconds? Not exactly a backup. And a malware will do that to all of your files.
  - cuvert 16d ago
    Yes, but I setup syncthing's file versioning to 25, in case of keepass, for example. Also, I set it up that only my PC can send, all other PCs, smartphones, can only receive. Sure, there are still many ways to attack this system, but I still prefer it.
OroPla 16d ago
Bit rot was mentioned in the article, but then no solution was offered. I also haven't found a good solution, unfortunately. If you only have a couple of Terabytes you could create check sums of everything and then verify regularly, but I feel this approach doesn't scale very well if you have lots of data.
- GTP 16d ago
  If you have lots of data, I would suggest using a filesystem designed for this, like ZFS or BTRFS. You would still have to spend in storage though, as ultimately protecting from bit rot requires redundancy.
  - OroPla 16d ago
    I do use btrfs, but I don't really understand how to use it to protect against bit rot. I tried to look into it at some point, but got overwhelmed. Did invest in two backup copies, one of them off site, though. However, looking at prices, I can see one of those backups just being dismantled to be used as more storage.
    - suncore 16d ago
      Zfs and btrfs will avoid corruption caused below the file system layer, not something like malware or misuse from a tired sysadmin. I run a tool called integrit that creates checksums and lets me detect non-wanted changes. I only run it on important stuff that does not change too much like photos and videos. Not fun to have to regularly check the log but it works fairly well. This could maybe be extended with automatic checking of the logs.
    - GTP 16d ago
      If you got overwhelmed in trying to understand how to do it, just do it the easy way: create a ZFS pool of two drives of identical size in mirroring. Then, make sure to have regular scrubs (this means ZFS will regularly check all files for corruption, and recover from eventual errors), use your OS' task scheduler (e.g. Systemd or cron) to run this once a month. This should be sufficient to protect you from bit rot.
b3lvedere 16d ago
"Also make sure not to run backups at 2AM or 3AM, or things may get scary." I never though of daylight savings times causing such an issue. Learned something today.
PeterStuer 16d ago
The only backup system that I truly love and trust (from experience) is Proxmox Backup Server.
amelius 16d ago
I'm adding this to my list of things that should be simple but aren't. It already contains "Sending a 1GB file to a friend without seeing advertisements" and "Making a mixtape for my friends who are on a different streaming service".
- Biganon 15d ago
  Making mixtapes for others is an act of love that must never disappear
robschmidt90 16d ago
I also recently improved my backup approach. I am using daily restic snapshots for my off-site backup and it works pretty well. I wrote about this here [1]. [1] https://staticnotes.org/posts/cloud-backups-with-restic/ https://staticnotes.org/posts/cloud-backups-with-restic/
crabbone 16d ago
Some complexity levels not mentioned in the article: * Consistency groups. The article kinda almost gets there, but stops right before it realizes there's a problem. If you have multiple programs accessing the storage, s.a. a database (that you are snapshotting) and the application using the database, then you are risking an inconsistent snapshot when the database state you wrote is ahead or behind the application state. You need a mechanism to freeze all I/O beside the snapshot in order to ensure that the snapshot data is useful. * ACLs. And similar external mapping to the storage being used, eg. what if the users come from an LDAP server? Similar problem with symlinks coming from/going to ouitside the snapshot. * Sparse file support. Let's say you have a few VM images in your snapshot: if your tool doesn't support sparse files, it will likely greatly inflate the image sizes. Bottom line, either use the product's own snapshot functionality (ZFS has those), or switch to the product that does.
- afilipovski 16d ago
  Thanks for the input. These things never crossed my mind, even the first one which in hindsight should have been obvious to me. I will update the article to include these.
cranberryjoe 16d ago
Yes, yes they are simple. Btrfs and zfs have incredible snapshot and replicate
Kinrany 16d ago
There should be a standard for backups, one that makes it possible for a (potentially distributed) system to talk to a backup system and to save or restore a snapshot.
Hive80Lab 16d ago
[flagged]
cranberryjoe 15d ago
Hey kids It’s simple 1) separate data from applications 2) use a real filesystem like zfs or btrfs 3) cronjob snapshot, replicate onsite, replicate offsite 4) monitor size of volumes and snapshots, delete them as your policy 5) periodic zfs and btrfs scrub End of fucking story. No need for all that tricky shit y’all are doing.
siddhuk 15d ago
[flagged]
jllyhill 15d ago
Where can I find any discussions on how to actually test the restoration process? How can I verify that I really restored my 10k photos without losing anything?

350 points by afilipovski 17d ago

"Obviously, none of this is worth anything if you don’t actually test restores." One of my teachers claimed that contrary to common behavior, no one actually valued backups; it was restores that were worth paying for.

TheCondor 16d ago
It is a special sensation when you go to your hand rolled backup "solution" to make some simple recoveries and start finding problems. It is a complex mix of not dread but disgust, sick to your stomach, self-anger, and loss. I can only image what that's like with like all of your children's baby pictures being the lost material. Borg and restic are fantastic, it still seems like some sort of dummy proof mode might be desirable, all the features in the world and they have some some reasonably sane policy or something that includes verifications. On the flip side of that though, really and truly, I've restored a laptop from a timemachine backup and it came back damn nearly identical and that was magical.
- innocent_name 16d ago
  >Borg ... fantastic I needed to temporarily move off 20TB from my NAS and bought a Hetzner storage box. I wanted to encrypt the data before sending it so i went with the borg ssh mount. Long story short, borg failed a couple of times due to https://github.com/borgbackup/borg/issues/7672 https://github.com/borgbackup/borg/issues/7672, i resumed it per documentation and got 2gb of silently corrupted data. There was no way to do data integrity checks due to broken pipes and there was no option to resume the job from the previous point either. I couldn't afford to go with the s3, backblaze wasn't an option due to placing trust into their proprietary client doing the encryption and i got majorly screwed in the end.
  - Dylan16807 16d ago
    That issue happens when the repo is corrupted? That's not the cause of the failure then. Also you must have been unaware of Backblaze B2? It doesn't have a proprietary client, and it launched 11 years ago only a couple months after attic was forked to create borg.
gerdesj 16d ago
Try running a business on that basis - where do the restores come from? 8) We charge our customers for backup storage by volume and that includes an annual demonstration of recovery. We fire up some or all of their systems on our gear in isolation and show that they are reasonably functional. Your teacher's comment is riffing on the well trod lines of: "heights don't kill people, it's depths that kill people", which is all about perspective. This is is not related to "guns don't kill people ..." which is about agency.
- bitwize 16d ago
  "It's not the fall that kills you, it's the sudden stop at the end."
- OkayPhysicist 16d ago
  Running a business on that basis would look like the white(ish)-hat version of ransomware. "Oh, your data is gone? How much is it really worth to you?"
- al_borland 16d ago
  It is true that very few people proactively care about backups. It’s like flossing. People know they’re supposed to do it, but in the moment it feels like a waste of time and energy, when there are more important things to do. When data is lost, or decay takes over, the value can skyrocket. Depending on what is lost, someone may be willing to pay 100x when they would have spent on backup in order to get their data back, this is why data recovery services can basically charge whatever they want. My assumption is the Venn diagram between people who pass the marshmallow test, and those who proactively backup their systems, has a lot of overlap.

PunchyHamster 16d ago

So is apparently making readable site. Software like https://restic.net/ https://restic.net/ does a good job. Few choices of backend Accepts stuff from pipe too so you can just pipe mysqldump or pg_dumpall without intermediate files Very decent options for checking repo integrity, personally (well, at work too, we use it on few hundred user machines and servers) I also added "roll a dice for backup and try to restore it" test script to make sure it is working Decent deduplication too so "store year's worth of weekly snapshot" is very sensible strategy once you exclude the "constantly changing and useless" (caches etc.) files out of it can mount FUSE directory with all backups on sensible OSes. https://kopia.io/ https://kopia.io/ does the same +GUI but a bit worse on CLI front (it insists on keeping local config which makes scripting a bit more involved, but not much)

bt1a 16d ago
importantly, does your die have a side that corresponds to a multi-continental cataclysmic natural disaster of magnitude only known unknown to pre-recorded-history
LilBytes 16d ago
I'm using Restic & Backrest (https://garethgeorge.github.io/backrest/ https://garethgeorge.github.io/backrest/) everywhere I can, they're both exceptional OSS. IMHO, the easiest way to perform a restore test is to use your production backups to refresh a lesser environment on a weekly cadence. Naturally this has to be appropriate to the type of data you're restoring, e.g., our E2E (end to end) testing environment has been built to the same risk tolerances of production so it's suitable for production data and the PII (Personally Identifiable Information) it hosts. If this isn't in your own risk tolerances, you can still perform the same test but ensure the data from prod is anonymised or alternatively destroyed, and perform a second restore after production restore with the dataset that preceeded it that has no PII data or similar. With this the E2E tests confirm the backups are expected and you can tick a box that when shit does eventually hit the fan, data corruption isn't a problem. Normal caveats apply, you must make sure your backups are immutable so they cannot be modified or changed _after_ the tests have been ran.
- 16d ago

bloomingeek 16d ago

"encrypted, chunk-level deduplicated, GFS-rotated, point-in-time archived, cloud, 3-2-1 backup solution" is now my newest password, no commas. (Don't tell anyone!)

rileymat2 16d ago
Change the o in rotated to a 0 and you might have a secure password.
b3lvedere 16d ago
Person entering coffee shop: "So what is the WiFI password here?" "You sure would to like to know that eh you effing hipster" "Does that start with a capital Y?" "Yes. No spaces."
- voiper1 16d ago
  Rocketjump! The wife password is four words all uppercase. https://www.youtube.com/watch?v=bLE7zsJk4AI https://www.youtube.com/watch?v=bLE7zsJk4AI
ChrisRR 16d ago
Really? It's showing up as ***** for me
- GTP 16d ago
  I think it's Hunter2

pizzafeelsright 16d ago

I don't want backups. I want synced duplicated data in geographically separated by shared credentials.

offmycloud 16d ago
> I don't want backups. I want synced duplicated data in geographically separated by shared credentials. If you accidentally delete something, how long does it take for the delete to propagate to all of your geographically separated datacenters? You need some sort of point-in-time snapshots to be able to recover from accidents and silent corruption.
- bob1029 16d ago
  > You need some sort of point-in-time snapshots to be able to recover from accidents and silent corruption. https://docs.aws.amazon.com/AmazonS3/latest/userguide/versioning-workflows.html https://docs.aws.amazon.com/AmazonS3/latest/userguide/versio...
- tim333 16d ago
  Dropbox is quite good like that - you can get back deleted or changed stuff up to 30 days old.
layer8 16d ago
Sounds like you are the second type of people mentioned in the article.
keithnz 16d ago
that's not really the same thing, what you are describing is redundancy so failure in one (or more) place doesn't bring the system down. Backups are about preventing data loss. Data loss can happen in many different ways, not just a system crashing and burning. It can be deliberately/accidently done such that it would be sync'd everywhere.

innocent_name 16d ago

>Motivated by our success in developing this solution, we try to use it to backup the homelab with its 10 Docker containers. But later we find out from logs on the individual machines that backups are failing. The reason being that many Docker containers like to create root-owned files, and if you’re not careful you can create a cronjob running as the default user. and a privesc also.

cvhc 16d ago
Yeah it feels it can be risky to "make the backup also dump the databases, and give it full filesystem permissions on our Docker volumes", depending on what binaries get the root privileges. What I do is having a minimal systemd timer under root that only calls restic for backing up files (with some additional systemd seucurity restrictions). App-dependent backup logics, like dumping databases, are done by user/container-level cronjobs separately.
afilipovski 16d ago
The ideal setup is rootless Podman and having all your containers not even be able to create root-owned files. It would also be great if services did their own database dumps. Immich does this and it's such a relief to just be able to copy over its volume for backup purposes. Unfortunately we don't live in an ideal world and so much of the software both doesn't dump it's own databases, and is incapable of running in a rootless container. If you weigh the probability of a privesc from your backup with full access to the volumes, against the probability that the software you run has a CVE, which one is worse? This is one of those things I consciously chose to ignore in my setup. Maybe it will come back to bite me in the future, sure. Or maybe I will learn a better approach that pushes my frontier of convenience-security forward.

gerdesj 16d ago

One of our customers decided to go for off site backups with us - they had been a customer for 15 odd years for other services. Backups are not our core business, its one of the services and I got around to setting them up sooner rather than later. Two weeks later they had a fire in one of their huge printing machines which caused quite a lot of direct damage due to heat and rendered a lot of equipment broken due to smoke/fumes. It was also, as it turns out, a week before the first COVID 19 lockdown in the UK. Their backups landed on a XFS file system with reflinks enabled. I cloned their repo and fired up NFS with their VMs running on our gear. It took me another hour to spin up another OpenVPN server (CA etc) for them to use and sort out a few other details (coffee doesn't brew itself). They ran via VPN out of our data centre for most of the pandemic. That was an utter triumph but I also have some rather less triumphant stories about backups and lack thereof. Let's skip over those 8)

senectus1 16d ago

Amazon are finding this out atm :-P https://archive.is/Ay7RJ https://archive.is/Ay7RJ

publlus_enigma 16d ago

There are four times in my life I have suffered regrettable data loss incidents. The first was when the telephone pole outside our house was struck directly by lightning. Not only was it the loudest thing I have ever heard, the current surged through the telephone line, into the internal fax modem, and fries everything within its vicinity. I was 10. I did have backuos, but only only floppy and they didn't cover everything. The second was storing data in OneDrive - a change to their terms surrounding "lifetime" unlikely noted storage, combined with a client that was unusably slow to download and a deadline for data retrieval meant that I lost most of my files. The third was SD card failure in digital camera on holiday, the controller chip died catastrophically, leaving the card completely unrecognised. It was a brand new Sony 128GB card, manufactured by Toshiba, and it seemed to be a common issue. I now shoot to two cards simultaneously. And the fourth time was ... Performing a backup. An errant script deleted the source content, but I'd also deleted the existing backup to free up space for the new backup. I've been weary of using rewritable media for some time now as a consequence, but I think backups themselves are high risk activities.

Forgeties79 16d ago
>I now shoot 2 cards simultaneously A very lovely change over the last 8 years or so lol. I came up in film during the DSLR revolution. 5D2’s/7D’s/Rebels (i series) years. 8bit 420 nasty aliasing recording on single cards and praying baby. Magic lantern booted on those same SD’s!
- jeffbee 16d ago
  Mainstream DSLRs with dual card slots came out 22 years ago.
  - mrheosuper 16d ago
    Sadly after 22 years, it's still considered a "high-end" feature. Even on $1600 Fujifilm x100vi, there is still single SD slot.
    - 16d ago

ebrahimh 16d ago

I’m setting up 3-2-1-ish backups for my infra of 3 hosts, and definitely leaning towards Restic + Backrest. All my hosts run the same CoreOS setup (https://github.com/ebrahim37/infra-template https://github.com/ebrahim37/infra-template), where container volumes are placed in one central volumes/ folder and that is the only thing I have to backup. I plan to implement it like this: vps1: - restic container with custom sh entrypoint that will backup volumes/ to homelab every 24 hours homelab: - backrest container, to back up volumes/, do prune/check, replicate repo to offsite - rest-server container, will store backups from vps1, homelab, offsite offsite: - restic container, backs up volumes/ to homelab every 24 hours - rest-server container, store copy of backups from homelab Only caveat is backing up databases, will either have to do: stop container, backup volume/database-data, start container; or use pg dump etc. The deduplication is nice, you can have a snapshot for each week of the past year without crazy storage cost

codys 16d ago
If your system has a way to take consistent snapshots in the filesystem (btrfs, zfs) or volume manager (lvm2, perhaps with a bit of filesystem support to obtain fs consistency), that can be used to avoid database downtime, if desired. https://www.postgresql.org/docs/current/backup-file.html https://www.postgresql.org/docs/current/backup-file.html
ray_v 16d ago
[flagged]
zenoprax 16d ago
I've been trying to find a solution for this too! I was considering using Rclone but too many things are using SQLite for me to trust rsync. I was also going to go with CoreOS but I'm leaning towards Fedora Cloud now in case I need to manage things a bit more (and "auto updating" is not something I want as that suggests auto rebooting). Your secrets.yaml makes me nervous though - too easy to miss a key and leave something exposed. Why not just add the whole file to the vault?
- ebrahimh 16d ago

Devin3162 16d ago

[dead]

AdieuToLogic 16d ago

A friend of mine used to work at Veritas[0] making enterprise data retention solutions. When I spoke about their product as being "making backups", he corrected me by saying: We are not in the backup business. We are in the restoration business. 0 - https://en.wikipedia.org/wiki/Backup_Exec https://en.wikipedia.org/wiki/Backup_Exec

voiceofunreason 16d ago
Backups are boring. Restores, on the other hand, are often exciting.
- BLKNSLVR 16d ago
  Backups can be automated. Restores are a manual, heightened-attention, deep-focus process.
- wallst07 16d ago
  If Seinfeld told a backup joke: "You know how to take the backup, you just don't know how to restore the backup. And that's really the most important part of the backup, the restoring. Anybody can just copy a file!"
- GarnetFloride 16d ago
  Restores can be Very exciting. I worked tech support and got trained on an email server backup program. I was alone for my first call on the program. Before I could finish the greeting I knew it was going to be a wild call as I could hear in the background someone shouting and someone sobbing, the guy actually on the phone sounded like he wasn't planning on making it home that night. Turned out they were testing out some sync software and after a test they deleted the sync without turning off the app which then did its thing and now nobody could log in except admin who was the only user left. I was able to find and confirm that the backups were in place and accessible. Which was good, I could hear them breathing again. But I had no way to get the restore going because the source server was so hosed. Just then the dev walking by and I got him on the call and he did some outright hacker magic to get things to the point that a restore could function. I thought about writing my notes into a troubleshooting guide but then realized it would be better to skip the hacking and make a setup guide that showed them how to set it up properly in the first place.

snizovtsev 16d ago

I'm polishing my backup system right now, and found that `tar | zstd | gpg` plus a small vibe-coded wrapper is pretty much all I need. GNU tar has its own incremental index via `--listed-incremental=FILE`. Unlike Borg or Restic, which have their own more complicated repository formats, this leaves me with just one additional file (the `.snar`) alongside a dumb, portable full-disk tar archive. The nice part is that, unlike repositories that require both read/write access patterns, tar can compute deltas using only the small `.snar` file, while the main `full-disk.tar` can remain buried in write-only Glacier storage. This makes it a much better fit for Glacier's write-once model and 180-day minimum retention. My current plan is to upload a full-disk tar to an external HDD + Glacier on a weekly cadence, and the `.snar`-based deltas daily. That gives me a pretty simple cloud backup solution for a few bucks a month (after burning through $100+ of free credits).

nine_k 16d ago
How well did it work for restoring? For me, the ability to mount a backup and look at a particular file proved to be important. Also, I restored from my backups four times; two of them was moving between machines, pretty quickly.
- snizovtsev 16d ago
  [dead]
cortesoft 16d ago
The problem with using glacier for backups is it's hard to run restore exercises, and without restore exercises, a backup is pretty dangerous
- snizovtsev 16d ago
  You can exercise it against regular, non-Glacier S3. Same flow; the only difference is that you skip one attribute on upload and one request/wait step on download.
- pmw 16d ago

chasil 16d ago

For my Oracle Linux systems, I rely on the vendor ReaR RPM package for the baseline backup. This is similar to the "Ignite" utility that I used in my HP-UX days. https://relax-and-recover.org/ https://relax-and-recover.org/ This is two-fold useful, first as it enables boot from backup media for total recovery, second in that it creates a backup.tar.gz that holds everything that was not explitictly ignored in the /etc/rear/local.conf file. I'm running Oracle databases on these systems (and the local.conf is configured to ignore the datafile directories). I have standby databases (not Dataguard in that I don't have online redolog replication) that allow me to recover them to the primary in a disaster, with some loss of committed transactions. I finally have rsync configured for some scratch temporary files. I have dallied with btrfs snapshot replication for home directories (in a loopback mount). My vendor support (via a CSI number) has been on and off, so I don't use btrfs in areas that we need it most. My replacements do not like the complexity (and I am retiring).

darvo31 16d ago

Yep, backup is a verb, restore is a noun. Testing restores is where the real work begins.

smallerfish 16d ago

On my home PC I use restic, and (shockingly) I run it manually (via a script). I have a terminal banner that nags me when backup hasn't run in 24 hours, or displays spanish verb tables when it has (so I will notice if the banner doesn't show). I've been burned in the past losing a backup cron, and dead man's snitch seemed overkill for my needs here. And I've also forgotten what backup system was in place on a prior install, and it took me way longer than I liked figuring out how it was setup when I needed to do a restore. Obviously for production systems automation & monitoring is the way. I kind of like this setup on my own machine though.

qiuguoli2 16d ago
[flagged]

mrheosuper 16d ago

As an average-techie person, the only piece of data that i truly practice 3-2-1 is photos and videos. 1 on my phone, 1 on my small server running Immich, and 1 is on Google Photo(sadly, the quality is loss, but i can live with that). Most of other data i only have 2 backup, usually 1 at my local machine(code) and 1 online(github). Yeah it's not ideal, but another question to ask yourself, is it truly worth it. I trust the engineers at MS doing much better job at backing up their data than me.

dsego 16d ago
I tried to set up something similar, photos and videos on local NAS + cloud storage. Also with data loss, eg. I don't sync any large RAW camera files to cloud. I clone my local machine with CCC onto an external drive, that way it should be easy to just restore the last state onto a new machine. So I think it's archival storage for media vs snapshots of your working setup so you can restore quickly.

bakinator 16d ago

On the topic of backups, 7zip recently changed its handling of the FILE_ATTRIBUTE_ARCHIVE bit on unzipped files which caused an issue with some old software I was using. This is from the old DOS days with the BACKUP/RESTORE commands - the bit was set if any backup was made of the file. It's comically naive because it only assumes a single backup and this bit was forgotten to the sands of time until it was disturbed by some meddling kids!

m463 16d ago

I've given out jwz's link to several people: https://www.jwz.org/doc/backups.html https://www.jwz.org/doc/backups.html (maybe copy/paste this into browser instead of following link, since referrer from hn apparently does something)

NetOpWibby 16d ago
Oh that's fantastic (HN redirect)
everybodyknows 16d ago
> have this happen every morning at 5AM by creating a temporary text file containing this line: 0 5 * * * rsync -vax --delete --ignore-errors / /Volumes/Backup/ This leaves you needlessly exposed to this failure mode: Commit some "braino" that wipes out some recent work. Go to bed without noticing the loss. 5am rsync run wipes your backup as well. The fix is to back up to not just one but to a rotation of images. Use --link-dest to reduce storage overhead to the size of the directory hierarchy only.
cranberryjoe 15d ago
Clicks on the link…yeah that’s accurate

justsomehnguy 16d ago

> So our first principle is to have a backup, i.e. a copy of your files someplace else. So far so good. So who are we? A household with a localhost administrator?[0] Then you are surely doesn't need encrypted, chunk-level corporate level bla-bla-bla. You need a Syncthing copy to some other device not at home - and an additional backup procedure to maintain the history and protect against PEBKAC errors - which can run on your local Syncthing copy or/and the other one. And no, nobody needs your 555GB of RAWs of sunsets/flowers/precious_family_moments you shot - not even you.[1] [0] well considering the tone of TFA and most of the comments here [1] that's Instagram/Google Photos/iCloud for nowadays, totally with "Remember this day N years ago?"

afilipovski 16d ago
> So who are we? A household with a localhost administrator? In my case, yes. I've gotten into the trap of building a homelab that over time became pretty important, more like a small prod environment. We take for granted the "stability" that cloud services provide (until the terms of service change and you are screwed over by either loss of privacy, loss of access, etc). Self-hosting these things yourself exposes you to this complexity. Photos aren't that important, how about your passwords and documents and the whole setup around all this? I also do a part of this just for the love of the game.

crossroadsguy 16d ago

If you don't control your data size, it will end up controlling you and your backup choices, which will eventually lead to many avoidable, disastrous outcomes. Somehow, this gets missed in data storage and backup planning.

branon 16d ago
Agreed - information always expands to fill the available space, and placing a cap on your available space is an underrated strategy

jongjong 16d ago

When it comes to software, it's almost never simple. What people think of as simple is rarely so. For example: Requirement: "Just copy a file from folder A to folder B." Minimal translation: "Implement the capability to copy any file, in any format and of any size from SharePoint, located at a configurable path, with appropriate authentication and access control checks, then stream it in chunks to a different, configurable path inside an S3 bucket, also with appropriate authentication and access controls in place. Ensure that any disruption in either service which may occur while the file is streaming can be recovered from at the point it failed instead of having to restart from the beginning. Ensure that the retry mechanism is built-in and that the retry window is configurable; if the file cannot be copied within the specific time window, then an error should be sent via email to a configured email address. Ensure that the entire transfer is encrypted in transit... If the file happens to be a folder, then you must copy across all of its contents recursively up to a certain configured MAX_DEPTH to avoid DoS and ensuring that the system does not get caught in an infinite loop due to symlinks pointing to a parent folder... In this case, send an email to the configured address... Etc... Etc..." And the thing is; if you tell AI "Just copy a file from folder A to folder B." - It will not meet your 'basic' needs because even if it does a great job at filling the gaps in your requirements, it will still take shortcuts. In order for an AI to avoid taking shortcuts, it would have to make you fill out a questionnaire and make you sign up for and configure services; it would not be a pleasant user experience. The user experience cannot be pleasant, because the AI cannot read your mind and it cannot know your intent.

TeMPOraL 16d ago
Here's a thing: my intent and requirements actually are: "Just copy a file from folder A to folder B." All that access control and authentication and symlink nonsense? That's a self-inflicted problem that exists only in enterprise, and shouldn't be assumed - much less created - for regular users.
- jongjong 15d ago
  This is your intent until I finish implementing and then you complain that it doesn't also work with folders or large files and it doesn't work with any file system. Or you will complain if your colleague accesses your file without your permission. Reality of software development is that the user would expect every single one of those things but they simply don't understand what they want up front. There are three different aspects to consider in my experience; what the user says they want, what the user thinks they want and what the user actually wants. The three can have substantial differences.

dsego 16d ago

Something is better than nothing. I accidentally spilled liquid on my macbook just the other day. It's still in the shop. Luckily, I periodically clone my drive with CCC onto a single external SSD. Don't wait to set up the perfect backup solution, just have a copy of your data in more than one place, that's a good start imho.

ryuuseijin 16d ago

A bit of a tangent, but the linked rsnapshot tool is about 7k lines of perl code. The COPYING file is 20 years old and it's still maintained, the last change was 6 months ago. I'm feeling really old. [1]: https://github.com/rsnapshot/rsnapshot/blob/master/rsnapshot-program.pl https://github.com/rsnapshot/rsnapshot/blob/master/rsnapshot...

geye1234 16d ago
Yes, rsnapshot is excellent, I'm glad the article mentioned it. I use it for my weekly backups. So wonderfully simple. I wish the config file weren't quite so strict about syntax, but I only need to change it about twice a decade so no big deal.

wolfi1 16d ago

concerning backups, when it comes to disaster recovery you need a strategy for restoring as well, sometimes it's not only the data you would need but the systems as well

bnegreve 16d ago

I have a cron script that rsync my entire home directory every day with a remote server. Since rsync doesn't remove remote files, I re-create a backup from scratch the first day of each month and store a copy of the previous one. Using old standardized open source tools means it survives system updates without having to fix anything for years. I got my laptop stolen/lost 3 times over the past 15 years, and I have always been able to restore everything the next day on a new laptop, in the time it takes to transfer the files over the network. I haven't given much thought, I'm sure there is a realistic scenario where this strategy would fails but I haven't found it yet.

graemep 16d ago
rsync can delete remote files with --delete or --delete-after What is missing is recovering accidentally deleted or corrupted data.
- bnegreve 16d ago
  > rsync can delete remote files with --delete or --delete-after Yes but I don't use it precisely because that lets me recover files that I have accidentally deleted. I assume it's the intended default behavior of rsync.
  - graemep 16d ago
    What if that happens after your monthly creation of a new backup.
    - bnegreve 16d ago
      I keep the previous backup for one month and then delete it. So eventually the file is lost yes, but this is inevitable if you assume finite amount of storage.
      - graemep 16d ago
        In that case I think its a good system. Its very similar to an incremental backup in what it achieves.

nirmeetimthebes 16d ago

[dead]

QuiEgo 16d ago

Time Machine (to NAS) + iCloud (w/ ADP) + local copy is the lowest friction path I have found.

nmacias 16d ago
I had an synology NAS power blip which caused a long term time machine to become completely corrupt, like, i’m not confident its quite so compatible with non apple disk formats

KPGv2 16d ago

Truenas, Crashplan gives me a 3-2-1 with rolling, incremental backups at a pretty low price

Helmut10001 16d ago

I really like ZFS snapshots with offsite pull-mode sync using Jim Salter's sanoid/syncoid [1]. ZFS is the base for all OS/filesystems on top of it. If you have a good system for organizing ZFS datasets, and separating ephemeral from persistent data (e.g. [2]), then this is 90% of the backup requirements already fullfilled. [1]: https://github.com/jimsalterjrs/sanoid [2]: https://du.nkel.dev/blog/2026-05-16_rootless_docker_virtiofs_proxmox/

aborsy 16d ago
Agree!
hovering_nox 16d ago
I like to use ZFS snapshots of running systems and then create and upload the backup from them via the excellent Borgmatic. (I know you can't depend on snapshots for databases and similar, but it's worked until now™.)
zenoprax 16d ago
Thanks for the second link - looks interesting. I have an old and loud 16-bay server that boots every 7 days or so if no one is home, `syncoid there here`, and shuts off. I'll get pinged by uptimerobot once in a while if it's overdue and I'll get a notification if a pool is unhealthy or reaching capacity (`sanoid --health` I think). Otherwise I forget I even have it set up. Automated cold backups are great peace-of-mind.
- Helmut10001 16d ago
  Exactly! My offsite Backup box boots using a Shelly Plug S set on timer; it doesn't need to unlock the enrypted ZFS datasets because I use zfs sync snapshots in raw mode (-R). After the zfs scrupt, the box sends an email to me and shuts down. Next week: Repeat.
Aldipower 16d ago
Using this in production for 5 years now. We have a replicated MongoDB and one replica lives on a ZFS doing snapshot backups via syncoid. Love it.

overtone1000 16d ago

+1 for restic. ZFS snapshots + restic backups to backblaze for my homeserver. My secret sauce is a healtchecks.io instance that blows up my phone if ZFS scrubs throw any errors, when local snapshots fail, or when restic checks or backups fail.

janci 16d ago

My setup: rsync to Synology NAS, snapshotting there, sync to remote Synology NAS over VPN. (The other NAS is at my parent's and they have similar setup syncing to me) The best part: it is just files, not some proprietary archive. I have high confidence in this. I can restore with the most rudimentary tools. This setup has worked for me well on linux, windows and now mac. I migrated to new computer&os by restoring the backup mostly.

Gud 16d ago

I use ZFS and do off site backups using rsync.net.

999900000999 16d ago

They are simple. The issue is companies love to fire QA people who should be the ones testing these. Microsoft says we don’t need no SDETs, our products can ship full of issues. What are the hostages, I mean customers going to do about it? I have no sympathy for any major corp who suffers data loss. Now individuals who lost grandmas wedding photos, that sucks. But my recovery strategy for that is simple. After scanning I emailed it out to one cousin who emailed them out to a few more. Plus I have a cloud backup ( I have no idea where the the original files are). Data is meant to be shared after all

l1am0 16d ago

Huge fan of borg + vorta for backups and time Maschine like „Jump back in time“. Wrote about it years ago: https://simon-frey.com/blog/borgvorta-is-finally-a-usable-backup-solution-for-linux-desktop/ https://simon-frey.com/blog/borgvorta-is-finally-a-usable-ba...

flippingheck 16d ago

> There are two types of people: those who have suffered a catastrophic loss of data, and those who will. This is an empirical claim, but is it grounded in reality? 99% of people don't backup, and that's probably the right choice because the risk is low, they can't meaningfully improve their restoration rate themselves and they don't care enough about their data to classify its loss as catastrophic. When a person hears someone talk about the importance of backups, but also haven't heard friends/family suffer this date, they will rightfully ignore this warning. Or do we all follow the best practices as it relates to backups, exercise, sleep, nutrition, accounting, house maintenance, ... Etc. I do not. Backups aren't near the top of that list. This stuff just isn't very important to most people and that's okay.

grumbelbart2 16d ago
> 99% of people don't backup This depends on your viewpoint. Using gmail for example is an implicit backup of your e-mails. Uploading your pictures to Facebook the same. The move from local, private compute and storage to the cloud solved this issue for many to a degree that it is now less important.
ygouzerh 16d ago
I think this is more a quote for engineers
afilipovski 16d ago
Let's say SSDs have an annualized failure rate of 1%. (The real figures are from 0.5% for more reliable ones, 1.5% for less reliable ones). The annualized failure rate for HDDs is a bit higher, but let's give it 1% anyway. Over 40 years (even with changing disks, upgrading PCs, etc.), the chances of you ending up with your data intact is 0.99^40=0.67. May not seem too bad, but this is just a hardware failure of your main drive we're talking about. When you factor in all the other types of screwups that happen, the annualized failure rate will skyrocket. So a more conservative version of this quote is that a 20-year old today is very likely to experience data loss over his lifetime. Thankfully with cloud being ubiquitous at least photos are given one copy in the cloud, but IMO even that is risky. People have different appetites for risk, and that's OK. But over the last 20 years the all of our data has become pretty important. I don't know how many people could stomach the loss of important documents, photos, etc.

jpthewall 16d ago

[dead]

bibelo 16d ago

> as long as you run restores every 6 months uh no, in a company, as a sysadmin, you try your backup files at least every month

BetterThanSober 16d ago

Backups isn't real until it's restored don't be like me folks

mamonoleechi 16d ago

3 rules of backup (don't remember where i read this) - do several backups (not using the same hardware brand or host if you do it online) - check your backup (if something was wrong, you have another backup from the first rule) - do not store your backups at the same place (if your place get flooded or burn, it will be pretty handy)

mattbillenstein 16d ago

Ran duplicity - it's really a blunt hammer, lots of duplication, rotation, etc. Then switched to Restic - so much better - highly recommend this.

ethanprk 16d ago

[flagged]

dirkc 16d ago

> “There are two types of people: those who have suffered a catastrophic loss of data, and those who will.” When I was a teenager, I was the reason for data loss for my dad, twice. Both times it was because I was re-partitioning a hard drive to install linux. You would think that taught me a lesson about backups, instead it reminds me every now and again to be grateful for an awesome dad and aspire to handle situations with my kid similarly :)

mesrik 16d ago
>> “There are two types of people: those who have suffered a catastrophic loss of data, and those who will.” Also "There are just two types of equipment; those which are already broken and those which will be broken". I've found a good way to point out anyone thinking right direction not to trust their valuable data in any single entity, product, location, etc. When I heard it decades ago IIRC 80's Nokia Data Unix courses I went. The distinction made Backups is just 1/3 of the triad Redundancy, Backups, Archives. 1) Redundancy is what you get using RAID, multiple network connectivity etc. ie. good for avoiding single point of failures and avoiding loss service or product availability. 2) Backups are for recovering lost data needed returning lost running known state right before fault or some time state before what backup schedule and used rotation cycle can provide. 3) Archives are meant to saving valuable data not expected to change or active use and but there is need saving much longer periods. Not seeing and understanding usefulness of difference between these seem hard some people I met over decades. Which then led to confusion, not knowing what they are up to with their data and also often later disappointment when they found out that for example backup programs are not great if you expected archiving instead.
wallst07 16d ago
Reminds me of a story: I was sitting at a restaurant waiting for my food. I casually notice two young children in the next booth playing with a phone, but pay little attention. Moments later their dad (presumably) comes back and asks for his phone back and sits down with their food. The kids are eating and the dad is fumbling with his phone with odd looks and I can see his frustration growing. Then out I hear something like "What did you do to my phone?" repeated loudly over and over. "You erased my phone, it's gone." The small children (guessing) had figured out how to factory reset his phone. My thought at the time was something like : How many times did I do something as a toddler I don't remember to piss of my dad like that and that he never got mad or brought it up later (Lots of times knowing myself).

abejora 16d ago

I agree with this sentiment! For our Abejora timesheet SaaS, setting up backups was one of the harder parts. We had to change direction a few times while implementing it. We finally got a nice setup with CloudNativePG + Barman. This allows for point-in-time restores, but there were a lot of lessons to learn along the way. - The various types of (database) backups (logical, binary, onsite, offsite, snapshots, write-ahead log...) in combination with the various types of data (database, files, cluster configuration...) - In our earlier approaches, we tried to preserve the old database volume if it was not corrupt, and use that in our restore. This caused so many complications, because you are fighting the recommended approach. So now, when we need to restore, we always restore from backups and the 'live volume' is dropped. - For a restore, we just spin up a completely new Kubernetes cluster, instead of trying to restore in-cluster. This is a lot easier. - Many object stores allow for retention periods, which you can put to good use to prevent malicious or accidental removal of backups. HOWEVER, not all of them are really 'locked'. In some services, you can still override the lock with a forced delete; in others, you can still remove the project holding the storage buckets, which will delete the buckets, and so on... so test those things, instead of just blindly depending on a 'retention period' claim. - We now automatically run a scheduled restore with verifications on a weekly basis. This requirement does shape your environment, so keep that in mind! There is also the question of how you can reliably and automatically verify that the restore restored the latest data (of a live prod environment). Various solutions exist here, but most are not very elegant! Honestly, this is only worth it if you are already handling sufficient volume. If you are just starting out, then the easier approach is to just go with a hosted database, which will handle backups and point-in-time restores for you.

cuvert 16d ago

For my small library photos, keepass passwords, etc around 30gb, I use syncthing, which will backup on my pc, on my parents pc (they live in another country), on my sisters pc and on my phone. They also use this triangular mode to backup their stuff. Syncthing is setup to only sync through tailscale or home routers network.

Oxodao 16d ago
sync is not backup Also why you sync through LAN only?
- cl3misch 16d ago
  It's tailscale or local LAN. syncthing allows you to specify multiple addresses for the same peer > tailscale or home routers network
- cuvert 16d ago
  I was in a vacation and I couldn't connect to a remote pc. And since then I set up wireguard in all routers and I also changed syncthing LAN only.
  - Oxodao 16d ago
    Ok, i do use it through their relay but i should do the same
cm2187 16d ago
So you delete a file by mistake and it replicates within seconds? Not exactly a backup. And a malware will do that to all of your files.
- cuvert 16d ago
  Yes, but I setup syncthing's file versioning to 25, in case of keepass, for example. Also, I set it up that only my PC can send, all other PCs, smartphones, can only receive. Sure, there are still many ways to attack this system, but I still prefer it.

OroPla 16d ago

Bit rot was mentioned in the article, but then no solution was offered. I also haven't found a good solution, unfortunately. If you only have a couple of Terabytes you could create check sums of everything and then verify regularly, but I feel this approach doesn't scale very well if you have lots of data.

GTP 16d ago
If you have lots of data, I would suggest using a filesystem designed for this, like ZFS or BTRFS. You would still have to spend in storage though, as ultimately protecting from bit rot requires redundancy.
- OroPla 16d ago
  I do use btrfs, but I don't really understand how to use it to protect against bit rot. I tried to look into it at some point, but got overwhelmed. Did invest in two backup copies, one of them off site, though. However, looking at prices, I can see one of those backups just being dismantled to be used as more storage.
  - suncore 16d ago
    Zfs and btrfs will avoid corruption caused below the file system layer, not something like malware or misuse from a tired sysadmin. I run a tool called integrit that creates checksums and lets me detect non-wanted changes. I only run it on important stuff that does not change too much like photos and videos. Not fun to have to regularly check the log but it works fairly well. This could maybe be extended with automatic checking of the logs.
  - GTP 16d ago
    If you got overwhelmed in trying to understand how to do it, just do it the easy way: create a ZFS pool of two drives of identical size in mirroring. Then, make sure to have regular scrubs (this means ZFS will regularly check all files for corruption, and recover from eventual errors), use your OS' task scheduler (e.g. Systemd or cron) to run this once a month. This should be sufficient to protect you from bit rot.

b3lvedere 16d ago

"Also make sure not to run backups at 2AM or 3AM, or things may get scary." I never though of daylight savings times causing such an issue. Learned something today.

PeterStuer 16d ago

The only backup system that I truly love and trust (from experience) is Proxmox Backup Server.

amelius 16d ago

I'm adding this to my list of things that should be simple but aren't. It already contains "Sending a 1GB file to a friend without seeing advertisements" and "Making a mixtape for my friends who are on a different streaming service".

Biganon 15d ago
Making mixtapes for others is an act of love that must never disappear

robschmidt90 16d ago

I also recently improved my backup approach. I am using daily restic snapshots for my off-site backup and it works pretty well. I wrote about this here [1]. [1] https://staticnotes.org/posts/cloud-backups-with-restic/ https://staticnotes.org/posts/cloud-backups-with-restic/

crabbone 16d ago

Some complexity levels not mentioned in the article: * Consistency groups. The article kinda almost gets there, but stops right before it realizes there's a problem. If you have multiple programs accessing the storage, s.a. a database (that you are snapshotting) and the application using the database, then you are risking an inconsistent snapshot when the database state you wrote is ahead or behind the application state. You need a mechanism to freeze all I/O beside the snapshot in order to ensure that the snapshot data is useful. * ACLs. And similar external mapping to the storage being used, eg. what if the users come from an LDAP server? Similar problem with symlinks coming from/going to ouitside the snapshot. * Sparse file support. Let's say you have a few VM images in your snapshot: if your tool doesn't support sparse files, it will likely greatly inflate the image sizes. Bottom line, either use the product's own snapshot functionality (ZFS has those), or switch to the product that does.

afilipovski 16d ago
Thanks for the input. These things never crossed my mind, even the first one which in hindsight should have been obvious to me. I will update the article to include these.

cranberryjoe 16d ago

Yes, yes they are simple. Btrfs and zfs have incredible snapshot and replicate

Kinrany 16d ago

There should be a standard for backups, one that makes it possible for a (potentially distributed) system to talk to a backup system and to save or restore a snapshot.

Hive80Lab 16d ago

[flagged]

cranberryjoe 15d ago

Hey kids It’s simple 1) separate data from applications 2) use a real filesystem like zfs or btrfs 3) cronjob snapshot, replicate onsite, replicate offsite 4) monitor size of volumes and snapshots, delete them as your policy 5) periodic zfs and btrfs scrub End of fucking story. No need for all that tricky shit y’all are doing.

siddhuk 15d ago

[flagged]

jllyhill 15d ago

Where can I find any discussions on how to actually test the restoration process? How can I verify that I really restored my 10k photos without losing anything?