25 ms·

I only lost 10 minutes of data, thanks to ZFS

436 points by chromakode 3y ago

Richard6544 3y ago
[dead]
simonebrunozzi 3y ago
I would love to be able to do this on a MacOS, with the click of a button.
- runeks 3y ago
  FWIW I’ve used Arq Backup[1] for several years now, and I’ve successfully restored at least twice after my MacBook died. It also encrypts the data before it leaves your computer, and supports tons of (cloud) storage solutions — I use Google Cloud Storagge and spend about a dollar per month on storage costs with hourly backups. [1] https://www.arqbackup.com/ https://www.arqbackup.com/
michaelcampbell 3y ago
"I don't backup my drives, I replicate them" <sigh> I mean, sure, he recognizes the difference which a lot don't, and I guess yay for zfs here to save him, but this is just irresponsible if you value your data.
windows2020 3y ago
Could snapshotting the filesystem every 10 minutes have contributed to its death?
- Filligree 3y ago
  No. Individual snapshots are a matter of kilobytes; desktop environments write considerably more.
  - istjohn 3y ago
    I read somewhere that snapshots are actually around 5 MB. Still not a lot, but a lot more than a few KB. A year's worth of hourly snapshots comes to over 40 GB just in snapshot overhead.
    - chromakode 3y ago
      Another factor to weigh in my case is this laptop probably spends at least 50% of its life suspended. The overhead should be measured in MB per hour of uptime.
- HankB99 3y ago
  Not likely. A snapshot just marks the most recently written block and prevents previous blocks from being altered. (More or less.) Since ZFS is copy on write, any changes to files will involve the same writes and some previously written data will not be deleted.
- baby_souffle 3y ago
  Nah, changes are COW so most snapshots are tiny.
- abrookewood 3y ago
  I guess it could contribute somewhat, but I don't think it is that much additional work: for every new write (since the last snapshot), there is one additional read as the data is sent. It isn't reading the whole file system, just the incremental data.
- codetrotter 3y ago
  ZFS is very special, and it is cheap to make snapshots with ZFS, because ZFS uses copy-on-write. Intuitively I would think that the amount of extra writes is pretty low, even if you snapshot very frequently. But scientific measurements would be nice. I used to do snapshots every minute, every hour and every day with ZFS on some servers I administered. I’d purge the minute snapshots after 60 minutes. And I had cron jobs on other machines to backup the hourly and daily snapshots. I had it set up so that hourly snapshots were kept for something like 72 hours. And the daily snapshots were kept forever. The idea with the every minute snapshots being that they were for undoing manually made mistakes during SQL migrations etc. It worked well for me. I still use ZFS on my FreeBSD servers. But at the moment my projects are low traffic and the data only changes in important ways some rare times. So with my current personal servers I manually snapshot about once a week and manually trigger a backup of that from another server. Another thing I’ve changed is that now I only snapshot the parts of the file system where I store PostgreSQL databases and other application data. I no longer care so much about snapshotting the operating system data and such. If I have a serious hardware malfunction I will do a fresh install of the OS, and I have a log of what important config values are used and so on, that my backup scripts copy when I run them, without copying all of the other things.
  - vasco 3y ago
    This sounds overkill even for production data, much less personal data, particularly the every minute and the fact you keep dailies forever. Unless you're a custodian of some secret society's files!
    - codetrotter 3y ago
      > sounds overkill But it wasn’t. It was very useful in fact.
  - oefrha 3y ago
    Copy-on-write and cheap snapshots was quite special when ZFS was created. It’s hardly special in 2023, when every single non-vintage iPhone, iPad and Mac has that.
- georgyo 3y ago
  Not likely. The snapshot doesn't write much, and both SSDs and ZFS are copy on write. Which means the cost of writing after a snapshot is the same as before the snapshot. On the other hand context is missing. Both SSDs and ZFS don't like being full or even close to full. The working set was ~650GB, of the drive was 1TB, then those snapshots could have easily made the drive over 90% full. This could have made ZFS unhappy all by itself.
  - chromakode 3y ago
    I agree that it was unlikely. The total size of all data and snapshots was 625 GiB on a 2 TB drive (which had seen less than 2 years of moderate use). It was a pretty unexpected failure.
  - Xaiph_Rahci 3y ago
    > cost of writing after a snapshot is the same as before the snapshot I didn't understand this, could you please clarify? If there was no snapshot, there would be only one write operation, the actual write. However, with snapshot in place, in addition to actual write, there is a copy operation which copies the original data and writes to snapshot location. So, there should be two write operations (actual + copy).
    - boomboomsubban 3y ago
      There's no copy operation, the previous data isn't overwritten and the new data is written to a new block. It's "copy-on-write."
    - rincebrain 3y ago
      ZFS is never overwriting in place in either case, you're just not freeing the old one if it's in a snapshot, and a snapshot is just a note that "nickname this point in time 'mysnapshot', and don't clean up anything referenced at this point in time", so it's very cheap to make, and you just check it later when you would be cleaning things up.
      - viraptor 3y ago
- xpe 3y ago
  As I understand it, taking a snapshot with ZFS involves writing a metadata object and some data references. Assuming 100 GB of data, 128K block size, and 64 bit pointers, I'd guesstimate * that new data written during a snapshot would be in the ballpark of 5 MB. Is doing that 6 times per hour (52,560 times per year) enough to cause premature wear on the drive? That would be ~256 GB per year. This is likely under 1% of an SSD's write endurance. So, I'd be surprised if taking 10 minute snapshots was a significant causal factor. * I could be wrong, I asked for some help from not the most reliable sources. Happy to be corrected. Still, if my estimate is higher than actual and yet still unlikely to affect drive longevity, it may be moot.
- endisneigh 3y ago
  Of course it contributed. But it probably wasn’t the main reason or a significant contributor.
- E39M5S62 3y ago
  Nope. As others have mentioned, ZFS is CoW. Snapshots are "free" in that they (basically) point to a transaction group in the filesystem. They record a small amount of metadata to disk on each snapshot - on the order of a few MB. This is much much lower than an rclone/sync style backup.
  - Izkata 3y ago
    That's also the least interesting part of comparing a ZFS backup to rsync/rclone: The rsync way is to crawl the entire tree being backed up to diff over the network then copy the differences. Because of snapshots, ZFS already knows all the changes that occurred between snapshot A and snapshot B, and (provided state up to A has already been backed up) can update the backup by pushing all changes between A and B as one big binary blob without having to scan or diff anything.
- deleted 3y ago
  [deleted]
- numpad0 3y ago
  Minimum write size of a modern Flash chip can be ~100MB(!) according to a comment found in a random orange website[1]. So 5MB write every 10 minutes can be 600MB/hr, which is 4.8TB/8-hr-day, which is 24TB/40-hour-week, which is 3.43 DWPD real time for a 1TB drive, and 2500 TBW in 2 years real time[2]. Official quoted specification for SN850 is 600 TBW of write endurance, likely after derating for obvious warranty implications. Incidentally, 2500TB is also a typical endurance figure for many SSDs in this market. Overall, to me, sounds not entirely impossible. I kind of wonder what's the controller says in SMART data, if still alive. On Linux the command is `apt install smartmontools; smartctl -s on /dev/sda; smartctl -A /dev/sda`, and it shall print out a table[4]. On Windows, just install CrystalDiskInfo[3]. 1: https://news.ycombinator.com/item?id=29165202 https://news.ycombinator.com/item?id=29165202 2: DWPD: drive writes per day, TBW: Total Bytes Written - in terabytes 3: https://crystalmark.info/en/software/crystaldiskinfo/ https://crystalmark.info/en/software/crystaldiskinfo/ 4: Note that "Pre-fail" means the value is supposed to change when about to fail and "Old_age" means the value is supposed to indicate age, NOT "this is bad and about to fail" and "this drive is old". It always says all Pre-fail and Old_age. Someone should have changed it to "somewhat_boolean" and "life_remain" long time ago in my opinion.
  - chromakode 3y ago
    Unfortunately the drive didn't appear accessible at all via nvme-cli. Interestingly it shows up in lspci but doesn't get a /dev/nvme. It tends to hang the UEFIs of the two systems I tried it in when they try to read it.
  - pseudalopex 3y ago
    Minimum write size is not erase block size.
  - kalleboo 3y ago
    My machines have always been just constantly writing logs, like every couple seconds (macOS does this), and the write wear has never been anywhere near that bad. The advertised endurance must take into account write amplification for typical loads.
  - mafuy 3y ago
- p_l 3y ago
  No. ZFS implements normal writes to the disk as snapshots (just unnamed ones), so in fact you can only write to disk through creation of a snapshot or by writing to "Intent Log" which is short-term log of data that is going into next snapshot - but which was synced before the snapshot was done, and as such it's secured in case of power failure.
riku_iki 3y ago
spoiler: he has 10 minutes incremental backups.
- ggm 3y ago
  Its "backups" join "zfs makes snapshots easy" join "snapshots make incremental backups easy" join "backups on device aren't a backup" join "I had off-device backups" which reduces to "I had backups" indeed. 3-2-1 forever!
  - Dylan16807 3y ago
    > "backups on device aren't a backup" That wasn't part of the article. It was a single drive failure, so RAID would have done fine.
    - ggm 3y ago
      Yes, RAID will get you over some failures. But, it still isn't a backup. Backup is what gets you over corrupted RAID, loss of both sides of the mirror stripe, entire disk failure when its not RAID. What he does is run zrep to make a backup. it covers his needs. ZFS snapshot by itself is only transitionally a "backup" for the immediacy of change, it's the least safe form of backup if it remains on the same logical drive structure.
      - deleted 3y ago
        [deleted]
      - riku_iki 3y ago
        > Yes, RAID will get you over some failures. But, it still isn't a backup. backups also can fail, that zrep can start failing after some os/kernel update without notifying owner. The question is in probabilities of failures, I kinda would trust industrial raid more than some custom made hobby solution.
xpe 3y ago
Very quick summary: The mastodon thread refers to https://zrepl.github.io https://zrepl.github.io "zrepl is a one-stop, integrated solution for ZFS replication."
- runeks 3y ago
  Looks like it needs to speak to a daemon running on the storage server? Would be cool if it could just use e.g. S3 for storage.
- istjohn 3y ago
  Does anyone know how zrepl compares to sanoid/syncoid other than that zrepl is written in Go and sanoid/syncoid are Perl scripts?
  - etherael 3y ago
    I use sanoid to do basically the same thing as this, and was interested in giving it a shot to see if it was more hands off but it's definitely a more complex setup to begin with, given you have to setup your own SSL certs etc, not sure why they wouldn't just use SSH transport for this like everything else.
    - chromakode 3y ago
      I use Wireguard to secure and authenticate the transport. Much easier to set up! SSH is also an option.
      - etherael 3y ago
        Thanks. Good to know that's possible, it's exactly what I use for sanoid also, so I guess the quickstart just assumes that layer isn't available.
ChrisMarshallNY 3y ago
I tend to use Apple’s Time Machine incremental backup to a Synology spinning rust server. I also have an external SSD that I’ll mirror the internal drive to, if I’ll be doing anything dodgy, or upgrading my machine. That works. TM restores can be quite slow, but almost all my important data is in Git (and hosted storage), so it’s not really been an issue. I just use TM every now and then, if I have a single file I want to backtrack. I also have one of the notorious[0] SanDisk drives. I don’t use it for anything important. It just has some game storage. Since I’m a Mac user, games aren’t really much of a factor for me, and I won’t cry, if they croak. [0] https://arstechnica.com/gadgets/2023/08/sandisk-extreme-ssds-are-worthless-multiple-lawsuits-against-wd-say/ https://arstechnica.com/gadgets/2023/08/sandisk-extreme-ssds...
- op00to 3y ago
  I’ve given up on Time Machine. It never seems to work past a month or two for me on my Synology w/ atalk etc.
  - jedberg 3y ago
    It was always breaking on my Synology too. I ended up just attaching an 8TB spinning rust directly to the Mac and it's been flawless since. Time Machine really doesn't like using remote disks that aren't official Apple gear.
    - kalleboo 3y ago
      Even when I had an Apple Time Capsule, it would break about once a year. It's just a flakey system. Wish they'd add the equivalent of zsend to APFS instead of using the weird "gigantic sparse disk image with hard links in it" system
      - praseodym 3y ago
        When backing up to an APFS Time Machine volume it does work a bit like that, at least no hard links are used any more: - https://eclecticlight.co/2021/03/11/time-machine-to-apfs-understanding-backups/ https://eclecticlight.co/2021/03/11/time-machine-to-apfs-und... - https://eclecticlight.co/2021/04/16/time-machine-to-apfs-using-a-network-share/ https://eclecticlight.co/2021/04/16/time-machine-to-apfs-usi...
NoZebra120vClip 3y ago
Just before I exited the Linux world entirely, I was beginning to chip away at the iceberg known as btrfs, and it was fascinating. I saw so much promise in many of its features, for revolutionizing backups and organizing my disks and everything. Now btrfs isn't ZFS, but it has some feature parity and perhaps the "poor man's ZFS". It's also much more reasonable to run on certain OS, due to the licensing, packaging, and in-kernel status of ZFS being kind of weird. One memorable time I was encouraged to use ZFS was when I mentioned to the Linux User's Group that I'd had to pull the power cord to reboot my computer, and I was roundly scorned for this foolish maneuver. But you may change your mind about the wisdom of doing either one when you consider that the system in question was a Raspberry Pi. Heh.
- KennyFromIT 3y ago
  Alright, I'll bite... What led you to leaving the Linux world entirely? What can "the community" learn from your experience to make it better for others?
  - NoZebra120vClip 3y ago
    Linux is a great fantastic experience, and I have no qualms or ill will about it. I simply had no use for it anymore, and I needed to simplify. I've said before, I'm not a sysadmin anymore, I don't tinker with systems, I need stuff to be operational and in production. I still love Linux and I'd use it for any given server or Raspi if that were part of my job. I do use it daily in my job, but to a very minimal extent.
    - theaiquestion 3y ago
      Did you switch to a mac or to a windows machine?
      - worthless-trash 3y ago
        They said "Operational and in production" :)
op00to 3y ago
I wonder how much he gained from entirely restoring the system versus simply reprovisioning (gasp, even manually reinstalling) and restoring needed files a will. I'm not sure there's a lot of value in snapshotting and restoring stuff in a lost ssd situation tthat's also available in mirrors across the world.
- theossuary 3y ago
  The biggest time save is in time spent recovering. It's so much faster to restore the entire system than to reinstall the OS, reconfigure the bootloader, resetup disk encryption, reconfigure user accounts, reinstall all software, manually reload configs, etc. Or put it more directly, full disk backups are a great way to get RTO down.
  - op00to 3y ago
    In this case the author had to do arcane magick to restore his zfs snapshot. This wasn’t a routine raw dd restore.
    - chromakode 3y ago
      Agreed. It was a large time investment that happened to pay off. If I ever have to do it again I'll be much faster. I hope that with wider ZFS adoption some of the routine tasks will be automated better in the future. I see no reason why in a couple years this couldn't be a mature fire and forget user experience.
      - op00to 3y ago
        If it can be done manually, it can be automated and made more reliable!
- chromakode 3y ago
  I've reflected similarly after this exercise. I've lost data before, but it felt terrible to lose my context and working memory. While I make sure the most important stuff is in git, there's a bunch of momentum and working memory in my bash history and system configuration. It's also nice to not have to think very hard about a patchwork of backup plans. It's nice to get a fresh start every now and then, but not under duress. I was in the middle of a multi day project and was gonna lose time either way. It was real nice to boot back into a machine that felt like home.
predictabl3 3y ago
Zrepl is a big part of why I feel secure doing the digital nomad thing. A script, run nightlyish, opens a separate-headered LUKS-protected ZFS pool and then copies all snapshots over. That NVME enclosure lives in my "purse" that never leaves me sight/body. Between this and NixOS, I can provision a new identical laptop in about 10 minutes. I recently added off-site replication as well, so even if I get completely devastatingly mugged, there's still about zero chance of serious data loss. Zrepl is absolutely brilliant software. Easy to run with, but incredibly sophisticated and powerful if you need all the knobs. I can't praise it enough.
- tmountain 3y ago
  You should do a write up about how this works. It sounds very interesting.
  - predictabl3 3y ago
    It's on my list, but... to be honest if you Google "separate header Luks", you'll find it's trivial to create a LUKS device with a detached header. Then the default ZRepl quick start will get you going with the basic pool-to-pool local replication. That will get you almost all the way there. :) I used their docs/guide to do the remote replication too, though it would make a good write-up as I could throw in how I use sops-nix for securing the Zrepl TLS bits for the remote scenario too...
    - LWIRVoltage 3y ago
      This does sound really cool- and like a way to ultimately set up a secure, not that complex backup method...
- xpe 3y ago
  Kudos. You're probably safer than most people who are only one theft, fire, flood, or other disast
  - xpe 3y ago
kristopolous 3y ago
I just have rsync running in a cronjob. How is this significantly different? I imagine it is, but I don't know how.
- kadoban 3y ago
  Mostly different in terms of performance and wear on the drives. If you rsync over and over, it has to scan basically the whole filesystem for changes each time. Zfs snapshots don't. The snapshot is ~instant and the calculation of what to send has no need to examine any files. I don't think the performance and drive-lifetime hit of running rsync every 10 minutes would be good. Zfs should have an edge in terms of atomicity as well, but in practice I'm not sure how much that matters. I _think_ it does matter but isn't perfect (zfs can't trick applications into doing atomic writes if they're not already, but it won't have _another_ worse layer of breaking atomicity like rsync must).
  - toast0 3y ago
    Depends what else the machines are doing, and how much ram, and the rsync settings. If you read all the files on both sides, every time and don't have more ram than disk, it's going to be a lot of work. If you're just looking at directory entries most of the time, there's a good chance that's all cached and it's no disk load, other than the small changes. I ageee with you though that atomicity is a big difference, if it matters, and in most cases, it probably doesn't. Personally, I've mostly stopped doing rsync backups in favor of zfs send, but I've still got one I need to get around to changing. Sanoid/syncoid is pretty decent for less effort snapshotting and syncing snapshots; but I haven't done anything with encrypted datasets. For most of my systems, I'd prefer recovery over security. For the one system in iffy hosting, it runs full disk encryption as a layer below zfs, so it's zfs sends are cleartext, too. (The hosting facility has given me other customer's unwiped disks; better for me to assume my disks won't be wiped)
    - kadoban 3y ago
      > If you read all the files on both sides, every time and don't have more ram than disk, it's going to be a lot of work. If you're just looking at directory entries most of the time, there's a good chance that's all cached and it's no disk load, other than smthe dmall changes. Yeah that's a good point. I know that rsync is _quite_ clever, but at least any incantations I've ever done it still hits the drives a good amount. I'd ballpark guess a couple of orders of magnitude better than just "cp -r" or something, but still a couple of orders of magnitude worse than zfs snapshots. Yeah you're 100% right it'll depend on bunch of variables though. > Sanoid/syncoid is pretty decent for less effor snapshotting and syncing snapshots; but I haven't done anything with encrypted datasets. I'm not sure I'd recommend it, but I use both directly on encrypted datasets. I have tested recovery a couple times and it works fine, but I've read some cautionary tales too. I _think_ they're all old issues?
xpe 3y ago
For those that don't know, there are many wonderful incremental backup solutions that don't require ZFS. * For one, I personally recommend Restic (https://restic.net https://restic.net) because of its deduplication. * People on macOS don't have ZFS, well... maybe they could? See https://github.com/spl/zfs-on-mac https://github.com/spl/zfs-on-mac
- chromakode 3y ago
  bupstash.io was my favored option other than ZFS. It's a beautiful and performant solution. Being filesystem agnostic is an advantage in many contexts. In the end I chose ZFS for the efficiency of snapshots (vs. a full disk scan) and atomicity. Both enable more frequent, smaller syncs, which is perfect for a laptop.
- drexlspivey 3y ago
  macOS Time machine does incremental backups, is there any other reason wou might need ZFS?
shruggedatlas 3y ago
How could I backup using incremental atomic snapshots on Windows?
- toast0 3y ago
  Look into the Windows Copy Shadow Service. Unfortunately, I haven't had luck with Open Source backup software that uses it (the shadow copy snapshot would fail, the error code would be no help, and finding no resources, I gave up), but commercial software I've used was great. When I was at a big corp, the commercial backup software whose name escapes me at the moment would litterally wait for files to be saved, then do an incremental backup. As of now, I'm using Veeam on my personal machines, and it runs an incremental backup nightly and saves to an smb share.
- deadbeeves 3y ago
  I've been using urbackup for years. It does disk image-based backups and/or file-based backups. Disk image-based backups are incremental at the block level, and file-based backups are incremental at the file level (so if a single byte of the file has changed, the entire file gets backed up). It uses the Volume Shadow Copy mechanism that the sibling comment mentioned to get atomicity and avoid file locking issues.
archo 3y ago
https://archive.is/VPEAP https://archive.is/VPEAP
SoftTalker 3y ago
tldr: my drive died, and I had a backup. zfs seems incidental to me. I could have a 10 minute cron job rsyncing changes from ext4 and been just as well off.
- poisonborz 3y ago
  Classic HN comment!
  - inshadows 3y ago
    [dead]
- bomewish 3y ago
  Is this so?
KyleSanderson 3y ago
bcachefs is the near future for Linux here. https://bcachefs.org/ https://bcachefs.org/
- brunoqc 3y ago
  Any tldr about why we should be looking forward to this? Any cool things?
  - KyleSanderson 3y ago
    Erasure Coding when it lands should be pretty solid. Until then per-directory data replicas is the killer feature for me (Music has 3, Documents has 5, Downloads has 1). Something to be very excited about with full compression and encryption.
    - brunoqc 3y ago
      Nice, thanks!
    - vladvasiliu 3y ago
      You can do that with ZFS at the cost of defining separate filesystems per directory. I don't use multiple replicas, but I use that to tailor my backups per directory. ~/documents is snapshotted and backed up on the regular, with long-lived snapshots. Code is snapshotted regularly, but snapshots don't live too long, and they're not shipped to a different drive. I don't care for ~/tmp so no snapshots. --- edit: the cost, besides having to actually create the file systems, is that moving data between them isn't instant.
      - philkrylov 3y ago
        > the cost, besides having to actually create the file systems, is that moving data between them isn't instant. No more the case after block cloning support goes production: https://github.com/openzfs/zfs/pull/13392 https://github.com/openzfs/zfs/pull/13392
- 3y ago
DavidSJ 3y ago
My snapshots are encrypted by the original computer (this is cool because the NAS can’t read them!). So I also needed to restore the encryption “wrapper key” to be able to use the backups. Not gonna lie, it was pretty terrifying until I had my first confirmation I could decrypt the data. Note: no bespoke backup method should be assumed functional unless you actually periodically check that you can restore data from it.
- raisin_churn 3y ago
  Or non-bespoke.
  - justinjlynn 3y ago
    In short, backups aren't actually taken until they have been verifiably restored.
    - yomlica8 3y ago
      I always see this advice but...how do you even do that without having an entire additional set of disks to restore too? You can't restore to production obviously, as aside from the downtime if the test fails you've just destroyed your good copy and proved the other copy is also bad. About the best I can think of is restoring a small part of the set as a sample, which isn't really testing the whole thing.
  - DavidSJ 3y ago
    I added that word because at least with non-bespoke backups, you have probably thousands of users each day testing restoration out of necessity, and word might get around if the method failed to restore. But nevertheless, one should still test even then.
- totetsu 3y ago
  I always meant to keep a copy of my LUKS header on separate disk, just in case, but..
ck2 3y ago
I really figured we'd have super easy hardware Raid1 in even consumer level PCs by now given how cheap drives are (and unreliable). My SSD boot drive makes me nervous as heck, constantly backing it up.
xet7 3y ago
With ZFS, he is better prepared than other WD and Sandisk SSD users. https://petapixel.com/2023/08/08/sandisk-portable-ssds-are-failing-so-frequently-we-can-no-longer-recommend-them/ https://petapixel.com/2023/08/08/sandisk-portable-ssds-are-f... https://www.theverge.com/22291828/sandisk-extreme-pro-portable-my-passport-failure-continued https://www.theverge.com/22291828/sandisk-extreme-pro-portab... https://news.ycombinator.com/item?id=37042587 https://news.ycombinator.com/item?id=37042587 https://www.theverge.com/23837513/western-digital-sandisk-ssd-corrupted-deleted-questions https://www.theverge.com/23837513/western-digital-sandisk-ss... https://news.ycombinator.com/item?id=37188736 https://news.ycombinator.com/item?id=37188736
- anonuser123456 3y ago
  I had two drives in my mirrored zpool die within 8 minutes of one another. Both HGST drives too. A very sad day. Thankfully I had been regularly zfs sending my contents to another site and lost very little data. ZFS is rad.
  - anotherhue 3y ago
    > ZFS is rad. Typo: RAID
    - LargoLasskhyfv 3y ago
      I'd bet that was intentional, because https://www.urbandictionary.com/define.php?term=Rad https://www.urbandictionary.com/define.php?term=Rad
    - anonuser123456 3y ago
      You deserve all my upvotes :)
    - btown 3y ago
      Only 1/4 of data lost! Meaning still recoverable!
    - quickthrower2 3y ago
minimalist 3y ago
For people who don't want to use ZFS but are okay with LVM: wyng-backup (formerly sparsebak) https://github.com/tasket/wyng-backup https://github.com/tasket/wyng-backup
bomewish 3y ago
What's the equivalent setup for a Mac user?
justinclift 3y ago
> Not gonna lie, it was pretty terrifying until I had my first confirmation I could decrypt the data. Sounds like the process was a bit less tested and documented than optimal. For a home system or personal desktop that's not super unusual. You don't want to be working out your restore procedure on the fly for production servers though. ;)
- chromakode 3y ago
  For sure. I knew I had all the right ingredients to restore, and had kicked the tires 6 months back, but I initially copied over the wrong key. When it failed to load I had a sad moment until I realized my mistake. In such fail moments there's a flash of clarity where every process gap becomes blindingly obvious.
BearhatBeer 3y ago
I feel like a computer running ZFS and serving files, is fine. And it should itself be treated as a strage device with a full parallel backup even though this has cost. But your computer shouldn't run ZFS, that's for the big boys upstairs. Code's too big, it's too hungry.
not_your_vase 3y ago
A few years ago we had capacitor plague. Are we living now the storage plague? It's getting ridiculous that all storage is getting worse and worse. WD is making HDDs crappy with SMR, manufacturers says that 3 years operating time is already too much for SSDs and HDDs, and they don't joke. I just had a Kingston SSD (okay, that was like 8 years old) and a portable WD HDD (~2 years old) die just this year. The internet is full of problems lately about data loss and longevity issues. I remember 20 years ago HDDs were not meant for eternity either, but they definitely outlived the usefulness of the computer that they were bought with...
- andromeduck 3y ago
  Don't worry, your chips will start glitching after a few soon too. We're hitting scaling limits. Exponential growth is slowing.
  - nicman23 3y ago
    that does not make any sense if you are talking about the controllers.
    - andromeduck 3y ago
      https://support.google.com/cloud/answer/10759085?hl=en https://support.google.com/cloud/answer/10759085?hl=en
- kalleboo 3y ago
  I've had a lot more luck with hard disks these days than I did 20 years ago. Remember 20 years ago was the era of the infamous IBM Deathstar drives where the magnetic coating would literally start sprinkling off the platters. Also the era of terrible terrible Maxtor drives that died in 1-2 years, which Seagate then bought and made their drives also unstable for a while. I ran a server with around 8 drives and had to keep replacing disks at the rate of about one per year. Meanwhile today I'm helping admin a ZFS server with 20+ drives and drives have about a 4-5 year lifespan. > but they definitely outlived the usefulness of the computer that they were bought with Computers were also much more quickly obsoleted back then. When today a 6 year old computer is totally useable, back then you really felt it if your machine was just 3 years old.
makach 3y ago
I read this piece as someone who just got lucky. He never tested it until he needed it. 10 minutes is good. Don’t let this story fool you to not take backups.
- mijoharas 3y ago
  > Don’t let this story fool you to not take backups. Do you mean don't let this story fool you to not test your backups? Because the whole point of the story is he was saved by having his backups. (Though you're right that he lucked out by having it work when he hadn't tested it)
- chromakode 3y ago
  I tested my backups about 6 months ago when I set up zrepl. When I mentioned it was scary until I could decrypt the data, that wasn't the whole story, actually: initially I restored the wrong wrapper key and it failed to load! It's also scary in general to go from 2 copies to only 1 copy of data. A friend and I have been planning to trade replicas but haven't set it up yet. There's definitely still room for improvement in my setup.
Helmut10001 3y ago
I wonder if zrepl could be run in WSL2 - would be nice to backup Windows computers as well using this approach. At the moment, I use Nextcloud to sync data to my server. It is a more selective approach and Nextcloud is, per se, not a backup solution because not all files can be backed up.. and live-sync is always a half-baked backup solution.
- vladvasiliu 3y ago
  How would that work? Zrepl ships ZFS snapshots. You could probably wrangle wsl2 to install its distro on zfs. If so, I see no reason why zrepl wouldn't work with the linux environment. But snapshotting the whole Windows drive? I don't think so. If you want similar features, I think ReFS comes close. AFAICT it's not supported as a boot drive.
  - Helmut10001 3y ago
    I thought maybe if Windows is installed on a ZFS volume, and WSL2 on another, Zrep from Linux should be able to backup both, snapshots from the Linux and the Windows volume. But a quick search on Google reveals that Windows on ZFS is not a thing, yet.
    - vladvasiliu 3y ago
      You could flip that on its head, and run Windows in a VM on Linux on a ZFS volume. Depending on what you do on Windows and your particular hardware setup, this may or may not work well enough. I see people use GPU pass-through to play games on Windows VMs. You could probably pass through practically all devices (GPU, sound, network, keyboard, etc.) and this could work well-enough if you don't need the absolute last drop of performance from your CPU and drives. And since KVM supports nested virtualization, you could run WSL2 in the Windows VM. And if I'm not mistaken, the KVM agent in windows can be told to ask the guest OS to sync the drives, and some Windows applications [0] can even cooperate with this and flush their buffers to disk. You could signal this before creating the ZFS snapshot. [0] probably not most, but I think MSSQL does.
Kiro 3y ago
Am I the only one who doesn't have any important data? If I lost everything today I would just start afresh and move on.
- dkh 3y ago
  At all? Like, anywhere? Or do you just have data living on cloud services instead of locally?
  - Kiro 3y ago
    At all. What important data do you have? I really can't think of anything that I would miss if everything disappeared.
    - Symbiote 3y ago
      Address book, photographs/video of people I care about and holidays, personal diary, hobby projects, old letters/emails, stored passwords, archived bank statements/contracts/insurance and other important documents. I think you are very unusual if you don't care about any of this.
      - ghaff 3y ago
        Most administrative stuff, especially if it's in digital form anyway, can probably be recovered without too much hassle if it's lost. But you're right that most people have a bunch of stuff (not all of which is admittedly digital) they wouldn't want to lose.
      - Kiro 3y ago
        Yes, I think it's unusual to not care about photographs/videos. I used to think that they were important and that I would want to revisit them some day. However, now when I'm old enough that I should it simply hasn't happened. They are just some files that I will never open.
- unmole 3y ago
  > Am I the only one who doesn't have any important data? Quite possibly.
brucewayne46 3y ago
[flagged]
sandreas 3y ago
For every ZFS fan, I can recommend zfs-auto-snapshot[1]. I use it on my proxmox server[2] to auto manage snapshots incl throwing away old ones. [1]: https://github.com/zfsonlinux/zfs-auto-snapshot https://github.com/zfsonlinux/zfs-auto-snapshot [2]: https://pilabor.com/series/proxmox/restore-virtual-machine-via-zfs-snapshot/ https://pilabor.com/series/proxmox/restore-virtual-machine-v...
- nicman23 3y ago
  i actually made 2 scripts to automate the sending and deletion of old snapshots along with one that calls auto snapshot on a vm rebooting / shutting down. I was thinking that they might be useful to other people.
copirate 3y ago
I've read that ZFS is less safe than other Linux filesystems if you don't use ECC RAM, because it assumes that there are no memory errors and therefore doesn't provide a tool to repair a filesystem corrupted by such errors. Is this true?
- Modified3019 3y ago
  It's not true. That's basically ancient forum myth, alongside the also incorrect "ZFS needs 1GB memory per TB of HDD" nonsense that has thankfully mostly died out finally. ZFS makes no additional assumptions when using ECC vs non-ECC memory. It is theoretically possibly to construct a scenario where evil ram does all the exactly right things needed fool ZFS and corrupt your filesystem. Any pearl clutching about this thing which has never happened somehow also ignores that every filesystem is going to get corrupted. In reality, while ECC memory is always nice to have, it's no more required than any other filesystem. Though personally now that amounts of +32gb are common, I generally prefer error correction/detection over ultimate speed these days. Though ironically ECC memory is actually really nice to overclock, because I can actually just check my logs and prove if my system is actually stable. There so many actual dangers to your data in comparison that it's laughable. The biggest one being you. Followed by hardware failure, malware, and genuine ZFS bugs. I'd stay far away from raw sends of encrypted datasets in ZFS for a while, there are edge cases that haven't been resolved yet. Edit Longer article saying the same thing: https://jrs-s.net/2015/02/03/will-zfs-and-non-ecc-ram-kill-your-data/ https://jrs-s.net/2015/02/03/will-zfs-and-non-ecc-ram-kill-y...
curt15 3y ago
Was he able to back up and restore his boot/ESP partition as well using ZFS, or did those need to be reprovisioned manually?
paldepind2 3y ago
I see that his NAS is using an external harddrive enclosure over USB. I'm really curious about how wise such a NAS setup is? In some ways it seems very attractive as you can just get a low-power SoC (like a Raspberry Pi or a NUC) and hook it up to the external drives. But there also seems to be many potential pitfalls. Like, how slow will a resilver be over the USB? Might it be unusable/dangerously slow? How reliable is the USB connection? Does it perform to spec or might it cause weird issues? Can you get SMART info over the USB connection? Other issues?
- Terretta 3y ago
  > NAS is using an external harddrive enclosure over USB. The N in NAS means "network" as in network attached storage so it's not a NAS. > really curious about how wise such a NAS setup is? For data use cases like this, USB 3 can be reasonably comparable to Thunderbolt 3, and that connection is generally faster than the media. This use case seems to be using the external device as a continuous external backup rather than as network attached storage, which is a great use of USB-C dongle SSD enclosures that are the same size or larger than the SSD inside the laptop. You effectively have mirroring as well, since you have both the internal SSD copy and the external SSD copy, in different makes and forms, unlikely to both fail.
  - LgWoodenBadger 3y ago
    In the context of the grandparent, a "NAS" is a device, attached to the network, that provides storage. How the storage the NAS provides is connected is irrelevant. On a related note, I can't think of any drive that has a network interface instead of something like USB, Firewire, Thunderbolt, SCSI, IDE, etc. so how exactly would you define a NAS device?
  - paldepind2 3y ago
    How I interpret the setup that OP has, is that he some computer (a PI, an old laptop, a NUC, etc.) which is connected to drives in an external drive bay through USB. This is a somewhat common setup and is definitely a NAS. > For data use cases like this, USB 3 can be reasonably comparable to Thunderbolt 3, and that connection is generally faster than the media. External HDD enclosures can often contain 4 drives. During a resliver all of these could be heavily accessed. I'm not sure how the single USB 3 connection fares in this scenario. In a normal desktop you'd have four separate SATA connections, and even then resilvering a large RAID setup can take quite some time.

436 points by chromakode 3y ago

Richard6544 3y ago

[dead]

simonebrunozzi 3y ago

I would love to be able to do this on a MacOS, with the click of a button.

runeks 3y ago
FWIW I’ve used Arq Backup[1] for several years now, and I’ve successfully restored at least twice after my MacBook died. It also encrypts the data before it leaves your computer, and supports tons of (cloud) storage solutions — I use Google Cloud Storagge and spend about a dollar per month on storage costs with hourly backups. [1] https://www.arqbackup.com/ https://www.arqbackup.com/

michaelcampbell 3y ago

"I don't backup my drives, I replicate them" <sigh> I mean, sure, he recognizes the difference which a lot don't, and I guess yay for zfs here to save him, but this is just irresponsible if you value your data.

windows2020 3y ago

Could snapshotting the filesystem every 10 minutes have contributed to its death?

Filligree 3y ago
No. Individual snapshots are a matter of kilobytes; desktop environments write considerably more.
- istjohn 3y ago
  I read somewhere that snapshots are actually around 5 MB. Still not a lot, but a lot more than a few KB. A year's worth of hourly snapshots comes to over 40 GB just in snapshot overhead.
  - chromakode 3y ago
    Another factor to weigh in my case is this laptop probably spends at least 50% of its life suspended. The overhead should be measured in MB per hour of uptime.
HankB99 3y ago
Not likely. A snapshot just marks the most recently written block and prevents previous blocks from being altered. (More or less.) Since ZFS is copy on write, any changes to files will involve the same writes and some previously written data will not be deleted.
baby_souffle 3y ago
Nah, changes are COW so most snapshots are tiny.
abrookewood 3y ago
I guess it could contribute somewhat, but I don't think it is that much additional work: for every new write (since the last snapshot), there is one additional read as the data is sent. It isn't reading the whole file system, just the incremental data.
codetrotter 3y ago
ZFS is very special, and it is cheap to make snapshots with ZFS, because ZFS uses copy-on-write. Intuitively I would think that the amount of extra writes is pretty low, even if you snapshot very frequently. But scientific measurements would be nice. I used to do snapshots every minute, every hour and every day with ZFS on some servers I administered. I’d purge the minute snapshots after 60 minutes. And I had cron jobs on other machines to backup the hourly and daily snapshots. I had it set up so that hourly snapshots were kept for something like 72 hours. And the daily snapshots were kept forever. The idea with the every minute snapshots being that they were for undoing manually made mistakes during SQL migrations etc. It worked well for me. I still use ZFS on my FreeBSD servers. But at the moment my projects are low traffic and the data only changes in important ways some rare times. So with my current personal servers I manually snapshot about once a week and manually trigger a backup of that from another server. Another thing I’ve changed is that now I only snapshot the parts of the file system where I store PostgreSQL databases and other application data. I no longer care so much about snapshotting the operating system data and such. If I have a serious hardware malfunction I will do a fresh install of the OS, and I have a log of what important config values are used and so on, that my backup scripts copy when I run them, without copying all of the other things.
- vasco 3y ago
  This sounds overkill even for production data, much less personal data, particularly the every minute and the fact you keep dailies forever. Unless you're a custodian of some secret society's files!
  - codetrotter 3y ago
    > sounds overkill But it wasn’t. It was very useful in fact.
- oefrha 3y ago
  Copy-on-write and cheap snapshots was quite special when ZFS was created. It’s hardly special in 2023, when every single non-vintage iPhone, iPad and Mac has that.
georgyo 3y ago
Not likely. The snapshot doesn't write much, and both SSDs and ZFS are copy on write. Which means the cost of writing after a snapshot is the same as before the snapshot. On the other hand context is missing. Both SSDs and ZFS don't like being full or even close to full. The working set was ~650GB, of the drive was 1TB, then those snapshots could have easily made the drive over 90% full. This could have made ZFS unhappy all by itself.
- chromakode 3y ago
  I agree that it was unlikely. The total size of all data and snapshots was 625 GiB on a 2 TB drive (which had seen less than 2 years of moderate use). It was a pretty unexpected failure.
- Xaiph_Rahci 3y ago
  > cost of writing after a snapshot is the same as before the snapshot I didn't understand this, could you please clarify? If there was no snapshot, there would be only one write operation, the actual write. However, with snapshot in place, in addition to actual write, there is a copy operation which copies the original data and writes to snapshot location. So, there should be two write operations (actual + copy).
  - boomboomsubban 3y ago
    There's no copy operation, the previous data isn't overwritten and the new data is written to a new block. It's "copy-on-write."
  - rincebrain 3y ago
    ZFS is never overwriting in place in either case, you're just not freeing the old one if it's in a snapshot, and a snapshot is just a note that "nickname this point in time 'mysnapshot', and don't clean up anything referenced at this point in time", so it's very cheap to make, and you just check it later when you would be cleaning things up.
    - viraptor 3y ago
xpe 3y ago
As I understand it, taking a snapshot with ZFS involves writing a metadata object and some data references. Assuming 100 GB of data, 128K block size, and 64 bit pointers, I'd guesstimate * that new data written during a snapshot would be in the ballpark of 5 MB. Is doing that 6 times per hour (52,560 times per year) enough to cause premature wear on the drive? That would be ~256 GB per year. This is likely under 1% of an SSD's write endurance. So, I'd be surprised if taking 10 minute snapshots was a significant causal factor. * I could be wrong, I asked for some help from not the most reliable sources. Happy to be corrected. Still, if my estimate is higher than actual and yet still unlikely to affect drive longevity, it may be moot.
endisneigh 3y ago
Of course it contributed. But it probably wasn’t the main reason or a significant contributor.
E39M5S62 3y ago
Nope. As others have mentioned, ZFS is CoW. Snapshots are "free" in that they (basically) point to a transaction group in the filesystem. They record a small amount of metadata to disk on each snapshot - on the order of a few MB. This is much much lower than an rclone/sync style backup.
- Izkata 3y ago
  That's also the least interesting part of comparing a ZFS backup to rsync/rclone: The rsync way is to crawl the entire tree being backed up to diff over the network then copy the differences. Because of snapshots, ZFS already knows all the changes that occurred between snapshot A and snapshot B, and (provided state up to A has already been backed up) can update the backup by pushing all changes between A and B as one big binary blob without having to scan or diff anything.
deleted 3y ago
[deleted]
numpad0 3y ago
Minimum write size of a modern Flash chip can be ~100MB(!) according to a comment found in a random orange website[1]. So 5MB write every 10 minutes can be 600MB/hr, which is 4.8TB/8-hr-day, which is 24TB/40-hour-week, which is 3.43 DWPD real time for a 1TB drive, and 2500 TBW in 2 years real time[2]. Official quoted specification for SN850 is 600 TBW of write endurance, likely after derating for obvious warranty implications. Incidentally, 2500TB is also a typical endurance figure for many SSDs in this market. Overall, to me, sounds not entirely impossible. I kind of wonder what's the controller says in SMART data, if still alive. On Linux the command is `apt install smartmontools; smartctl -s on /dev/sda; smartctl -A /dev/sda`, and it shall print out a table[4]. On Windows, just install CrystalDiskInfo[3]. 1: https://news.ycombinator.com/item?id=29165202 https://news.ycombinator.com/item?id=29165202 2: DWPD: drive writes per day, TBW: Total Bytes Written - in terabytes 3: https://crystalmark.info/en/software/crystaldiskinfo/ https://crystalmark.info/en/software/crystaldiskinfo/ 4: Note that "Pre-fail" means the value is supposed to change when about to fail and "Old_age" means the value is supposed to indicate age, NOT "this is bad and about to fail" and "this drive is old". It always says all Pre-fail and Old_age. Someone should have changed it to "somewhat_boolean" and "life_remain" long time ago in my opinion.
- chromakode 3y ago
  Unfortunately the drive didn't appear accessible at all via nvme-cli. Interestingly it shows up in lspci but doesn't get a /dev/nvme. It tends to hang the UEFIs of the two systems I tried it in when they try to read it.
- pseudalopex 3y ago
  Minimum write size is not erase block size.
- kalleboo 3y ago
  My machines have always been just constantly writing logs, like every couple seconds (macOS does this), and the write wear has never been anywhere near that bad. The advertised endurance must take into account write amplification for typical loads.
- mafuy 3y ago
p_l 3y ago
No. ZFS implements normal writes to the disk as snapshots (just unnamed ones), so in fact you can only write to disk through creation of a snapshot or by writing to "Intent Log" which is short-term log of data that is going into next snapshot - but which was synced before the snapshot was done, and as such it's secured in case of power failure.

riku_iki 3y ago

spoiler: he has 10 minutes incremental backups.

ggm 3y ago
Its "backups" join "zfs makes snapshots easy" join "snapshots make incremental backups easy" join "backups on device aren't a backup" join "I had off-device backups" which reduces to "I had backups" indeed. 3-2-1 forever!
- Dylan16807 3y ago
  > "backups on device aren't a backup" That wasn't part of the article. It was a single drive failure, so RAID would have done fine.
  - ggm 3y ago
    Yes, RAID will get you over some failures. But, it still isn't a backup. Backup is what gets you over corrupted RAID, loss of both sides of the mirror stripe, entire disk failure when its not RAID. What he does is run zrep to make a backup. it covers his needs. ZFS snapshot by itself is only transitionally a "backup" for the immediacy of change, it's the least safe form of backup if it remains on the same logical drive structure.
    - deleted 3y ago
      [deleted]
    - riku_iki 3y ago
      > Yes, RAID will get you over some failures. But, it still isn't a backup. backups also can fail, that zrep can start failing after some os/kernel update without notifying owner. The question is in probabilities of failures, I kinda would trust industrial raid more than some custom made hobby solution.

xpe 3y ago

Very quick summary: The mastodon thread refers to https://zrepl.github.io https://zrepl.github.io "zrepl is a one-stop, integrated solution for ZFS replication."

runeks 3y ago
Looks like it needs to speak to a daemon running on the storage server? Would be cool if it could just use e.g. S3 for storage.
istjohn 3y ago
Does anyone know how zrepl compares to sanoid/syncoid other than that zrepl is written in Go and sanoid/syncoid are Perl scripts?
- etherael 3y ago
  I use sanoid to do basically the same thing as this, and was interested in giving it a shot to see if it was more hands off but it's definitely a more complex setup to begin with, given you have to setup your own SSL certs etc, not sure why they wouldn't just use SSH transport for this like everything else.
  - chromakode 3y ago
    I use Wireguard to secure and authenticate the transport. Much easier to set up! SSH is also an option.
    - etherael 3y ago
      Thanks. Good to know that's possible, it's exactly what I use for sanoid also, so I guess the quickstart just assumes that layer isn't available.

ChrisMarshallNY 3y ago

I tend to use Apple’s Time Machine incremental backup to a Synology spinning rust server. I also have an external SSD that I’ll mirror the internal drive to, if I’ll be doing anything dodgy, or upgrading my machine. That works. TM restores can be quite slow, but almost all my important data is in Git (and hosted storage), so it’s not really been an issue. I just use TM every now and then, if I have a single file I want to backtrack. I also have one of the notorious[0] SanDisk drives. I don’t use it for anything important. It just has some game storage. Since I’m a Mac user, games aren’t really much of a factor for me, and I won’t cry, if they croak. [0] https://arstechnica.com/gadgets/2023/08/sandisk-extreme-ssds-are-worthless-multiple-lawsuits-against-wd-say/ https://arstechnica.com/gadgets/2023/08/sandisk-extreme-ssds...

op00to 3y ago
I’ve given up on Time Machine. It never seems to work past a month or two for me on my Synology w/ atalk etc.
- jedberg 3y ago
  It was always breaking on my Synology too. I ended up just attaching an 8TB spinning rust directly to the Mac and it's been flawless since. Time Machine really doesn't like using remote disks that aren't official Apple gear.
  - kalleboo 3y ago
    Even when I had an Apple Time Capsule, it would break about once a year. It's just a flakey system. Wish they'd add the equivalent of zsend to APFS instead of using the weird "gigantic sparse disk image with hard links in it" system
    - praseodym 3y ago
      When backing up to an APFS Time Machine volume it does work a bit like that, at least no hard links are used any more: - https://eclecticlight.co/2021/03/11/time-machine-to-apfs-understanding-backups/ https://eclecticlight.co/2021/03/11/time-machine-to-apfs-und... - https://eclecticlight.co/2021/04/16/time-machine-to-apfs-using-a-network-share/ https://eclecticlight.co/2021/04/16/time-machine-to-apfs-usi...

NoZebra120vClip 3y ago

Just before I exited the Linux world entirely, I was beginning to chip away at the iceberg known as btrfs, and it was fascinating. I saw so much promise in many of its features, for revolutionizing backups and organizing my disks and everything. Now btrfs isn't ZFS, but it has some feature parity and perhaps the "poor man's ZFS". It's also much more reasonable to run on certain OS, due to the licensing, packaging, and in-kernel status of ZFS being kind of weird. One memorable time I was encouraged to use ZFS was when I mentioned to the Linux User's Group that I'd had to pull the power cord to reboot my computer, and I was roundly scorned for this foolish maneuver. But you may change your mind about the wisdom of doing either one when you consider that the system in question was a Raspberry Pi. Heh.

KennyFromIT 3y ago
Alright, I'll bite... What led you to leaving the Linux world entirely? What can "the community" learn from your experience to make it better for others?
- NoZebra120vClip 3y ago
  Linux is a great fantastic experience, and I have no qualms or ill will about it. I simply had no use for it anymore, and I needed to simplify. I've said before, I'm not a sysadmin anymore, I don't tinker with systems, I need stuff to be operational and in production. I still love Linux and I'd use it for any given server or Raspi if that were part of my job. I do use it daily in my job, but to a very minimal extent.
  - theaiquestion 3y ago
    Did you switch to a mac or to a windows machine?
    - worthless-trash 3y ago
      They said "Operational and in production" :)

op00to 3y ago

I wonder how much he gained from entirely restoring the system versus simply reprovisioning (gasp, even manually reinstalling) and restoring needed files a will. I'm not sure there's a lot of value in snapshotting and restoring stuff in a lost ssd situation tthat's also available in mirrors across the world.

theossuary 3y ago
The biggest time save is in time spent recovering. It's so much faster to restore the entire system than to reinstall the OS, reconfigure the bootloader, resetup disk encryption, reconfigure user accounts, reinstall all software, manually reload configs, etc. Or put it more directly, full disk backups are a great way to get RTO down.
- op00to 3y ago
  In this case the author had to do arcane magick to restore his zfs snapshot. This wasn’t a routine raw dd restore.
  - chromakode 3y ago
    Agreed. It was a large time investment that happened to pay off. If I ever have to do it again I'll be much faster. I hope that with wider ZFS adoption some of the routine tasks will be automated better in the future. I see no reason why in a couple years this couldn't be a mature fire and forget user experience.
    - op00to 3y ago
      If it can be done manually, it can be automated and made more reliable!
chromakode 3y ago
I've reflected similarly after this exercise. I've lost data before, but it felt terrible to lose my context and working memory. While I make sure the most important stuff is in git, there's a bunch of momentum and working memory in my bash history and system configuration. It's also nice to not have to think very hard about a patchwork of backup plans. It's nice to get a fresh start every now and then, but not under duress. I was in the middle of a multi day project and was gonna lose time either way. It was real nice to boot back into a machine that felt like home.

predictabl3 3y ago

Zrepl is a big part of why I feel secure doing the digital nomad thing. A script, run nightlyish, opens a separate-headered LUKS-protected ZFS pool and then copies all snapshots over. That NVME enclosure lives in my "purse" that never leaves me sight/body. Between this and NixOS, I can provision a new identical laptop in about 10 minutes. I recently added off-site replication as well, so even if I get completely devastatingly mugged, there's still about zero chance of serious data loss. Zrepl is absolutely brilliant software. Easy to run with, but incredibly sophisticated and powerful if you need all the knobs. I can't praise it enough.

tmountain 3y ago
You should do a write up about how this works. It sounds very interesting.
- predictabl3 3y ago
  It's on my list, but... to be honest if you Google "separate header Luks", you'll find it's trivial to create a LUKS device with a detached header. Then the default ZRepl quick start will get you going with the basic pool-to-pool local replication. That will get you almost all the way there. :) I used their docs/guide to do the remote replication too, though it would make a good write-up as I could throw in how I use sops-nix for securing the Zrepl TLS bits for the remote scenario too...
  - LWIRVoltage 3y ago
    This does sound really cool- and like a way to ultimately set up a secure, not that complex backup method...
xpe 3y ago
Kudos. You're probably safer than most people who are only one theft, fire, flood, or other disast
- xpe 3y ago

kristopolous 3y ago

I just have rsync running in a cronjob. How is this significantly different? I imagine it is, but I don't know how.

kadoban 3y ago
Mostly different in terms of performance and wear on the drives. If you rsync over and over, it has to scan basically the whole filesystem for changes each time. Zfs snapshots don't. The snapshot is ~instant and the calculation of what to send has no need to examine any files. I don't think the performance and drive-lifetime hit of running rsync every 10 minutes would be good. Zfs should have an edge in terms of atomicity as well, but in practice I'm not sure how much that matters. I _think_ it does matter but isn't perfect (zfs can't trick applications into doing atomic writes if they're not already, but it won't have _another_ worse layer of breaking atomicity like rsync must).
- toast0 3y ago
  Depends what else the machines are doing, and how much ram, and the rsync settings. If you read all the files on both sides, every time and don't have more ram than disk, it's going to be a lot of work. If you're just looking at directory entries most of the time, there's a good chance that's all cached and it's no disk load, other than the small changes. I ageee with you though that atomicity is a big difference, if it matters, and in most cases, it probably doesn't. Personally, I've mostly stopped doing rsync backups in favor of zfs send, but I've still got one I need to get around to changing. Sanoid/syncoid is pretty decent for less effort snapshotting and syncing snapshots; but I haven't done anything with encrypted datasets. For most of my systems, I'd prefer recovery over security. For the one system in iffy hosting, it runs full disk encryption as a layer below zfs, so it's zfs sends are cleartext, too. (The hosting facility has given me other customer's unwiped disks; better for me to assume my disks won't be wiped)
  - kadoban 3y ago
    > If you read all the files on both sides, every time and don't have more ram than disk, it's going to be a lot of work. If you're just looking at directory entries most of the time, there's a good chance that's all cached and it's no disk load, other than smthe dmall changes. Yeah that's a good point. I know that rsync is _quite_ clever, but at least any incantations I've ever done it still hits the drives a good amount. I'd ballpark guess a couple of orders of magnitude better than just "cp -r" or something, but still a couple of orders of magnitude worse than zfs snapshots. Yeah you're 100% right it'll depend on bunch of variables though. > Sanoid/syncoid is pretty decent for less effor snapshotting and syncing snapshots; but I haven't done anything with encrypted datasets. I'm not sure I'd recommend it, but I use both directly on encrypted datasets. I have tested recovery a couple times and it works fine, but I've read some cautionary tales too. I _think_ they're all old issues?

xpe 3y ago

For those that don't know, there are many wonderful incremental backup solutions that don't require ZFS. * For one, I personally recommend Restic (https://restic.net https://restic.net) because of its deduplication. * People on macOS don't have ZFS, well... maybe they could? See https://github.com/spl/zfs-on-mac https://github.com/spl/zfs-on-mac

chromakode 3y ago
bupstash.io was my favored option other than ZFS. It's a beautiful and performant solution. Being filesystem agnostic is an advantage in many contexts. In the end I chose ZFS for the efficiency of snapshots (vs. a full disk scan) and atomicity. Both enable more frequent, smaller syncs, which is perfect for a laptop.
drexlspivey 3y ago
macOS Time machine does incremental backups, is there any other reason wou might need ZFS?

shruggedatlas 3y ago

How could I backup using incremental atomic snapshots on Windows?

toast0 3y ago
Look into the Windows Copy Shadow Service. Unfortunately, I haven't had luck with Open Source backup software that uses it (the shadow copy snapshot would fail, the error code would be no help, and finding no resources, I gave up), but commercial software I've used was great. When I was at a big corp, the commercial backup software whose name escapes me at the moment would litterally wait for files to be saved, then do an incremental backup. As of now, I'm using Veeam on my personal machines, and it runs an incremental backup nightly and saves to an smb share.
deadbeeves 3y ago
I've been using urbackup for years. It does disk image-based backups and/or file-based backups. Disk image-based backups are incremental at the block level, and file-based backups are incremental at the file level (so if a single byte of the file has changed, the entire file gets backed up). It uses the Volume Shadow Copy mechanism that the sibling comment mentioned to get atomicity and avoid file locking issues.

archo 3y ago

https://archive.is/VPEAP https://archive.is/VPEAP

SoftTalker 3y ago

tldr: my drive died, and I had a backup. zfs seems incidental to me. I could have a 10 minute cron job rsyncing changes from ext4 and been just as well off.

poisonborz 3y ago
Classic HN comment!
- inshadows 3y ago
  [dead]
bomewish 3y ago
Is this so?

KyleSanderson 3y ago

bcachefs is the near future for Linux here. https://bcachefs.org/ https://bcachefs.org/

brunoqc 3y ago
Any tldr about why we should be looking forward to this? Any cool things?
- KyleSanderson 3y ago
  Erasure Coding when it lands should be pretty solid. Until then per-directory data replicas is the killer feature for me (Music has 3, Documents has 5, Downloads has 1). Something to be very excited about with full compression and encryption.
  - brunoqc 3y ago
    Nice, thanks!
  - vladvasiliu 3y ago
    You can do that with ZFS at the cost of defining separate filesystems per directory. I don't use multiple replicas, but I use that to tailor my backups per directory. ~/documents is snapshotted and backed up on the regular, with long-lived snapshots. Code is snapshotted regularly, but snapshots don't live too long, and they're not shipped to a different drive. I don't care for ~/tmp so no snapshots. --- edit: the cost, besides having to actually create the file systems, is that moving data between them isn't instant.
    - philkrylov 3y ago
      > the cost, besides having to actually create the file systems, is that moving data between them isn't instant. No more the case after block cloning support goes production: https://github.com/openzfs/zfs/pull/13392 https://github.com/openzfs/zfs/pull/13392
3y ago

DavidSJ 3y ago

My snapshots are encrypted by the original computer (this is cool because the NAS can’t read them!). So I also needed to restore the encryption “wrapper key” to be able to use the backups. Not gonna lie, it was pretty terrifying until I had my first confirmation I could decrypt the data. Note: no bespoke backup method should be assumed functional unless you actually periodically check that you can restore data from it.

raisin_churn 3y ago
Or non-bespoke.
- justinjlynn 3y ago
  In short, backups aren't actually taken until they have been verifiably restored.
  - yomlica8 3y ago
    I always see this advice but...how do you even do that without having an entire additional set of disks to restore too? You can't restore to production obviously, as aside from the downtime if the test fails you've just destroyed your good copy and proved the other copy is also bad. About the best I can think of is restoring a small part of the set as a sample, which isn't really testing the whole thing.
- DavidSJ 3y ago
  I added that word because at least with non-bespoke backups, you have probably thousands of users each day testing restoration out of necessity, and word might get around if the method failed to restore. But nevertheless, one should still test even then.
totetsu 3y ago
I always meant to keep a copy of my LUKS header on separate disk, just in case, but..

ck2 3y ago

I really figured we'd have super easy hardware Raid1 in even consumer level PCs by now given how cheap drives are (and unreliable). My SSD boot drive makes me nervous as heck, constantly backing it up.

xet7 3y ago

With ZFS, he is better prepared than other WD and Sandisk SSD users. https://petapixel.com/2023/08/08/sandisk-portable-ssds-are-failing-so-frequently-we-can-no-longer-recommend-them/ https://petapixel.com/2023/08/08/sandisk-portable-ssds-are-f... https://www.theverge.com/22291828/sandisk-extreme-pro-portable-my-passport-failure-continued https://www.theverge.com/22291828/sandisk-extreme-pro-portab... https://news.ycombinator.com/item?id=37042587 https://news.ycombinator.com/item?id=37042587 https://www.theverge.com/23837513/western-digital-sandisk-ssd-corrupted-deleted-questions https://www.theverge.com/23837513/western-digital-sandisk-ss... https://news.ycombinator.com/item?id=37188736 https://news.ycombinator.com/item?id=37188736

anonuser123456 3y ago
I had two drives in my mirrored zpool die within 8 minutes of one another. Both HGST drives too. A very sad day. Thankfully I had been regularly zfs sending my contents to another site and lost very little data. ZFS is rad.
- anotherhue 3y ago
  > ZFS is rad. Typo: RAID
  - LargoLasskhyfv 3y ago
    I'd bet that was intentional, because https://www.urbandictionary.com/define.php?term=Rad https://www.urbandictionary.com/define.php?term=Rad
  - anonuser123456 3y ago
    You deserve all my upvotes :)
  - btown 3y ago
    Only 1/4 of data lost! Meaning still recoverable!
  - quickthrower2 3y ago

minimalist 3y ago

For people who don't want to use ZFS but are okay with LVM: wyng-backup (formerly sparsebak) https://github.com/tasket/wyng-backup https://github.com/tasket/wyng-backup

bomewish 3y ago

What's the equivalent setup for a Mac user?

justinclift 3y ago

> Not gonna lie, it was pretty terrifying until I had my first confirmation I could decrypt the data. Sounds like the process was a bit less tested and documented than optimal. For a home system or personal desktop that's not super unusual. You don't want to be working out your restore procedure on the fly for production servers though. ;)

chromakode 3y ago
For sure. I knew I had all the right ingredients to restore, and had kicked the tires 6 months back, but I initially copied over the wrong key. When it failed to load I had a sad moment until I realized my mistake. In such fail moments there's a flash of clarity where every process gap becomes blindingly obvious.

BearhatBeer 3y ago

I feel like a computer running ZFS and serving files, is fine. And it should itself be treated as a strage device with a full parallel backup even though this has cost. But your computer shouldn't run ZFS, that's for the big boys upstairs. Code's too big, it's too hungry.

not_your_vase 3y ago

A few years ago we had capacitor plague. Are we living now the storage plague? It's getting ridiculous that all storage is getting worse and worse. WD is making HDDs crappy with SMR, manufacturers says that 3 years operating time is already too much for SSDs and HDDs, and they don't joke. I just had a Kingston SSD (okay, that was like 8 years old) and a portable WD HDD (~2 years old) die just this year. The internet is full of problems lately about data loss and longevity issues. I remember 20 years ago HDDs were not meant for eternity either, but they definitely outlived the usefulness of the computer that they were bought with...

andromeduck 3y ago
Don't worry, your chips will start glitching after a few soon too. We're hitting scaling limits. Exponential growth is slowing.
- nicman23 3y ago
  that does not make any sense if you are talking about the controllers.
  - andromeduck 3y ago
    https://support.google.com/cloud/answer/10759085?hl=en https://support.google.com/cloud/answer/10759085?hl=en
kalleboo 3y ago
I've had a lot more luck with hard disks these days than I did 20 years ago. Remember 20 years ago was the era of the infamous IBM Deathstar drives where the magnetic coating would literally start sprinkling off the platters. Also the era of terrible terrible Maxtor drives that died in 1-2 years, which Seagate then bought and made their drives also unstable for a while. I ran a server with around 8 drives and had to keep replacing disks at the rate of about one per year. Meanwhile today I'm helping admin a ZFS server with 20+ drives and drives have about a 4-5 year lifespan. > but they definitely outlived the usefulness of the computer that they were bought with Computers were also much more quickly obsoleted back then. When today a 6 year old computer is totally useable, back then you really felt it if your machine was just 3 years old.

makach 3y ago

I read this piece as someone who just got lucky. He never tested it until he needed it. 10 minutes is good. Don’t let this story fool you to not take backups.

mijoharas 3y ago
> Don’t let this story fool you to not take backups. Do you mean don't let this story fool you to not test your backups? Because the whole point of the story is he was saved by having his backups. (Though you're right that he lucked out by having it work when he hadn't tested it)
chromakode 3y ago
I tested my backups about 6 months ago when I set up zrepl. When I mentioned it was scary until I could decrypt the data, that wasn't the whole story, actually: initially I restored the wrong wrapper key and it failed to load! It's also scary in general to go from 2 copies to only 1 copy of data. A friend and I have been planning to trade replicas but haven't set it up yet. There's definitely still room for improvement in my setup.

Helmut10001 3y ago

I wonder if zrepl could be run in WSL2 - would be nice to backup Windows computers as well using this approach. At the moment, I use Nextcloud to sync data to my server. It is a more selective approach and Nextcloud is, per se, not a backup solution because not all files can be backed up.. and live-sync is always a half-baked backup solution.

vladvasiliu 3y ago
How would that work? Zrepl ships ZFS snapshots. You could probably wrangle wsl2 to install its distro on zfs. If so, I see no reason why zrepl wouldn't work with the linux environment. But snapshotting the whole Windows drive? I don't think so. If you want similar features, I think ReFS comes close. AFAICT it's not supported as a boot drive.
- Helmut10001 3y ago
  I thought maybe if Windows is installed on a ZFS volume, and WSL2 on another, Zrep from Linux should be able to backup both, snapshots from the Linux and the Windows volume. But a quick search on Google reveals that Windows on ZFS is not a thing, yet.
  - vladvasiliu 3y ago
    You could flip that on its head, and run Windows in a VM on Linux on a ZFS volume. Depending on what you do on Windows and your particular hardware setup, this may or may not work well enough. I see people use GPU pass-through to play games on Windows VMs. You could probably pass through practically all devices (GPU, sound, network, keyboard, etc.) and this could work well-enough if you don't need the absolute last drop of performance from your CPU and drives. And since KVM supports nested virtualization, you could run WSL2 in the Windows VM. And if I'm not mistaken, the KVM agent in windows can be told to ask the guest OS to sync the drives, and some Windows applications [0] can even cooperate with this and flush their buffers to disk. You could signal this before creating the ZFS snapshot. [0] probably not most, but I think MSSQL does.

Kiro 3y ago

Am I the only one who doesn't have any important data? If I lost everything today I would just start afresh and move on.

dkh 3y ago
At all? Like, anywhere? Or do you just have data living on cloud services instead of locally?
- Kiro 3y ago
  At all. What important data do you have? I really can't think of anything that I would miss if everything disappeared.
  - Symbiote 3y ago
    Address book, photographs/video of people I care about and holidays, personal diary, hobby projects, old letters/emails, stored passwords, archived bank statements/contracts/insurance and other important documents. I think you are very unusual if you don't care about any of this.
    - ghaff 3y ago
      Most administrative stuff, especially if it's in digital form anyway, can probably be recovered without too much hassle if it's lost. But you're right that most people have a bunch of stuff (not all of which is admittedly digital) they wouldn't want to lose.
    - Kiro 3y ago
      Yes, I think it's unusual to not care about photographs/videos. I used to think that they were important and that I would want to revisit them some day. However, now when I'm old enough that I should it simply hasn't happened. They are just some files that I will never open.
unmole 3y ago
> Am I the only one who doesn't have any important data? Quite possibly.

brucewayne46 3y ago

[flagged]

sandreas 3y ago

For every ZFS fan, I can recommend zfs-auto-snapshot[1]. I use it on my proxmox server[2] to auto manage snapshots incl throwing away old ones. [1]: https://github.com/zfsonlinux/zfs-auto-snapshot https://github.com/zfsonlinux/zfs-auto-snapshot [2]: https://pilabor.com/series/proxmox/restore-virtual-machine-via-zfs-snapshot/ https://pilabor.com/series/proxmox/restore-virtual-machine-v...

nicman23 3y ago
i actually made 2 scripts to automate the sending and deletion of old snapshots along with one that calls auto snapshot on a vm rebooting / shutting down. I was thinking that they might be useful to other people.

copirate 3y ago

I've read that ZFS is less safe than other Linux filesystems if you don't use ECC RAM, because it assumes that there are no memory errors and therefore doesn't provide a tool to repair a filesystem corrupted by such errors. Is this true?

Modified3019 3y ago
It's not true. That's basically ancient forum myth, alongside the also incorrect "ZFS needs 1GB memory per TB of HDD" nonsense that has thankfully mostly died out finally. ZFS makes no additional assumptions when using ECC vs non-ECC memory. It is theoretically possibly to construct a scenario where evil ram does all the exactly right things needed fool ZFS and corrupt your filesystem. Any pearl clutching about this thing which has never happened somehow also ignores that every filesystem is going to get corrupted. In reality, while ECC memory is always nice to have, it's no more required than any other filesystem. Though personally now that amounts of +32gb are common, I generally prefer error correction/detection over ultimate speed these days. Though ironically ECC memory is actually really nice to overclock, because I can actually just check my logs and prove if my system is actually stable. There so many actual dangers to your data in comparison that it's laughable. The biggest one being you. Followed by hardware failure, malware, and genuine ZFS bugs. I'd stay far away from raw sends of encrypted datasets in ZFS for a while, there are edge cases that haven't been resolved yet. Edit Longer article saying the same thing: https://jrs-s.net/2015/02/03/will-zfs-and-non-ecc-ram-kill-your-data/ https://jrs-s.net/2015/02/03/will-zfs-and-non-ecc-ram-kill-y...

curt15 3y ago

Was he able to back up and restore his boot/ESP partition as well using ZFS, or did those need to be reprovisioned manually?

paldepind2 3y ago

I see that his NAS is using an external harddrive enclosure over USB. I'm really curious about how wise such a NAS setup is? In some ways it seems very attractive as you can just get a low-power SoC (like a Raspberry Pi or a NUC) and hook it up to the external drives. But there also seems to be many potential pitfalls. Like, how slow will a resilver be over the USB? Might it be unusable/dangerously slow? How reliable is the USB connection? Does it perform to spec or might it cause weird issues? Can you get SMART info over the USB connection? Other issues?

Terretta 3y ago
> NAS is using an external harddrive enclosure over USB. The N in NAS means "network" as in network attached storage so it's not a NAS. > really curious about how wise such a NAS setup is? For data use cases like this, USB 3 can be reasonably comparable to Thunderbolt 3, and that connection is generally faster than the media. This use case seems to be using the external device as a continuous external backup rather than as network attached storage, which is a great use of USB-C dongle SSD enclosures that are the same size or larger than the SSD inside the laptop. You effectively have mirroring as well, since you have both the internal SSD copy and the external SSD copy, in different makes and forms, unlikely to both fail.
- LgWoodenBadger 3y ago
  In the context of the grandparent, a "NAS" is a device, attached to the network, that provides storage. How the storage the NAS provides is connected is irrelevant. On a related note, I can't think of any drive that has a network interface instead of something like USB, Firewire, Thunderbolt, SCSI, IDE, etc. so how exactly would you define a NAS device?
- paldepind2 3y ago
  How I interpret the setup that OP has, is that he some computer (a PI, an old laptop, a NUC, etc.) which is connected to drives in an external drive bay through USB. This is a somewhat common setup and is definitely a NAS. > For data use cases like this, USB 3 can be reasonably comparable to Thunderbolt 3, and that connection is generally faster than the media. External HDD enclosures can often contain 4 drives. During a resliver all of these could be heavily accessed. I'm not sure how the single USB 3 connection fares in this scenario. In a normal desktop you'd have four separate SATA connections, and even then resilvering a large RAID setup can take quite some time.