5 ms·

Could snapshotting the filesystem every 10 minutes have contributed to its death?

by windows2020 3y ago

Filligree 3y ago
No. Individual snapshots are a matter of kilobytes; desktop environments write considerably more.
- istjohn 3y ago
  I read somewhere that snapshots are actually around 5 MB. Still not a lot, but a lot more than a few KB. A year's worth of hourly snapshots comes to over 40 GB just in snapshot overhead.
  - chromakode 3y ago
    Another factor to weigh in my case is this laptop probably spends at least 50% of its life suspended. The overhead should be measured in MB per hour of uptime.
HankB99 3y ago
Not likely. A snapshot just marks the most recently written block and prevents previous blocks from being altered. (More or less.) Since ZFS is copy on write, any changes to files will involve the same writes and some previously written data will not be deleted.
baby_souffle 3y ago
Nah, changes are COW so most snapshots are tiny.
abrookewood 3y ago
I guess it could contribute somewhat, but I don't think it is that much additional work: for every new write (since the last snapshot), there is one additional read as the data is sent. It isn't reading the whole file system, just the incremental data.
codetrotter 3y ago
ZFS is very special, and it is cheap to make snapshots with ZFS, because ZFS uses copy-on-write. Intuitively I would think that the amount of extra writes is pretty low, even if you snapshot very frequently. But scientific measurements would be nice. I used to do snapshots every minute, every hour and every day with ZFS on some servers I administered. I’d purge the minute snapshots after 60 minutes. And I had cron jobs on other machines to backup the hourly and daily snapshots. I had it set up so that hourly snapshots were kept for something like 72 hours. And the daily snapshots were kept forever. The idea with the every minute snapshots being that they were for undoing manually made mistakes during SQL migrations etc. It worked well for me. I still use ZFS on my FreeBSD servers. But at the moment my projects are low traffic and the data only changes in important ways some rare times. So with my current personal servers I manually snapshot about once a week and manually trigger a backup of that from another server. Another thing I’ve changed is that now I only snapshot the parts of the file system where I store PostgreSQL databases and other application data. I no longer care so much about snapshotting the operating system data and such. If I have a serious hardware malfunction I will do a fresh install of the OS, and I have a log of what important config values are used and so on, that my backup scripts copy when I run them, without copying all of the other things.
- vasco 3y ago
  This sounds overkill even for production data, much less personal data, particularly the every minute and the fact you keep dailies forever. Unless you're a custodian of some secret society's files!
  - codetrotter 3y ago
    > sounds overkill But it wasn’t. It was very useful in fact.
- oefrha 3y ago
  Copy-on-write and cheap snapshots was quite special when ZFS was created. It’s hardly special in 2023, when every single non-vintage iPhone, iPad and Mac has that.
georgyo 3y ago
Not likely. The snapshot doesn't write much, and both SSDs and ZFS are copy on write. Which means the cost of writing after a snapshot is the same as before the snapshot. On the other hand context is missing. Both SSDs and ZFS don't like being full or even close to full. The working set was ~650GB, of the drive was 1TB, then those snapshots could have easily made the drive over 90% full. This could have made ZFS unhappy all by itself.
- chromakode 3y ago
  I agree that it was unlikely. The total size of all data and snapshots was 625 GiB on a 2 TB drive (which had seen less than 2 years of moderate use). It was a pretty unexpected failure.
- Xaiph_Rahci 3y ago
  > cost of writing after a snapshot is the same as before the snapshot I didn't understand this, could you please clarify? If there was no snapshot, there would be only one write operation, the actual write. However, with snapshot in place, in addition to actual write, there is a copy operation which copies the original data and writes to snapshot location. So, there should be two write operations (actual + copy).
  - boomboomsubban 3y ago
    There's no copy operation, the previous data isn't overwritten and the new data is written to a new block. It's "copy-on-write."
  - rincebrain 3y ago
    ZFS is never overwriting in place in either case, you're just not freeing the old one if it's in a snapshot, and a snapshot is just a note that "nickname this point in time 'mysnapshot', and don't clean up anything referenced at this point in time", so it's very cheap to make, and you just check it later when you would be cleaning things up.
    - viraptor 3y ago
xpe 3y ago
As I understand it, taking a snapshot with ZFS involves writing a metadata object and some data references. Assuming 100 GB of data, 128K block size, and 64 bit pointers, I'd guesstimate * that new data written during a snapshot would be in the ballpark of 5 MB. Is doing that 6 times per hour (52,560 times per year) enough to cause premature wear on the drive? That would be ~256 GB per year. This is likely under 1% of an SSD's write endurance. So, I'd be surprised if taking 10 minute snapshots was a significant causal factor. * I could be wrong, I asked for some help from not the most reliable sources. Happy to be corrected. Still, if my estimate is higher than actual and yet still unlikely to affect drive longevity, it may be moot.
endisneigh 3y ago
Of course it contributed. But it probably wasn’t the main reason or a significant contributor.
E39M5S62 3y ago
Nope. As others have mentioned, ZFS is CoW. Snapshots are "free" in that they (basically) point to a transaction group in the filesystem. They record a small amount of metadata to disk on each snapshot - on the order of a few MB. This is much much lower than an rclone/sync style backup.
- Izkata 3y ago
  That's also the least interesting part of comparing a ZFS backup to rsync/rclone: The rsync way is to crawl the entire tree being backed up to diff over the network then copy the differences. Because of snapshots, ZFS already knows all the changes that occurred between snapshot A and snapshot B, and (provided state up to A has already been backed up) can update the backup by pushing all changes between A and B as one big binary blob without having to scan or diff anything.
deleted 3y ago
[deleted]
numpad0 3y ago
Minimum write size of a modern Flash chip can be ~100MB(!) according to a comment found in a random orange website[1]. So 5MB write every 10 minutes can be 600MB/hr, which is 4.8TB/8-hr-day, which is 24TB/40-hour-week, which is 3.43 DWPD real time for a 1TB drive, and 2500 TBW in 2 years real time[2]. Official quoted specification for SN850 is 600 TBW of write endurance, likely after derating for obvious warranty implications. Incidentally, 2500TB is also a typical endurance figure for many SSDs in this market. Overall, to me, sounds not entirely impossible. I kind of wonder what's the controller says in SMART data, if still alive. On Linux the command is `apt install smartmontools; smartctl -s on /dev/sda; smartctl -A /dev/sda`, and it shall print out a table[4]. On Windows, just install CrystalDiskInfo[3]. 1: https://news.ycombinator.com/item?id=29165202 https://news.ycombinator.com/item?id=29165202 2: DWPD: drive writes per day, TBW: Total Bytes Written - in terabytes 3: https://crystalmark.info/en/software/crystaldiskinfo/ https://crystalmark.info/en/software/crystaldiskinfo/ 4: Note that "Pre-fail" means the value is supposed to change when about to fail and "Old_age" means the value is supposed to indicate age, NOT "this is bad and about to fail" and "this drive is old". It always says all Pre-fail and Old_age. Someone should have changed it to "somewhat_boolean" and "life_remain" long time ago in my opinion.
- chromakode 3y ago
  Unfortunately the drive didn't appear accessible at all via nvme-cli. Interestingly it shows up in lspci but doesn't get a /dev/nvme. It tends to hang the UEFIs of the two systems I tried it in when they try to read it.
- pseudalopex 3y ago
  Minimum write size is not erase block size.
- kalleboo 3y ago
  My machines have always been just constantly writing logs, like every couple seconds (macOS does this), and the write wear has never been anywhere near that bad. The advertised endurance must take into account write amplification for typical loads.
- mafuy 3y ago
p_l 3y ago
No. ZFS implements normal writes to the disk as snapshots (just unnamed ones), so in fact you can only write to disk through creation of a snapshot or by writing to "Intent Log" which is short-term log of data that is going into next snapshot - but which was synced before the snapshot was done, and as such it's secured in case of power failure.

Could snapshotting the filesystem every 10 minutes have contributed to its death?

Filligree 3y ago

No. Individual snapshots are a matter of kilobytes; desktop environments write considerably more.

istjohn 3y ago
I read somewhere that snapshots are actually around 5 MB. Still not a lot, but a lot more than a few KB. A year's worth of hourly snapshots comes to over 40 GB just in snapshot overhead.
- chromakode 3y ago
  Another factor to weigh in my case is this laptop probably spends at least 50% of its life suspended. The overhead should be measured in MB per hour of uptime.

HankB99 3y ago

Not likely. A snapshot just marks the most recently written block and prevents previous blocks from being altered. (More or less.) Since ZFS is copy on write, any changes to files will involve the same writes and some previously written data will not be deleted.

baby_souffle 3y ago

Nah, changes are COW so most snapshots are tiny.

abrookewood 3y ago

I guess it could contribute somewhat, but I don't think it is that much additional work: for every new write (since the last snapshot), there is one additional read as the data is sent. It isn't reading the whole file system, just the incremental data.

codetrotter 3y ago

ZFS is very special, and it is cheap to make snapshots with ZFS, because ZFS uses copy-on-write. Intuitively I would think that the amount of extra writes is pretty low, even if you snapshot very frequently. But scientific measurements would be nice. I used to do snapshots every minute, every hour and every day with ZFS on some servers I administered. I’d purge the minute snapshots after 60 minutes. And I had cron jobs on other machines to backup the hourly and daily snapshots. I had it set up so that hourly snapshots were kept for something like 72 hours. And the daily snapshots were kept forever. The idea with the every minute snapshots being that they were for undoing manually made mistakes during SQL migrations etc. It worked well for me. I still use ZFS on my FreeBSD servers. But at the moment my projects are low traffic and the data only changes in important ways some rare times. So with my current personal servers I manually snapshot about once a week and manually trigger a backup of that from another server. Another thing I’ve changed is that now I only snapshot the parts of the file system where I store PostgreSQL databases and other application data. I no longer care so much about snapshotting the operating system data and such. If I have a serious hardware malfunction I will do a fresh install of the OS, and I have a log of what important config values are used and so on, that my backup scripts copy when I run them, without copying all of the other things.

vasco 3y ago
This sounds overkill even for production data, much less personal data, particularly the every minute and the fact you keep dailies forever. Unless you're a custodian of some secret society's files!
- codetrotter 3y ago
  > sounds overkill But it wasn’t. It was very useful in fact.
oefrha 3y ago
Copy-on-write and cheap snapshots was quite special when ZFS was created. It’s hardly special in 2023, when every single non-vintage iPhone, iPad and Mac has that.

georgyo 3y ago

Not likely. The snapshot doesn't write much, and both SSDs and ZFS are copy on write. Which means the cost of writing after a snapshot is the same as before the snapshot. On the other hand context is missing. Both SSDs and ZFS don't like being full or even close to full. The working set was ~650GB, of the drive was 1TB, then those snapshots could have easily made the drive over 90% full. This could have made ZFS unhappy all by itself.

chromakode 3y ago
I agree that it was unlikely. The total size of all data and snapshots was 625 GiB on a 2 TB drive (which had seen less than 2 years of moderate use). It was a pretty unexpected failure.
Xaiph_Rahci 3y ago
> cost of writing after a snapshot is the same as before the snapshot I didn't understand this, could you please clarify? If there was no snapshot, there would be only one write operation, the actual write. However, with snapshot in place, in addition to actual write, there is a copy operation which copies the original data and writes to snapshot location. So, there should be two write operations (actual + copy).
- boomboomsubban 3y ago
  There's no copy operation, the previous data isn't overwritten and the new data is written to a new block. It's "copy-on-write."
- rincebrain 3y ago
  ZFS is never overwriting in place in either case, you're just not freeing the old one if it's in a snapshot, and a snapshot is just a note that "nickname this point in time 'mysnapshot', and don't clean up anything referenced at this point in time", so it's very cheap to make, and you just check it later when you would be cleaning things up.
  - viraptor 3y ago

xpe 3y ago

As I understand it, taking a snapshot with ZFS involves writing a metadata object and some data references. Assuming 100 GB of data, 128K block size, and 64 bit pointers, I'd guesstimate * that new data written during a snapshot would be in the ballpark of 5 MB. Is doing that 6 times per hour (52,560 times per year) enough to cause premature wear on the drive? That would be ~256 GB per year. This is likely under 1% of an SSD's write endurance. So, I'd be surprised if taking 10 minute snapshots was a significant causal factor. * I could be wrong, I asked for some help from not the most reliable sources. Happy to be corrected. Still, if my estimate is higher than actual and yet still unlikely to affect drive longevity, it may be moot.

endisneigh 3y ago

Of course it contributed. But it probably wasn’t the main reason or a significant contributor.

E39M5S62 3y ago

Nope. As others have mentioned, ZFS is CoW. Snapshots are "free" in that they (basically) point to a transaction group in the filesystem. They record a small amount of metadata to disk on each snapshot - on the order of a few MB. This is much much lower than an rclone/sync style backup.

Izkata 3y ago
That's also the least interesting part of comparing a ZFS backup to rsync/rclone: The rsync way is to crawl the entire tree being backed up to diff over the network then copy the differences. Because of snapshots, ZFS already knows all the changes that occurred between snapshot A and snapshot B, and (provided state up to A has already been backed up) can update the backup by pushing all changes between A and B as one big binary blob without having to scan or diff anything.

deleted 3y ago

[deleted]

numpad0 3y ago

Minimum write size of a modern Flash chip can be ~100MB(!) according to a comment found in a random orange website[1]. So 5MB write every 10 minutes can be 600MB/hr, which is 4.8TB/8-hr-day, which is 24TB/40-hour-week, which is 3.43 DWPD real time for a 1TB drive, and 2500 TBW in 2 years real time[2]. Official quoted specification for SN850 is 600 TBW of write endurance, likely after derating for obvious warranty implications. Incidentally, 2500TB is also a typical endurance figure for many SSDs in this market. Overall, to me, sounds not entirely impossible. I kind of wonder what's the controller says in SMART data, if still alive. On Linux the command is `apt install smartmontools; smartctl -s on /dev/sda; smartctl -A /dev/sda`, and it shall print out a table[4]. On Windows, just install CrystalDiskInfo[3]. 1: https://news.ycombinator.com/item?id=29165202 https://news.ycombinator.com/item?id=29165202 2: DWPD: drive writes per day, TBW: Total Bytes Written - in terabytes 3: https://crystalmark.info/en/software/crystaldiskinfo/ https://crystalmark.info/en/software/crystaldiskinfo/ 4: Note that "Pre-fail" means the value is supposed to change when about to fail and "Old_age" means the value is supposed to indicate age, NOT "this is bad and about to fail" and "this drive is old". It always says all Pre-fail and Old_age. Someone should have changed it to "somewhat_boolean" and "life_remain" long time ago in my opinion.

chromakode 3y ago
Unfortunately the drive didn't appear accessible at all via nvme-cli. Interestingly it shows up in lspci but doesn't get a /dev/nvme. It tends to hang the UEFIs of the two systems I tried it in when they try to read it.
pseudalopex 3y ago
Minimum write size is not erase block size.
kalleboo 3y ago
My machines have always been just constantly writing logs, like every couple seconds (macOS does this), and the write wear has never been anywhere near that bad. The advertised endurance must take into account write amplification for typical loads.
mafuy 3y ago

p_l 3y ago

No. ZFS implements normal writes to the disk as snapshots (just unnamed ones), so in fact you can only write to disk through creation of a snapshot or by writing to "Intent Log" which is short-term log of data that is going into next snapshot - but which was synced before the snapshot was done, and as such it's secured in case of power failure.