5 ms·

Not likely. The snapshot doesn't write much, and both SSDs and ZFS are copy on write. Which means the cost of writing after a snapshot is the same as before th

Not likely. The snapshot doesn't write much, and both SSDs and ZFS are copy on write. Which means the cost of writing after a snapshot is the same as before the snapshot. On the other hand context is missing. Both SSDs and ZFS don't like being full or even close to full. The working set was ~650GB, of the drive was 1TB, then those snapshots could have easily made the drive over 90% full. This could have made ZFS unhappy all by itself.

chromakode 3y ago
I agree that it was unlikely. The total size of all data and snapshots was 625 GiB on a 2 TB drive (which had seen less than 2 years of moderate use). It was a pretty unexpected failure.
Xaiph_Rahci 3y ago
> cost of writing after a snapshot is the same as before the snapshot I didn't understand this, could you please clarify? If there was no snapshot, there would be only one write operation, the actual write. However, with snapshot in place, in addition to actual write, there is a copy operation which copies the original data and writes to snapshot location. So, there should be two write operations (actual + copy).
- boomboomsubban 3y ago
  There's no copy operation, the previous data isn't overwritten and the new data is written to a new block. It's "copy-on-write."
- rincebrain 3y ago
  ZFS is never overwriting in place in either case, you're just not freeing the old one if it's in a snapshot, and a snapshot is just a note that "nickname this point in time 'mysnapshot', and don't clean up anything referenced at this point in time", so it's very cheap to make, and you just check it later when you would be cleaning things up.
  - viraptor 3y ago
    Does it actually not update in place even for areas with a single reference? I haven't checked the source, but that sounds like fragmentation hell on spinning disks. That would absolutely kill the performance on zfs-hosted VM images / databases, which I didn't think actually happens... (Apart from the intent log, which sure, that's append only)
    - rincebrain 3y ago
      I promise you, it does not. ZFS really deeply assumes that, when a region is in use, it will not change until it's no longer in use anywhere, and it also won't reuse things you just freed for a certain number of txgs afterward to let you get away with having to roll back a couple txgs in case of dire problems without excitement. (Since having enough writes will cause more txgs to happen faster, this isn't an issue people run into with being unable to use newly free space in practice.) Also in practice, defining what "sequential" means with multiple disks in nontrivial topologies becomes...exciting anyway, and for writes, you only care that things are relatively, not absolutely, sequential for spinning media, and on reads, prefetch is going to notice you doing heavily sequential IO and queue things up anyway. (IMO) If you like, you could go check on your configurations, what the DVAs for the different data blocks in your VM images are - something like zdb -dbdbdbdbdbdb [dataset] [object id, which you can get from the "inode number" of the file, or if it's a zvol, I think it's always just 1 that all the data you think of as the "disk" goes in...] You'll almost certainly find that the regions that changed more than a couple txgs apart (the "birth=" value is the logical/physical txg the record was created) are mostly not remotely sequential. (Nit - the two exceptions that come to mind are, the uberblocks are basically a fixed position on disk relative to the disk's size, and a fixed size, and you get [fixed size]/[minimum allocation size] of them in a ring buffer, basically, before you overwrite the oldest one, and that happens by just overwriting it, since it's technically not in use any more, someone just might want to roll back to it in a "This Should Never Happen(tm)" case...or the newly added feature of corrective send/recv, to let you feed ZFS a send stream of an "intact" copy of something that had an uncorrectable data error and have it scribble over the mangled copy with the fixed one in-place, assuming it passes the checksums.)

chromakode 3y ago

I agree that it was unlikely. The total size of all data and snapshots was 625 GiB on a 2 TB drive (which had seen less than 2 years of moderate use). It was a pretty unexpected failure.

Xaiph_Rahci 3y ago

> cost of writing after a snapshot is the same as before the snapshot I didn't understand this, could you please clarify? If there was no snapshot, there would be only one write operation, the actual write. However, with snapshot in place, in addition to actual write, there is a copy operation which copies the original data and writes to snapshot location. So, there should be two write operations (actual + copy).

boomboomsubban 3y ago
There's no copy operation, the previous data isn't overwritten and the new data is written to a new block. It's "copy-on-write."
rincebrain 3y ago
ZFS is never overwriting in place in either case, you're just not freeing the old one if it's in a snapshot, and a snapshot is just a note that "nickname this point in time 'mysnapshot', and don't clean up anything referenced at this point in time", so it's very cheap to make, and you just check it later when you would be cleaning things up.
- viraptor 3y ago
  Does it actually not update in place even for areas with a single reference? I haven't checked the source, but that sounds like fragmentation hell on spinning disks. That would absolutely kill the performance on zfs-hosted VM images / databases, which I didn't think actually happens... (Apart from the intent log, which sure, that's append only)
  - rincebrain 3y ago
    I promise you, it does not. ZFS really deeply assumes that, when a region is in use, it will not change until it's no longer in use anywhere, and it also won't reuse things you just freed for a certain number of txgs afterward to let you get away with having to roll back a couple txgs in case of dire problems without excitement. (Since having enough writes will cause more txgs to happen faster, this isn't an issue people run into with being unable to use newly free space in practice.) Also in practice, defining what "sequential" means with multiple disks in nontrivial topologies becomes...exciting anyway, and for writes, you only care that things are relatively, not absolutely, sequential for spinning media, and on reads, prefetch is going to notice you doing heavily sequential IO and queue things up anyway. (IMO) If you like, you could go check on your configurations, what the DVAs for the different data blocks in your VM images are - something like zdb -dbdbdbdbdbdb [dataset] [object id, which you can get from the "inode number" of the file, or if it's a zvol, I think it's always just 1 that all the data you think of as the "disk" goes in...] You'll almost certainly find that the regions that changed more than a couple txgs apart (the "birth=" value is the logical/physical txg the record was created) are mostly not remotely sequential. (Nit - the two exceptions that come to mind are, the uberblocks are basically a fixed position on disk relative to the disk's size, and a fixed size, and you get [fixed size]/[minimum allocation size] of them in a ring buffer, basically, before you overwrite the oldest one, and that happens by just overwriting it, since it's technically not in use any more, someone just might want to roll back to it in a "This Should Never Happen(tm)" case...or the newly added feature of corrective send/recv, to let you feed ZFS a send stream of an "intact" copy of something that had an uncorrectable data error and have it scribble over the mangled copy with the fixed one in-place, assuming it passes the checksums.)