7 ms·

Modern (well, post-ZFS) filesystems operate by moving the filesystem through state changes where data is not (immediately) destroyed, but older versions of the

Modern (well, post-ZFS) filesystems operate by moving the filesystem through state changes where data is not (immediately) destroyed, but older versions of the data are still available for various purposes. Similar to an ACID-compliant database, something like a backup or recovery process can still access older snapshots of the filesystem, for various values of "older" that might range from milliseconds to seconds to years. With that in mind, you can see how we get in a scenario where deleting a file will require a minor bit of storage for recordkeeping the old and new states, before it can actually free up the storage by releasing the old state. There is supposed to be an escape hatch for getting yourself out of a situation where there isn't even enough storage for this little bit of record keeping, but either the author didn't know whatever trick is needed or the filesystem code wasn't well-behaved in this area (it's a corner-case that isn't often tested).

dataflow 3y ago
It feels like insanity that the default configuration of any filesystem intended for laymen can fail to delete a file due to anything other than an I/O error. If you want to keep a snapshot, at least bypass it when disk space runs out? How many customers do the vendors think would prefer the alternative?!
- kamray23 3y ago
  It's not really just keeping snapshots that is the issue, usually. It's just normal FS operation, meant to prevent data corruption if any of these actions is interrupted, as well as various space-saving measures. Some FSs link files together when saving mass data so that identical blocks between them are only stored once, which means any of those files can only be fully deleted when all of them are. Some FSs log actions onto disk before and after doing them so that they can be restarted if interrupted. Some FSs do genuinely keep files on disk if they're already referenced in a snapshot even if you delete them – this is one instance where a modal about the issue should probably pop up if disk space is low. And some OSes really really really want to move things to .Trash1000 or something else stupid instead of deleting them.
- p_l 3y ago
  Pretty much by the time you get to 100% full on ZFS, the latency is going to get atrocious anyway, but from my understanding there are multiple steps (from simplest to worst case) that ZFS permits in case you do hit the error: 1. Just remove some files - ZFS will attempt to do the right thing 2. Remove old snapshots 3. Mount the drive from another system (so nothing tries writing to it), then remove some files, reboot back to normal 4. Use `zfs send` to copy the data you want to keep to another bigger drive temporarily, then either prune the data or if you already filtered out any old snapshots, zero the original pool and reload it by `zfs send` from before.
  - Shorel 3y ago
    Modern defrag seems very cumbersome xD
    - p_l 3y ago
      Defragmentation and ability to do it are not free. You can have cheap defrag but comparatively brittle filesystems by making things modifiable in place. You can have filesystem that has as its primary value "never lose your data", but in exchange defragmentation is expensive.
      - dataflow 3y ago
        I don't buy this? What does defragmentation have to do with snapshotting? Defragmentation is just a rearrangement of the underlying blocks. Wouldn't snapshots just get moved around?
        p_l 3y ago
        The problem is that you have to track down all pointers pointing to specific block. With snapshotting, especially with filesystems that can only write data through snapshots (like ZFS), blocks can be referred to by many pointers. It's similar to evaluating liveness of object in a GC, except you're now operating on possibly gigantic heap with very... pointer-ful objects, that you have to rewrite - which goes against core principle of ZFS which is data safety. You're doing essentially a huge history rewrite on something like git repo, with billions of small objects, and doing it safely means you have to rewrite every metadata block that in any way refers to given data block - and rewrite every metadata block pointing to those metadata blocks.
jrockway 3y ago
I'm most surprised by the lack of testing. Macs tend to ship with much smaller SSDs than other computers because that's how Apple makes money ($600 for 1.5TB of flash vs. $100/2TB if you buy an NVMe SSD), so I'd expect that people run out of space pretty frequently.
- callalex 3y ago
  And if you make the experience broken and frustrating people will throw the whole computer away and buy a new one since the storage can’t be upgraded.
  - kjkjadksj 3y ago
    Not to mention potentially paying for your cloud storage for life.
  - brokenmachine 3y ago
    Don't forget to make it so nothing they have works properly with any other brands devices so the next one they buy must also be Apple. Rinse and repeat.
werid 3y ago
i've filled up an zfs array to the point where i could not delete files. the trick is to truncate a large enough files, or enough small files, to zero. not sure if this is a universal shell trick, but worked on those i tried: "> filename"
- pdimitar 3y ago
  For reasons I am completely unwilling to research, just doing `> filename` has not worked for me in a while. Since then I memorized this: `cat /dev/null >! filename`, and it has worked on systems with zsh and bash.
  - alias_neo 3y ago
    "truncate -s0 filename" I believe "> filename" only works correctly if you're root (at least in my experience, if I remember correctly). EDIT: To remove <> from filename placeholder which might be confusing, and to put commands in quotes.
    - pdimitar 3y ago
      Oh yes, that one also worked everywhere I tried, thanks for reminding me.
      - alias_neo 3y ago
        Pleasure. It saved me just yesterday when I needed to truncate hundreds of gigabytes of Docker logs on a system that had been having some issues for a while but I didn't want to recreate containers. "truncate -s 0 /var/lib/docker/containers/**/*-json.log" Will truncate all of the json logs for all of the containers on the host to 0 bytes. Of course the system should have had logging configured better (rotation, limits, remote log) in the first place, but it isn't my system. EDIT: Missing double-star.*
  - matja 3y ago
    Simple to verify with strace -f bash -c "> file": openat(AT_FDCWD, "file", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 3 man 2 openat: O_TRUNC If the file already exists and is a regular file and the access mode allows writing (i.e., is O_RDWR or O_WRONLY) it will be truncated to length 0. ...

dataflow 3y ago

It feels like insanity that the default configuration of any filesystem intended for laymen can fail to delete a file due to anything other than an I/O error. If you want to keep a snapshot, at least bypass it when disk space runs out? How many customers do the vendors think would prefer the alternative?!

kamray23 3y ago
It's not really just keeping snapshots that is the issue, usually. It's just normal FS operation, meant to prevent data corruption if any of these actions is interrupted, as well as various space-saving measures. Some FSs link files together when saving mass data so that identical blocks between them are only stored once, which means any of those files can only be fully deleted when all of them are. Some FSs log actions onto disk before and after doing them so that they can be restarted if interrupted. Some FSs do genuinely keep files on disk if they're already referenced in a snapshot even if you delete them – this is one instance where a modal about the issue should probably pop up if disk space is low. And some OSes really really really want to move things to .Trash1000 or something else stupid instead of deleting them.
p_l 3y ago
Pretty much by the time you get to 100% full on ZFS, the latency is going to get atrocious anyway, but from my understanding there are multiple steps (from simplest to worst case) that ZFS permits in case you do hit the error: 1. Just remove some files - ZFS will attempt to do the right thing 2. Remove old snapshots 3. Mount the drive from another system (so nothing tries writing to it), then remove some files, reboot back to normal 4. Use `zfs send` to copy the data you want to keep to another bigger drive temporarily, then either prune the data or if you already filtered out any old snapshots, zero the original pool and reload it by `zfs send` from before.
- Shorel 3y ago
  Modern defrag seems very cumbersome xD
  - p_l 3y ago
    Defragmentation and ability to do it are not free. You can have cheap defrag but comparatively brittle filesystems by making things modifiable in place. You can have filesystem that has as its primary value "never lose your data", but in exchange defragmentation is expensive.
    - dataflow 3y ago
      I don't buy this? What does defragmentation have to do with snapshotting? Defragmentation is just a rearrangement of the underlying blocks. Wouldn't snapshots just get moved around?
      - p_l 3y ago
        The problem is that you have to track down all pointers pointing to specific block. With snapshotting, especially with filesystems that can only write data through snapshots (like ZFS), blocks can be referred to by many pointers. It's similar to evaluating liveness of object in a GC, except you're now operating on possibly gigantic heap with very... pointer-ful objects, that you have to rewrite - which goes against core principle of ZFS which is data safety. You're doing essentially a huge history rewrite on something like git repo, with billions of small objects, and doing it safely means you have to rewrite every metadata block that in any way refers to given data block - and rewrite every metadata block pointing to those metadata blocks.

jrockway 3y ago

I'm most surprised by the lack of testing. Macs tend to ship with much smaller SSDs than other computers because that's how Apple makes money ($600 for 1.5TB of flash vs. $100/2TB if you buy an NVMe SSD), so I'd expect that people run out of space pretty frequently.

callalex 3y ago
And if you make the experience broken and frustrating people will throw the whole computer away and buy a new one since the storage can’t be upgraded.
- kjkjadksj 3y ago
  Not to mention potentially paying for your cloud storage for life.
- brokenmachine 3y ago
  Don't forget to make it so nothing they have works properly with any other brands devices so the next one they buy must also be Apple. Rinse and repeat.

werid 3y ago

i've filled up an zfs array to the point where i could not delete files. the trick is to truncate a large enough files, or enough small files, to zero. not sure if this is a universal shell trick, but worked on those i tried: "> filename"

pdimitar 3y ago
For reasons I am completely unwilling to research, just doing `> filename` has not worked for me in a while. Since then I memorized this: `cat /dev/null >! filename`, and it has worked on systems with zsh and bash.
- alias_neo 3y ago
  "truncate -s0 filename" I believe "> filename" only works correctly if you're root (at least in my experience, if I remember correctly). EDIT: To remove <> from filename placeholder which might be confusing, and to put commands in quotes.
  - pdimitar 3y ago
    Oh yes, that one also worked everywhere I tried, thanks for reminding me.
    - alias_neo 3y ago
      Pleasure. It saved me just yesterday when I needed to truncate hundreds of gigabytes of Docker logs on a system that had been having some issues for a while but I didn't want to recreate containers. "truncate -s 0 /var/lib/docker/containers/**/*-json.log" Will truncate all of the json logs for all of the containers on the host to 0 bytes. Of course the system should have had logging configured better (rotation, limits, remote log) in the first place, but it isn't my system. EDIT: Missing double-star.*
- matja 3y ago
  Simple to verify with strace -f bash -c "> file": openat(AT_FDCWD, "file", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 3 man 2 openat: O_TRUNC If the file already exists and is a regular file and the access mode allows writing (i.e., is O_RDWR or O_WRONLY) it will be truncated to length 0. ...