12 ms·

What do you mean by "deadlocked state" for a filesystem?

pxx 3y ago
Where `rm`, or more technically unlink(2), fails due to ENOSPC, like in the article... Hilariously this failure case doesn't seem to be listed in the docs. https://developer.apple.com/library/archive/documentation/System/Conceptual/ManPages_iPhoneOS/man2/unlink.2.html https://developer.apple.com/library/archive/documentation/Sy...
aidenn0 3y ago
Some filesystems may require allocating metadata to delete a file. AFIK it's a non issue with traditional Berkeley-style systems, since metadata and data come from a separate pools. Notably ZFS has this problem.
- em-bee 3y ago
  btrfs has this problem too it seems. but there it is usually easy to add a usb stick to extend the filesystem and fix the problem. i find it really frustrating though. why not just reserve some space?
  - aidenn0 3y ago
    Yeah, with ZFS some will make an unused dataset with a small reservation (say 1G) that you can then shrink to delete files if the disk is full.
    - rincebrain 3y ago
      This hasn't been a problem you should be able to hit in ZFS in a long time. It reserves a percent of your pool's total space precisely to avoid having 0 actual free space and only allows using space from that amount if the operation is a net gain on free space.
      - p_l 3y ago
        Yeah, a situation where you pool gets suspended due to no space and you can't delete files is considered a bug by OpenZFS.
        rincebrain 3y ago
        I mean, the pool should never have gotten suspended by that, even before OpenZFS was forked; just ENOSPC on rm.
      - aidenn0 3y ago
        Oh, that's good to know. I hit it in the past, but it was long enough ago that ZFS still had format versions.
        rincebrain 3y ago
        Yeah, the whole dance around slop space, if I remember my archaeology, went in shortly after the fork.
      - MarkSweep 3y ago
        For more details about this slop space, see this comment: https://github.com/openzfs/zfs/blob/99741bde59d1d1df0963009bb624ddc105f7d8dc/module/zfs/spa_misc.c#L351 https://github.com/openzfs/zfs/blob/99741bde59d1d1df0963009b...
    - p_l 3y ago
      The recommended solution is to apply a quota on top-level dataset, but that's mainly for preventing fragmentation or runaway writes.
      - b112 3y ago
        I think the solution is to not use a filesystem that is broken in this way.
        p_l 3y ago
        Note that ZFS explicitly has safeguards against total failure. No filesystem will work well with near full state when it comes to fragmentation.
        b112 3y ago
        This is a whataboutism. Being unable to use the filesystem, due to space full, without arcane knowledge, is not the same as "not working well". This is a brokwn implementation.
        aidenn0 3y ago
        You're misunderstanding. See the sibling thread where p_l says that this problem has been resolved, and any further occurrence would be treated as a bug. Setting the quota is only done now to reduce fragmentation (ZFS's fragmentation avoidance requires sufficient free space to be effective).
        b112 3y ago
        No, I'm not. They said the "recommended solution" for this issue is to use a quota. They also said it was mainly used for other issues, such as fragmentation. In other words, this was stated as a fix for the file delete issue. How does this invalidate my comment, that this was a broken implementation? It doesn't matter if it will be fixed in the future, or was just fixed.
  - steve_rambo 3y ago
    btrfs does reserve some space for exactly this issue, although it might not always be enough. https://btrfs.readthedocs.io/en/latest/btrfs-filesystem.html https://btrfs.readthedocs.io/en/latest/btrfs-filesystem.html > GlobalReserve is an artificial and internal emergency space. It is used e.g. when the filesystem is full. Its total size is dynamic based on the filesystem size, usually not larger than 512MiB, used may fluctuate.
syncsynchalt 3y ago
Modern (well, post-ZFS) filesystems operate by moving the filesystem through state changes where data is not (immediately) destroyed, but older versions of the data are still available for various purposes. Similar to an ACID-compliant database, something like a backup or recovery process can still access older snapshots of the filesystem, for various values of "older" that might range from milliseconds to seconds to years. With that in mind, you can see how we get in a scenario where deleting a file will require a minor bit of storage for recordkeeping the old and new states, before it can actually free up the storage by releasing the old state. There is supposed to be an escape hatch for getting yourself out of a situation where there isn't even enough storage for this little bit of record keeping, but either the author didn't know whatever trick is needed or the filesystem code wasn't well-behaved in this area (it's a corner-case that isn't often tested).
- dataflow 3y ago
  It feels like insanity that the default configuration of any filesystem intended for laymen can fail to delete a file due to anything other than an I/O error. If you want to keep a snapshot, at least bypass it when disk space runs out? How many customers do the vendors think would prefer the alternative?!
  - kamray23 3y ago
    It's not really just keeping snapshots that is the issue, usually. It's just normal FS operation, meant to prevent data corruption if any of these actions is interrupted, as well as various space-saving measures. Some FSs link files together when saving mass data so that identical blocks between them are only stored once, which means any of those files can only be fully deleted when all of them are. Some FSs log actions onto disk before and after doing them so that they can be restarted if interrupted. Some FSs do genuinely keep files on disk if they're already referenced in a snapshot even if you delete them – this is one instance where a modal about the issue should probably pop up if disk space is low. And some OSes really really really want to move things to .Trash1000 or something else stupid instead of deleting them.
  - p_l 3y ago

pxx 3y ago

Where `rm`, or more technically unlink(2), fails due to ENOSPC, like in the article... Hilariously this failure case doesn't seem to be listed in the docs. https://developer.apple.com/library/archive/documentation/System/Conceptual/ManPages_iPhoneOS/man2/unlink.2.html https://developer.apple.com/library/archive/documentation/Sy...

aidenn0 3y ago

Some filesystems may require allocating metadata to delete a file. AFIK it's a non issue with traditional Berkeley-style systems, since metadata and data come from a separate pools. Notably ZFS has this problem.

em-bee 3y ago
btrfs has this problem too it seems. but there it is usually easy to add a usb stick to extend the filesystem and fix the problem. i find it really frustrating though. why not just reserve some space?
- aidenn0 3y ago
  Yeah, with ZFS some will make an unused dataset with a small reservation (say 1G) that you can then shrink to delete files if the disk is full.
  - rincebrain 3y ago
    This hasn't been a problem you should be able to hit in ZFS in a long time. It reserves a percent of your pool's total space precisely to avoid having 0 actual free space and only allows using space from that amount if the operation is a net gain on free space.
    - p_l 3y ago
      Yeah, a situation where you pool gets suspended due to no space and you can't delete files is considered a bug by OpenZFS.
      - rincebrain 3y ago
        I mean, the pool should never have gotten suspended by that, even before OpenZFS was forked; just ENOSPC on rm.
    - aidenn0 3y ago
      Oh, that's good to know. I hit it in the past, but it was long enough ago that ZFS still had format versions.
      - rincebrain 3y ago
        Yeah, the whole dance around slop space, if I remember my archaeology, went in shortly after the fork.
    - MarkSweep 3y ago
      For more details about this slop space, see this comment: https://github.com/openzfs/zfs/blob/99741bde59d1d1df0963009bb624ddc105f7d8dc/module/zfs/spa_misc.c#L351 https://github.com/openzfs/zfs/blob/99741bde59d1d1df0963009b...
  - p_l 3y ago
    The recommended solution is to apply a quota on top-level dataset, but that's mainly for preventing fragmentation or runaway writes.
    - b112 3y ago
      I think the solution is to not use a filesystem that is broken in this way.
      - p_l 3y ago
        Note that ZFS explicitly has safeguards against total failure. No filesystem will work well with near full state when it comes to fragmentation.
        b112 3y ago
        This is a whataboutism. Being unable to use the filesystem, due to space full, without arcane knowledge, is not the same as "not working well". This is a brokwn implementation.
        aidenn0 3y ago
        You're misunderstanding. See the sibling thread where p_l says that this problem has been resolved, and any further occurrence would be treated as a bug. Setting the quota is only done now to reduce fragmentation (ZFS's fragmentation avoidance requires sufficient free space to be effective).
        b112 3y ago
        No, I'm not. They said the "recommended solution" for this issue is to use a quota. They also said it was mainly used for other issues, such as fragmentation. In other words, this was stated as a fix for the file delete issue. How does this invalidate my comment, that this was a broken implementation? It doesn't matter if it will be fixed in the future, or was just fixed.
- steve_rambo 3y ago
  btrfs does reserve some space for exactly this issue, although it might not always be enough. https://btrfs.readthedocs.io/en/latest/btrfs-filesystem.html https://btrfs.readthedocs.io/en/latest/btrfs-filesystem.html > GlobalReserve is an artificial and internal emergency space. It is used e.g. when the filesystem is full. Its total size is dynamic based on the filesystem size, usually not larger than 512MiB, used may fluctuate.

syncsynchalt 3y ago

Modern (well, post-ZFS) filesystems operate by moving the filesystem through state changes where data is not (immediately) destroyed, but older versions of the data are still available for various purposes. Similar to an ACID-compliant database, something like a backup or recovery process can still access older snapshots of the filesystem, for various values of "older" that might range from milliseconds to seconds to years. With that in mind, you can see how we get in a scenario where deleting a file will require a minor bit of storage for recordkeeping the old and new states, before it can actually free up the storage by releasing the old state. There is supposed to be an escape hatch for getting yourself out of a situation where there isn't even enough storage for this little bit of record keeping, but either the author didn't know whatever trick is needed or the filesystem code wasn't well-behaved in this area (it's a corner-case that isn't often tested).

dataflow 3y ago
It feels like insanity that the default configuration of any filesystem intended for laymen can fail to delete a file due to anything other than an I/O error. If you want to keep a snapshot, at least bypass it when disk space runs out? How many customers do the vendors think would prefer the alternative?!
- kamray23 3y ago
  It's not really just keeping snapshots that is the issue, usually. It's just normal FS operation, meant to prevent data corruption if any of these actions is interrupted, as well as various space-saving measures. Some FSs link files together when saving mass data so that identical blocks between them are only stored once, which means any of those files can only be fully deleted when all of them are. Some FSs log actions onto disk before and after doing them so that they can be restarted if interrupted. Some FSs do genuinely keep files on disk if they're already referenced in a snapshot even if you delete them – this is one instance where a modal about the issue should probably pop up if disk space is low. And some OSes really really really want to move things to .Trash1000 or something else stupid instead of deleting them.
- p_l 3y ago