8 ms·

Maintaining sufficient free space with ZFS

124 points by DerekBickerton 4y ago

deleted 4y ago
[deleted]
mustache_kimono 4y ago
I mean 5-10 seconds for the free space to be updated doesn't sound extraordinary? With all the accounting ZFS has do, doesn't this sound like something you would want running asynchronously in the background? The blog makes a point (with references) of ZFS being transactional, but, as a related matter, it's also COW, one would assume finding/guaranteeing "free" space is much harder than it would be on a traditional filesystem.
- toast0 4y ago
  It's clearly not extraordinary, it's normal for ZFS. UFS2 on FreeBSD does is too, it also has snapshots, but isn't generally COW. The first time you delete something bit and it doesn't show up in df right away, it might be surprising though. Or this case where automation deleted more than expected because the author wasn't aware of the need to wait.
- rtp4me 4y ago
  I have seen the same behavior with BTRFS. We create nightly snapshots on our servers then prune older snapshots. Watching 'top' output shows a 5-8 second delay until the BTRFS cleanup agent starts. The agent could take many seconds (20?) to completely remove the old snapshots
  - deleted 4y ago
    [deleted]
- ilyt 4y ago
  > I mean 5-10 seconds for the free space to be updated doesn't sound extraordinary? With all the accounting ZFS has do, doesn't this sound like something you would want running asynchronously in the background? If system is busy, sure, batching requests helps performance, especially on spinning rust. In one that it isn't ? Why wait ? Then again checking space after every removed file (which I assume what the problem was here) instead of "generate list of files to remove, remove them, then go to sleep for an hour" is a bit suboptimal in the first place
londons_explore 4y ago
I have certainly written code which consists of "while (free disk space is less than X) {, delete oldest log file}". Such code would break on zfs. Is there even a better way to achieve "I want all spare space to be filled with logfiles, but I never want to run out of space."?
- deleted 4y ago
  [deleted]
- londons_explore 4y ago
  One could imagine an "opportunistic storage" flag on a file which means 'consider this file deletable if you ever need the space'. It would be especially good for caches of remote files or data that could be recalculated on demand. We already have lots of similar things for RAM - for example Javas SoftReference.
- puffoflogic 4y ago
  You can do the math yourself every loop instead of asking the OS. Ask how much free space is available once at the start, track how big each file is, delete. You can sleep and check free space again at the end as a sanity check; if the free space isn't going down nearly as much as you expected report it to the user and inhibit future script executions b/c it means there's something referencing extents and your script is about to go on a rampage.
- lathiat 4y ago
  It also would be broken by having snapshots, since, the space wouldn't actually be freed, so this space accounting is not the only possible issue.
lousken 4y ago
is 90% used space even good with ZFS these days? I always reserve 25% in order to keep fragmentation in check
- Filligree 4y ago
  It really depends on how much performance you want. And don’t forget that 10% of a disk is a lot more space than it used to be.
- magicalhippo 4y ago
  It's not just keeping fragmentation in check, above a threshold ZFS switches allocation strategy. The "nearly full" allocator is typically slower, IIRC because it tries to find better fitting free-space holes than the default allocator. I can't recall if the threshold is 80% or 90% right now, but around there.
  - rincebrain 4y ago
    It is, I believe, 96% full in the spacemap for it to resort to best-fit instead of first-fit. [1] [1] - https://github.com/openzfs/zfs/blob/eaaed26ffb3a14c0c98ce1e6e039e62327ab4447/module/zfs/metaslab.c#L166-L172 https://github.com/openzfs/zfs/blob/eaaed26ffb3a14c0c98ce1e6...
- rsync 4y ago
  "is 90% used space even good with ZFS these days?" Yes, you can run busy zpools with many different random access consumers at 90%. I wouldn't do it without a SLOG but it is really only at 92%, these days, that we start to see performance really degrade. YMMV.
  - rincebrain 4y ago
    In theory, the dedicated log metaslab[1] feature should help avoid that sort of problem in the same way, I think, and is in 2.1, with no slog required. I'd be curious if you're suggesting you're running something based on 2.1 and still finding you can run it fuller more reasonably. [1] - https://github.com/openzfs/zfs/commit/aa755b35493a2d27dbbc59f1527c6c93788b28f6 https://github.com/openzfs/zfs/commit/aa755b35493a2d27dbbc59...
jamesrom 4y ago
The proposed change to free-disk is a hack. Using --track-bytes-deleted on ZFS wont really track the bytes deleted, but just the size of the file that was deleted -- as the author points out, that may not be the number of bytes actually freed. For my money, "sleep 10 before checking free-space" is a more robust solution.
- Andys 4y ago
  Yes, and file_size != space_consumed is pretty much the main point of using ZFS!
- drpixie 4y ago
  It is a hack, but given the complexity and non-timeliness of what's going on behind the scenes, and the lack of control that the user has, perhaps a good hack is the right solution. My choice would be a cron job to run every few hours (or every day) that calculates space required and deletes logs files to that size. As long as the desired free space leaves an adequate margin, this would be robust and work in the presence of compressed files, snapshots, etc. "Sleep 10" should be fine, but it's also a hack, and the author would need to consider how long it takes for cleared space to become available on a heavily loaded system. I would expect that under severe load, the clear-up lag could be very long.
- mustache_kimono 4y ago
  > For my money, "sleep 10 before checking free-space" is a more robust solution. And running more often. 50GB(?) at a time can back up any filesystem. Not to mention, there is another option which perhaps better matches this use case. And author seems to be using the zfs quota property when the refquota property might better account for descendants and perhaps make a better initial calculation of what free space is available easier? ZFS is great but absolutely has tradeoffs. It makes really good choices. Maybe this is one?
- mnw21cam 4y ago
  The "sleep 10 seconds" solution isn't robust enough. Consider if you have created a snapshot on the filesystem. That snapshot has to keep hold of all the data, so when you delete files, the amount of free space doesn't change. A loop with "sleep 10 seconds" is likely to delete every single file in the filesystem in an attempt to get that desired amount of free space. A very common thing to do with ZFS is to have automatically rotating snapshots, so that you can go and retrieve a file that you accidentally deleted or changed, without having to go to the backups. In this case, the space is freed when the last snapshot containing those files is rotated away, which could be day or a week later, depending on how it is set up.
rincebrain 4y ago
The author appears to be mistaken about what's going on here, I think. It's not that ZFS takes a long time to update free space - it's that it actually doesn't necessarily free the space immediately. Deletes over a certain size (by default 20480 blocks, so 10/80 MB for 512b/4k sectors, respectively, I believe) dump the thing to be deleted (assuming it's to be freed, dedup refcounting/snapshots/etc might mean it's not actually freed) on an async queue that gets worked through in the background, rather than blocking rm on it. (Otherwise, if the files are smaller than that, I think it's as the author says, with bundling things into a txg and only updating once they're flushed out. But the example was ~40 files totalling several GB, so I think it's as I describe above.)
- tbrownaw 4y ago
  Does this queue have a way to check the age (timestamp, txg id, whatever) of the oldest thing in it?
  - toast0 4y ago
    There's zfs wait, which can wait on the delete queue. Otoh, files that are deleted and open will be on the delete queue until they're closed, so that queue doesn't have any guarantee of progress while the system remains running. A process to delete less important files to maintain free space that waited on the delete queue to drain before reassessing the situation would need something more sophisticated.
  - rincebrain 4y ago
    I'm not aware of one, but don't think it would be terribly hard to implement. I don't know of anyone who has, probably because it's not really a thing that's broken for anyone in a way where that data point would be interesting...that I know of. Could be wrong. Maybe I'll remember and go try it in a day or two, when I'm not at an event[1]... [1] - https://openzfs.org/wiki/OpenZFS_Developer_Summit_2022 https://openzfs.org/wiki/OpenZFS_Developer_Summit_2022
- cryptonector 4y ago
  Yes, there is this and then too the fact that it's hard to predict how much space you'll get back by deleting files because of snapshots and compression. Combine the two things and managing free space is hard for users.
tbrownaw 4y ago
A bit of searching suggests that finding the current txg (transaction group) id might be helpful? https://www.delphix.com/blog/delphix-engineering/zfs-fundamentals-transaction-groups https://www.delphix.com/blog/delphix-engineering/zfs-fundame... A bit more turns up `zdb -e mainpool -ul` (or maybe without the -e?) that includes a bunch of txg ids along with tons of other stuff. I'd think there ought to be a less noisy way to find that, but I don't know where to start looking.
OhNoNotAgain_99 4y ago
rsync 4y ago
FWIW, we (rsync.net) defrag our zpools. But, of course, there is no defrag for ZFS and so we do it the trailer-park way - we add free space to a zpool when adding a vdev, and then we 'zfs send' datasets to ourselves on that same zpool thus allowing ZFS to lay down those bytes in an efficient and orderly fashion ... as opposed to the inefficient way they were laid down over time. This works.
- fire 4y ago
  do you have more info on this? I'd love to read a blog post about it. Are there any performance implications to doing ( or not doing ) this?
  - rsync 4y ago
    We've started publishing "technical notes" - some of which are always ZFS related: https://rsync.net/resources/notes/ https://rsync.net/resources/notes/ ... we've not disussed this item yet but I am finalizing the most recent "notes" and will include it. If you don't have busy filesystems filling beyond ~90% (used to be 80%...) then I wouldn't ever worry about this. OTOH, if you have very busy filesystems with lots of different consumers and hundreds of millions (or billions) of inodes then a recent dataset might be scattered all over your zpool. There are absolutely performance ramifications to this. So you expand that zpool (you have to expand it for this to work) by adding a vdev, and then you 'zfs send' that dataset onto the same zpool and it will be laid down nice and orderly onto the new free space that was added to the zpool. That dataset will then be more performant. This is analogous to "defrag" as we think of it but, again, in a duct-taped-engineering kind of way.
    - kernelbugs 4y ago
      Is there a way to subscribe to these technical notes postings? I (and I assume others) would be quite interested in being alerted when a new one is posted!
      - rsync 4y ago
Proven 4y ago
compsciphd 4y ago
isn't the concept of deletes not being instantaneous a normal aspect of unix file systems? i.e. do an unlink a 1TB file that is in use in linux on ext4. file disappears immediately form namespace but space isn't freed up. close program keeping that file opened and space will slowly be reclaimed. similarly, due to unlink / ext4 semantics, if one did unlink that file without anyone keeping it open, the unlink() would block until the file was deleted, but I'm not sure that's really required by posix (i.e. the file could be opened by someone). Therefore I think one has to believe that the expectation on any file system is that unlink returning doesn't mean the space was freed up yet.
unethical_ban 4y ago
Late to the party. It seems to that #3 at the bottom, snapshots holding onto a file, is the biggest or most common culprit of free space not getting freed on delete. Having a good snapshot management strategy is critical if you use the feature at all. And if you're using it lazily, just know that you should delete old snapshots when going through a free-space exercise.

deleted 4y ago

[deleted]

I mean 5-10 seconds for the free space to be updated doesn't sound extraordinary? With all the accounting ZFS has do, doesn't this sound like something you would want running asynchronously in the background? The blog makes a point (with references) of ZFS being transactional, but, as a related matter, it's also COW, one would assume finding/guaranteeing "free" space is much harder than it would be on a traditional filesystem.

toast0 4y ago
It's clearly not extraordinary, it's normal for ZFS. UFS2 on FreeBSD does is too, it also has snapshots, but isn't generally COW. The first time you delete something bit and it doesn't show up in df right away, it might be surprising though. Or this case where automation deleted more than expected because the author wasn't aware of the need to wait.
rtp4me 4y ago
I have seen the same behavior with BTRFS. We create nightly snapshots on our servers then prune older snapshots. Watching 'top' output shows a 5-8 second delay until the BTRFS cleanup agent starts. The agent could take many seconds (20?) to completely remove the old snapshots
- deleted 4y ago
  [deleted]
ilyt 4y ago
> I mean 5-10 seconds for the free space to be updated doesn't sound extraordinary? With all the accounting ZFS has do, doesn't this sound like something you would want running asynchronously in the background? If system is busy, sure, batching requests helps performance, especially on spinning rust. In one that it isn't ? Why wait ? Then again checking space after every removed file (which I assume what the problem was here) instead of "generate list of files to remove, remove them, then go to sleep for an hour" is a bit suboptimal in the first place

londons_explore 4y ago

I have certainly written code which consists of "while (free disk space is less than X) {, delete oldest log file}". Such code would break on zfs. Is there even a better way to achieve "I want all spare space to be filled with logfiles, but I never want to run out of space."?

deleted 4y ago
[deleted]
londons_explore 4y ago
One could imagine an "opportunistic storage" flag on a file which means 'consider this file deletable if you ever need the space'. It would be especially good for caches of remote files or data that could be recalculated on demand. We already have lots of similar things for RAM - for example Javas SoftReference.
puffoflogic 4y ago
You can do the math yourself every loop instead of asking the OS. Ask how much free space is available once at the start, track how big each file is, delete. You can sleep and check free space again at the end as a sanity check; if the free space isn't going down nearly as much as you expected report it to the user and inhibit future script executions b/c it means there's something referencing extents and your script is about to go on a rampage.
lathiat 4y ago
It also would be broken by having snapshots, since, the space wouldn't actually be freed, so this space accounting is not the only possible issue.

lousken 4y ago

is 90% used space even good with ZFS these days? I always reserve 25% in order to keep fragmentation in check

Filligree 4y ago
It really depends on how much performance you want. And don’t forget that 10% of a disk is a lot more space than it used to be.
magicalhippo 4y ago
It's not just keeping fragmentation in check, above a threshold ZFS switches allocation strategy. The "nearly full" allocator is typically slower, IIRC because it tries to find better fitting free-space holes than the default allocator. I can't recall if the threshold is 80% or 90% right now, but around there.
- rincebrain 4y ago
  It is, I believe, 96% full in the spacemap for it to resort to best-fit instead of first-fit. [1] [1] - https://github.com/openzfs/zfs/blob/eaaed26ffb3a14c0c98ce1e6e039e62327ab4447/module/zfs/metaslab.c#L166-L172 https://github.com/openzfs/zfs/blob/eaaed26ffb3a14c0c98ce1e6...
rsync 4y ago
"is 90% used space even good with ZFS these days?" Yes, you can run busy zpools with many different random access consumers at 90%. I wouldn't do it without a SLOG but it is really only at 92%, these days, that we start to see performance really degrade. YMMV.
- rincebrain 4y ago
  In theory, the dedicated log metaslab[1] feature should help avoid that sort of problem in the same way, I think, and is in 2.1, with no slog required. I'd be curious if you're suggesting you're running something based on 2.1 and still finding you can run it fuller more reasonably. [1] - https://github.com/openzfs/zfs/commit/aa755b35493a2d27dbbc59f1527c6c93788b28f6 https://github.com/openzfs/zfs/commit/aa755b35493a2d27dbbc59...

jamesrom 4y ago

The proposed change to free-disk is a hack. Using --track-bytes-deleted on ZFS wont really track the bytes deleted, but just the size of the file that was deleted -- as the author points out, that may not be the number of bytes actually freed. For my money, "sleep 10 before checking free-space" is a more robust solution.

Andys 4y ago
Yes, and file_size != space_consumed is pretty much the main point of using ZFS!
drpixie 4y ago
It is a hack, but given the complexity and non-timeliness of what's going on behind the scenes, and the lack of control that the user has, perhaps a good hack is the right solution. My choice would be a cron job to run every few hours (or every day) that calculates space required and deletes logs files to that size. As long as the desired free space leaves an adequate margin, this would be robust and work in the presence of compressed files, snapshots, etc. "Sleep 10" should be fine, but it's also a hack, and the author would need to consider how long it takes for cleared space to become available on a heavily loaded system. I would expect that under severe load, the clear-up lag could be very long.
mustache_kimono 4y ago
> For my money, "sleep 10 before checking free-space" is a more robust solution. And running more often. 50GB(?) at a time can back up any filesystem. Not to mention, there is another option which perhaps better matches this use case. And author seems to be using the zfs quota property when the refquota property might better account for descendants and perhaps make a better initial calculation of what free space is available easier? ZFS is great but absolutely has tradeoffs. It makes really good choices. Maybe this is one?
mnw21cam 4y ago
The "sleep 10 seconds" solution isn't robust enough. Consider if you have created a snapshot on the filesystem. That snapshot has to keep hold of all the data, so when you delete files, the amount of free space doesn't change. A loop with "sleep 10 seconds" is likely to delete every single file in the filesystem in an attempt to get that desired amount of free space. A very common thing to do with ZFS is to have automatically rotating snapshots, so that you can go and retrieve a file that you accidentally deleted or changed, without having to go to the backups. In this case, the space is freed when the last snapshot containing those files is rotated away, which could be day or a week later, depending on how it is set up.

rincebrain 4y ago

The author appears to be mistaken about what's going on here, I think. It's not that ZFS takes a long time to update free space - it's that it actually doesn't necessarily free the space immediately. Deletes over a certain size (by default 20480 blocks, so 10/80 MB for 512b/4k sectors, respectively, I believe) dump the thing to be deleted (assuming it's to be freed, dedup refcounting/snapshots/etc might mean it's not actually freed) on an async queue that gets worked through in the background, rather than blocking rm on it. (Otherwise, if the files are smaller than that, I think it's as the author says, with bundling things into a txg and only updating once they're flushed out. But the example was ~40 files totalling several GB, so I think it's as I describe above.)

tbrownaw 4y ago
Does this queue have a way to check the age (timestamp, txg id, whatever) of the oldest thing in it?
- toast0 4y ago
  There's zfs wait, which can wait on the delete queue. Otoh, files that are deleted and open will be on the delete queue until they're closed, so that queue doesn't have any guarantee of progress while the system remains running. A process to delete less important files to maintain free space that waited on the delete queue to drain before reassessing the situation would need something more sophisticated.
- rincebrain 4y ago
  I'm not aware of one, but don't think it would be terribly hard to implement. I don't know of anyone who has, probably because it's not really a thing that's broken for anyone in a way where that data point would be interesting...that I know of. Could be wrong. Maybe I'll remember and go try it in a day or two, when I'm not at an event[1]... [1] - https://openzfs.org/wiki/OpenZFS_Developer_Summit_2022 https://openzfs.org/wiki/OpenZFS_Developer_Summit_2022
cryptonector 4y ago
Yes, there is this and then too the fact that it's hard to predict how much space you'll get back by deleting files because of snapshots and compression. Combine the two things and managing free space is hard for users.

tbrownaw 4y ago

A bit of searching suggests that finding the current txg (transaction group) id might be helpful? https://www.delphix.com/blog/delphix-engineering/zfs-fundamentals-transaction-groups https://www.delphix.com/blog/delphix-engineering/zfs-fundame... A bit more turns up `zdb -e mainpool -ul` (or maybe without the -e?) that includes a bunch of txg ids along with tons of other stuff. I'd think there ought to be a less noisy way to find that, but I don't know where to start looking.

OhNoNotAgain_99 4y ago

rsync 4y ago

FWIW, we (rsync.net) defrag our zpools. But, of course, there is no defrag for ZFS and so we do it the trailer-park way - we add free space to a zpool when adding a vdev, and then we 'zfs send' datasets to ourselves on that same zpool thus allowing ZFS to lay down those bytes in an efficient and orderly fashion ... as opposed to the inefficient way they were laid down over time. This works.

fire 4y ago
do you have more info on this? I'd love to read a blog post about it. Are there any performance implications to doing ( or not doing ) this?
- rsync 4y ago
  We've started publishing "technical notes" - some of which are always ZFS related: https://rsync.net/resources/notes/ https://rsync.net/resources/notes/ ... we've not disussed this item yet but I am finalizing the most recent "notes" and will include it. If you don't have busy filesystems filling beyond ~90% (used to be 80%...) then I wouldn't ever worry about this. OTOH, if you have very busy filesystems with lots of different consumers and hundreds of millions (or billions) of inodes then a recent dataset might be scattered all over your zpool. There are absolutely performance ramifications to this. So you expand that zpool (you have to expand it for this to work) by adding a vdev, and then you 'zfs send' that dataset onto the same zpool and it will be laid down nice and orderly onto the new free space that was added to the zpool. That dataset will then be more performant. This is analogous to "defrag" as we think of it but, again, in a duct-taped-engineering kind of way.
  - kernelbugs 4y ago
    Is there a way to subscribe to these technical notes postings? I (and I assume others) would be quite interested in being alerted when a new one is posted!
    - rsync 4y ago

Proven 4y ago

compsciphd 4y ago

isn't the concept of deletes not being instantaneous a normal aspect of unix file systems? i.e. do an unlink a 1TB file that is in use in linux on ext4. file disappears immediately form namespace but space isn't freed up. close program keeping that file opened and space will slowly be reclaimed. similarly, due to unlink / ext4 semantics, if one did unlink that file without anyone keeping it open, the unlink() would block until the file was deleted, but I'm not sure that's really required by posix (i.e. the file could be opened by someone). Therefore I think one has to believe that the expectation on any file system is that unlink returning doesn't mean the space was freed up yet.

unethical_ban 4y ago

Late to the party. It seems to that #3 at the bottom, snapshots holding onto a file, is the biggest or most common culprit of free space not getting freed on delete. Having a good snapshot management strategy is critical if you use the feature at all. And if you're using it lazily, just know that you should delete old snapshots when going through a free-space exercise.