3 ms·

Btrfs Allocator Hints

57 points by forza_user 1y ago

deleted 1y ago
[deleted]
forza_user 1y ago
I was surprised of the new attempt for performance profiles/device roles/hints when we already have a very good patch set maintained by kakra. - https://github.com/kakra/linux/pull/36 https://github.com/kakra/linux/pull/36 - https://wiki.tnonline.net/w/Btrfs/Allocator_Hints https://wiki.tnonline.net/w/Btrfs/Allocator_Hints What do you think?
- dontdoxxme 1y ago
  > One of the reasons why these patches are not included in the kernel is that the free space calculations do not work properly. It seems these patches possibly fix that.
hyperman1 1y ago
I feel a bit lost here. In the good old days, I ran ext2/ext3/ext4 and forgot about it, or Reiserfs if I felt fancy (and which was great until it wasn't). Now, there is a cambrian explosion going on. Ext4, xfs, btrfs,bcachefs, zfs. They each have their pros and cons, and it takes a while before you find out you run into an expensive limit. E.g. Ext3/4 is good, until it ran out of inodes. ZFS is good, but has only 1 password for full disk encryption and I want to store a second one with IT. According to the jungle drums, btrfs eats your data once in a while. Bcachefs stupidly tries to get itself rejected from Linux, not good for long term stability. I'm on XFS now, but let's see how that ends.
- swinglock 1y ago
  And XFS will at unexpected shutdowns sometimes leave you with files that previously contained data now being 0 bytes. I only really trust ZFS on Linux, but it's such a bother it can't be upstreamed and isn't fully integrated with the native Linux caching, as the native file systems are. Ext is fine too but it's missing features like checksumming and compression, and has limitations as you mentioned.
- rcxdude 1y ago
  That doesn't really match my recollection of timeline. I remember xfs being mentioned in the same sources contemporary with reiserfs (it predates ext3, even!). ZFS is about a decade newer, but not by much, and was probably the main reason most people would pay any real attention to their filesystem at that point, since it meaningfully added features not available in anything else at that point. BTRFS was basically a 'let's build the same thing, but in linux', but seems to have kinda stalled in terms of reliability (or at least in terms of reputation), and bcachefs is very much the new kid on the block, but seems to have a little bit more of a focus on getting to the reliability of ZFS, but it certainly is still not something to trust even as much as BTRFS. So it doesn't really feel like a cambrian explosion, more a new filesystem every ~5 years or so at a reasonably steady pace. (pretty much the 3 filesystems I think about ATM are ext4 as a standard boot drive, zfs for large, long-lived data storage, and FAT/exFAT for interoperability with windows. It'd have to be a pretty niche use-case for me to consider another option. BcacheFS sounds really interesting but only to experiment with right now)
- mdedetrich 1y ago
  FDE with ZFS is kind of fighting the way things are meant to be done with ZFS. ZFS allows encryption on a per dataset/zvol basis which is the officially recommended way to do encryption (see https://arstechnica.com/gadgets/2021/06/a-quick-start-guide-to-openzfs-native-encryption/ https://arstechnica.com/gadgets/2021/06/a-quick-start-guide-...)
bjoli 1y ago
I wonder if I can use a smaller SSD for this and make it avoid HDD wakeups due to some process reading metadata. That alone would make me love this feature.
- bionade24 1y ago
  Most likely yes, but the also envisioned periodically repacking oft multiple small data extents into one big that gets written to the HDD would wake up the HDD. And if you'd make the SSD "metadata only", browser cache and logging will keep the HDD spinning. This feature is for performance, not the case you described.
  - bjoli 1y ago
    The disk is unused except for once a day things get backed up to it (and other places, of course). Nothing will get written to it except for when it is getting written to for the backup. I will definitely try this.
- the8472 1y ago
  I think you'd rather want a cache device (or some more complicated storage tiering) for that so that both metadata and frequently accessed files get moved to that dynamically based on access patterns. Afaik btrfs doesn't support that. LVM, bcache, device mapper, bcachefs and zfs support that (though zfs would require separate caches for reading and synchronous write). And idk which of these let you control the writeback interval.
  - viraptor 1y ago
    Bcache allows lots of writeback configuration, including intervals https://www.kernel.org/doc/html/latest/admin-guide/bcache.html https://www.kernel.org/doc/html/latest/admin-guide/bcache.ht...
- ajross 1y ago
  Just buy more RAM and you get that for free. Really I guess that's my sense of patches like this in general: while sure, filesystem research has a long and storied history and it's a very hard problem in general that attracts some of the smartest people in the field to do genius-tier work... Does it really matter in the modern world where a vanilla two-socket rack unit has a terabyte of DRAM? Everything at scale happens in RAM these days. Everything. Replicating across datacenters gets you all the reliability you need, with none of the fussing about storage latency and block device I/O strategy.
sandreas 1y ago
Well, first of all: I'm not trying to bash BTRFS at all, it probably is just not meant for me. However, I'm trying to gain information it is really considered stable (like rock solid) or it might have been a hardware Problem on my system. I used cryptsetup with BTRFS because I encrypt all of my stuff. One day, the system froze and after reboot the partition was unrecoverably gone (the whole story[1]). Not a real problem because I had a recent backup, but somehow I lost trust in BTRFS that day. Anyone experienced something like that? Since then I switched to ZFS (on the same hardware) and never had problems - while it was a real pain to setup until I finished my script [2], which still is kind of a collection of dirty hacks :-) 1: https://forum.cgsecurity.org/phpBB3/viewtopic.php?t=13013 https://forum.cgsecurity.org/phpBB3/viewtopic.php?t=13013 2: https://github.com/sandreas/zarch https://github.com/sandreas/zarch
- ghostly_s 1y ago
  Yes, my story with btrfs is quite similar- used it for a couple years, suddenly threw some undocumented error and refused to mount, asked about it on the dev irc channel and was told apparently it was a known issue with no solution, have fun rebuilding from backups. No suggestion that anyone was interested in documenting this issue, let alone fixing it. These same people are the only ones in the world suggesting btrfs is "basically" stable. I'll never touch this project again with a ten foot pole, afaic it's run by children. I'll trust adults with my data.
  - amy214 1y ago
    I ran it on opensuse and it would 100% lock out a core on some sort of crontab tree structure rebalancing (?) .. I mean.. hello? online algorithm? Dynamic rebalancing? scheduled FS restructuring, really? ReiserFS dancing trees from 20 years ago? If thats how they think, "meh, the user just has to deal", no wonder this is how they handle bugs.
  - sandreas 1y ago
    Ok, thank you. At least I'm not alone with this. However, I'm not too much into it and would not go as far to say it's not a recommendable project, but boy was I mad it just died without any way to recover ANYTHING :-)

57 points by forza_user 1y ago

deleted 1y ago

[deleted]

I was surprised of the new attempt for performance profiles/device roles/hints when we already have a very good patch set maintained by kakra. - https://github.com/kakra/linux/pull/36 https://github.com/kakra/linux/pull/36 - https://wiki.tnonline.net/w/Btrfs/Allocator_Hints https://wiki.tnonline.net/w/Btrfs/Allocator_Hints What do you think?

dontdoxxme 1y ago
> One of the reasons why these patches are not included in the kernel is that the free space calculations do not work properly. It seems these patches possibly fix that.

hyperman1 1y ago

I feel a bit lost here. In the good old days, I ran ext2/ext3/ext4 and forgot about it, or Reiserfs if I felt fancy (and which was great until it wasn't). Now, there is a cambrian explosion going on. Ext4, xfs, btrfs,bcachefs, zfs. They each have their pros and cons, and it takes a while before you find out you run into an expensive limit. E.g. Ext3/4 is good, until it ran out of inodes. ZFS is good, but has only 1 password for full disk encryption and I want to store a second one with IT. According to the jungle drums, btrfs eats your data once in a while. Bcachefs stupidly tries to get itself rejected from Linux, not good for long term stability. I'm on XFS now, but let's see how that ends.

swinglock 1y ago
And XFS will at unexpected shutdowns sometimes leave you with files that previously contained data now being 0 bytes. I only really trust ZFS on Linux, but it's such a bother it can't be upstreamed and isn't fully integrated with the native Linux caching, as the native file systems are. Ext is fine too but it's missing features like checksumming and compression, and has limitations as you mentioned.
rcxdude 1y ago
That doesn't really match my recollection of timeline. I remember xfs being mentioned in the same sources contemporary with reiserfs (it predates ext3, even!). ZFS is about a decade newer, but not by much, and was probably the main reason most people would pay any real attention to their filesystem at that point, since it meaningfully added features not available in anything else at that point. BTRFS was basically a 'let's build the same thing, but in linux', but seems to have kinda stalled in terms of reliability (or at least in terms of reputation), and bcachefs is very much the new kid on the block, but seems to have a little bit more of a focus on getting to the reliability of ZFS, but it certainly is still not something to trust even as much as BTRFS. So it doesn't really feel like a cambrian explosion, more a new filesystem every ~5 years or so at a reasonably steady pace. (pretty much the 3 filesystems I think about ATM are ext4 as a standard boot drive, zfs for large, long-lived data storage, and FAT/exFAT for interoperability with windows. It'd have to be a pretty niche use-case for me to consider another option. BcacheFS sounds really interesting but only to experiment with right now)
mdedetrich 1y ago
FDE with ZFS is kind of fighting the way things are meant to be done with ZFS. ZFS allows encryption on a per dataset/zvol basis which is the officially recommended way to do encryption (see https://arstechnica.com/gadgets/2021/06/a-quick-start-guide-to-openzfs-native-encryption/ https://arstechnica.com/gadgets/2021/06/a-quick-start-guide-...)

bjoli 1y ago

I wonder if I can use a smaller SSD for this and make it avoid HDD wakeups due to some process reading metadata. That alone would make me love this feature.

bionade24 1y ago
Most likely yes, but the also envisioned periodically repacking oft multiple small data extents into one big that gets written to the HDD would wake up the HDD. And if you'd make the SSD "metadata only", browser cache and logging will keep the HDD spinning. This feature is for performance, not the case you described.
- bjoli 1y ago
  The disk is unused except for once a day things get backed up to it (and other places, of course). Nothing will get written to it except for when it is getting written to for the backup. I will definitely try this.
the8472 1y ago
I think you'd rather want a cache device (or some more complicated storage tiering) for that so that both metadata and frequently accessed files get moved to that dynamically based on access patterns. Afaik btrfs doesn't support that. LVM, bcache, device mapper, bcachefs and zfs support that (though zfs would require separate caches for reading and synchronous write). And idk which of these let you control the writeback interval.
- viraptor 1y ago
  Bcache allows lots of writeback configuration, including intervals https://www.kernel.org/doc/html/latest/admin-guide/bcache.html https://www.kernel.org/doc/html/latest/admin-guide/bcache.ht...
ajross 1y ago
Just buy more RAM and you get that for free. Really I guess that's my sense of patches like this in general: while sure, filesystem research has a long and storied history and it's a very hard problem in general that attracts some of the smartest people in the field to do genius-tier work... Does it really matter in the modern world where a vanilla two-socket rack unit has a terabyte of DRAM? Everything at scale happens in RAM these days. Everything. Replicating across datacenters gets you all the reliability you need, with none of the fussing about storage latency and block device I/O strategy.

sandreas 1y ago

Well, first of all: I'm not trying to bash BTRFS at all, it probably is just not meant for me. However, I'm trying to gain information it is really considered stable (like rock solid) or it might have been a hardware Problem on my system. I used cryptsetup with BTRFS because I encrypt all of my stuff. One day, the system froze and after reboot the partition was unrecoverably gone (the whole story[1]). Not a real problem because I had a recent backup, but somehow I lost trust in BTRFS that day. Anyone experienced something like that? Since then I switched to ZFS (on the same hardware) and never had problems - while it was a real pain to setup until I finished my script [2], which still is kind of a collection of dirty hacks :-) 1: https://forum.cgsecurity.org/phpBB3/viewtopic.php?t=13013 https://forum.cgsecurity.org/phpBB3/viewtopic.php?t=13013 2: https://github.com/sandreas/zarch https://github.com/sandreas/zarch

ghostly_s 1y ago
Yes, my story with btrfs is quite similar- used it for a couple years, suddenly threw some undocumented error and refused to mount, asked about it on the dev irc channel and was told apparently it was a known issue with no solution, have fun rebuilding from backups. No suggestion that anyone was interested in documenting this issue, let alone fixing it. These same people are the only ones in the world suggesting btrfs is "basically" stable. I'll never touch this project again with a ten foot pole, afaic it's run by children. I'll trust adults with my data.
- amy214 1y ago
  I ran it on opensuse and it would 100% lock out a core on some sort of crontab tree structure rebalancing (?) .. I mean.. hello? online algorithm? Dynamic rebalancing? scheduled FS restructuring, really? ReiserFS dancing trees from 20 years ago? If thats how they think, "meh, the user just has to deal", no wonder this is how they handle bugs.
- sandreas 1y ago
  Ok, thank you. At least I'm not alone with this. However, I'm not too much into it and would not go as far to say it's not a recommendable project, but boy was I mad it just died without any way to recover ANYTHING :-)