5 ms·

Allocator Hints for Btrfs

27 points by GalaxySnail 2y ago

ThatPlayer 2y ago
The article mentions bcache, but not bcachefs which you can also do this on. I also like bcachefs more for mixed drive type filesystems because of the bcache feature of using the SSDs as write caches for data that gets moved to the HDD in the background. Also read cache. This btrfs feature only supports having the data chunks prioritize the HDDs. Whether or not that matters depends on your use case of course: it's not worse than a RAID setup of only HDD. I'm using bcachefs on budget gaming PCs built from old spare parts, so having a small SSD and a bunch of HDDs for games is great.
- raj_hun 2y ago
  I like the idea behind bcache, sadly it was never mainlined in the upstream kernel, because the maintainer is generally uncooperative with the community and is building his own walled garden because btrfs was "Not Invented Here".
  - koverstreet 2y ago
    bcache and bcachefs are both mainlined - please don't spread this sort of FUD. I'm taking the experimental label off in 6.16, so it'd be nice to avoid the drama and have a smooth release :) (also, fun to see btrfs copying bcachefs)
  - homebrewer 2y ago
    I'm pretty sure Kent explained how btrfs's problems lie in its design and are fundamentally unfixable. It's got nothing to do with NIH.
    - kjs3 2y ago
      No. Kent explained in his transparently biased opinion why btrfs is fundamentally unfixable. Which include such gems as "they wrote it too fast to get it right" (opinion much?) and "it has more code than XFS" (FS that tries to do more has more code is bad?). Many people, including people who just might know a few things about filesystems, have addressed his claims, both in detail and in the large. Now, I am in absolutely no position to objectively evaluate the detailed claims of dueling filesystem gurus, but having read quite a bit, I do feel pretty confident in thinking "it's complicated" and "btrfs for all its issues certainly isn't the flaming shitshow Kent really, really wants everyone to believe it is".
      - koverstreet 2y ago
        If an opportunity arises to respond to this elsewhere, I will. I don't think I'll dump on btrfs in a btrfs thread, though :) Keep in mind though I wrote all that ~10 years ago, and it was very much an opinionated mission statement. bcachefs started out 15 years ago as a couple of us sitting around in the office drinking beers "you know, this looks like a really elegant basis for a filesystem, it's going to be way smaller and cleaner!" (talking utter shit, of course). But, somehow, I stuck with it...
        kjs3 2y ago
        That's nice. The point being a 10 year old, self-confessed 'opinionate mission statement' which has in the intervening years seen some notable pushback (and perhaps progress in the opposition) should not be trotted out as the grandparent did and presented as a case of cadit quaestio.
        koverstreet 2y ago
        Because? I don't see anything there that's fundamentally changed. Btrfs repair code is still quite weak, and "btrfs ate my data" stories have reduced somewhat, but I'm still seeing them come up, and the core architectural issues haven't changed. Josef was recently saying on lkml that they (the btrfs team, at least at facebook) have accepted that XFS will always be better in many scenarios, and to me that's an admission of failure - a COW filesystem should work just as well as a previous generation non-COW filesystem in nocow mode. A mission statement like that is fundamentally saying "these are my priorities, this is why we need to do better" and I stand by what I wrote, because taking on writing a new filesystem - the part of the operating system with the highest requirements for "this absolutely has to work" robustness - really is about having the right priorities. Gotta take your time your time to get things right, make sure the fundamentals are solid, before chasing all the features - this isn't an area where typical silicon valley "go big or go home, fake it till you make it" attitudes can work.
        kjs3 2y ago
Cumpiler69 2y ago
[dead]
jeroenhd 2y ago
Are these tweaks exclusively beneficial to mixed-media RAID setups, or do they also pose an advantage to single-device BTRFS setups? I know BTRFS is the default FS for a bunch of distros now, but I haven't heard much about it in the RAID space. Most people seem to use ZFS, despite the massive memory pressure it adds, because of fear of bugs and reports of filesystem corruption years ago. Perhaps someone who uses BTRFS for RAID can comment on how well BTRFS RAID works these days?
- ThatPlayer 2y ago
  RAID1 has been fine for years, but hardly anyone is running that and RAID5/6 are still considered unstable even by the devs: https://btrfs.readthedocs.io/en/latest/Status.html#block-group-profiles https://btrfs.readthedocs.io/en/latest/Status.html#block-gro... I ran into Btrfs RAID6 corruption myself about ten years ago. Though to btrfs's credit, I didn't lose any data: the filesystem only locked up read-only on me. But this time I'll take their word on unstable.
  - nisa 2y ago
    > RAID1 has been fine for years It's never been RAID1 it's the odness of the pid decides which disk to read from.
    - forza_user 2y ago
      There is a patch for this. Hopefully it will be available in mainline kernels soon. https://git.kernel.org/pub/scm/linux/kernel/git/kdave/linux.git/commit/?h=for-next&id=185fa5c7ac5add0f1a109a22a972ca99780ee49e https://git.kernel.org/pub/scm/linux/kernel/git/kdave/linux....
- bjoli 2y ago
  Btrfs raid 1 works great. I have 3 10tb drives in a raid1c2 for a total of 15tb of space. Btrfs raid1 allows you to use any combination of drives and sizes. If I would have one 12tb drive instead of one 10tb I would have had 16tb of usable space. For raid5/6 I dont use the built in raid, but mdraid. I stopped waiting for btrfs raid56 to stabilise.

27 points by GalaxySnail 2y ago

The article mentions bcache, but not bcachefs which you can also do this on. I also like bcachefs more for mixed drive type filesystems because of the bcache feature of using the SSDs as write caches for data that gets moved to the HDD in the background. Also read cache. This btrfs feature only supports having the data chunks prioritize the HDDs. Whether or not that matters depends on your use case of course: it's not worse than a RAID setup of only HDD. I'm using bcachefs on budget gaming PCs built from old spare parts, so having a small SSD and a bunch of HDDs for games is great.

raj_hun 2y ago
I like the idea behind bcache, sadly it was never mainlined in the upstream kernel, because the maintainer is generally uncooperative with the community and is building his own walled garden because btrfs was "Not Invented Here".
- koverstreet 2y ago
  bcache and bcachefs are both mainlined - please don't spread this sort of FUD. I'm taking the experimental label off in 6.16, so it'd be nice to avoid the drama and have a smooth release :) (also, fun to see btrfs copying bcachefs)
- homebrewer 2y ago
  I'm pretty sure Kent explained how btrfs's problems lie in its design and are fundamentally unfixable. It's got nothing to do with NIH.
  - kjs3 2y ago
    No. Kent explained in his transparently biased opinion why btrfs is fundamentally unfixable. Which include such gems as "they wrote it too fast to get it right" (opinion much?) and "it has more code than XFS" (FS that tries to do more has more code is bad?). Many people, including people who just might know a few things about filesystems, have addressed his claims, both in detail and in the large. Now, I am in absolutely no position to objectively evaluate the detailed claims of dueling filesystem gurus, but having read quite a bit, I do feel pretty confident in thinking "it's complicated" and "btrfs for all its issues certainly isn't the flaming shitshow Kent really, really wants everyone to believe it is".
    - koverstreet 2y ago
      If an opportunity arises to respond to this elsewhere, I will. I don't think I'll dump on btrfs in a btrfs thread, though :) Keep in mind though I wrote all that ~10 years ago, and it was very much an opinionated mission statement. bcachefs started out 15 years ago as a couple of us sitting around in the office drinking beers "you know, this looks like a really elegant basis for a filesystem, it's going to be way smaller and cleaner!" (talking utter shit, of course). But, somehow, I stuck with it...
      - kjs3 2y ago
        That's nice. The point being a 10 year old, self-confessed 'opinionate mission statement' which has in the intervening years seen some notable pushback (and perhaps progress in the opposition) should not be trotted out as the grandparent did and presented as a case of cadit quaestio.
        koverstreet 2y ago
        Because? I don't see anything there that's fundamentally changed. Btrfs repair code is still quite weak, and "btrfs ate my data" stories have reduced somewhat, but I'm still seeing them come up, and the core architectural issues haven't changed. Josef was recently saying on lkml that they (the btrfs team, at least at facebook) have accepted that XFS will always be better in many scenarios, and to me that's an admission of failure - a COW filesystem should work just as well as a previous generation non-COW filesystem in nocow mode. A mission statement like that is fundamentally saying "these are my priorities, this is why we need to do better" and I stand by what I wrote, because taking on writing a new filesystem - the part of the operating system with the highest requirements for "this absolutely has to work" robustness - really is about having the right priorities. Gotta take your time your time to get things right, make sure the fundamentals are solid, before chasing all the features - this isn't an area where typical silicon valley "go big or go home, fake it till you make it" attitudes can work.
        kjs3 2y ago

Cumpiler69 2y ago

[dead]

jeroenhd 2y ago

Are these tweaks exclusively beneficial to mixed-media RAID setups, or do they also pose an advantage to single-device BTRFS setups? I know BTRFS is the default FS for a bunch of distros now, but I haven't heard much about it in the RAID space. Most people seem to use ZFS, despite the massive memory pressure it adds, because of fear of bugs and reports of filesystem corruption years ago. Perhaps someone who uses BTRFS for RAID can comment on how well BTRFS RAID works these days?

ThatPlayer 2y ago
RAID1 has been fine for years, but hardly anyone is running that and RAID5/6 are still considered unstable even by the devs: https://btrfs.readthedocs.io/en/latest/Status.html#block-group-profiles https://btrfs.readthedocs.io/en/latest/Status.html#block-gro... I ran into Btrfs RAID6 corruption myself about ten years ago. Though to btrfs's credit, I didn't lose any data: the filesystem only locked up read-only on me. But this time I'll take their word on unstable.
- nisa 2y ago
  > RAID1 has been fine for years It's never been RAID1 it's the odness of the pid decides which disk to read from.
  - forza_user 2y ago
    There is a patch for this. Hopefully it will be available in mainline kernels soon. https://git.kernel.org/pub/scm/linux/kernel/git/kdave/linux.git/commit/?h=for-next&id=185fa5c7ac5add0f1a109a22a972ca99780ee49e https://git.kernel.org/pub/scm/linux/kernel/git/kdave/linux....
bjoli 2y ago
Btrfs raid 1 works great. I have 3 10tb drives in a raid1c2 for a total of 15tb of space. Btrfs raid1 allows you to use any combination of drives and sizes. If I would have one 12tb drive instead of one 10tb I would have had 16tb of usable space. For raid5/6 I dont use the built in raid, but mdraid. I stopped waiting for btrfs raid56 to stabilise.