9 ms·

Oh! This is very exciting. Bcachefs could be the next gen filesystem that Linux needs[1]. Advantages over other filesystems: * ext4 or xfs — these two don't u

by orra 3y ago

Oh! This is very exciting. Bcachefs could be the next gen filesystem that Linux needs[1]. Advantages over other filesystems: * ext4 or xfs — these two don't use ECC to protect your data, only the filesystem metadata * zfs — zfs is technically great, but binary distribution of the zfs code is tricky, because the CDDL is GPL incompatible * btrfs — btrfs still doesn't have reliable RAID5 [1] It's been in development for a number of years. It now being proposed for inclusion in the mainline kernel is a major milestone.

sedatk 3y ago
> btrfs still doesn't have reliable RAID5 Synology offers btrfs + RAID5 without warning the user. I wonder why they’re so confident with it.
- bestham 3y ago
  They are running brtrfs on top of DM. https://kb.synology.com/en-nz/DSM/tutorial/What_was_the_RAID_implementation_for_Btrfs_File_System_on_SynologyNAS https://kb.synology.com/en-nz/DSM/tutorial/What_was_the_RAID...
  - sedatk 3y ago
    Thanks for the link!
- sporkle-feet 3y ago
  Synology doesn't use the btrfs raid - AIUI they layer non-raid btrfs over raid LVM
  - sedatk 3y ago
    Thanks, I wonder what the advantage of using FS-based RAID is.
vladvasiliu 3y ago
> * zfs — zfs is technically great, but binary distribution of the zfs code is tricky, because the CDDL is GPL incompatible Building your own ZFS module is easy enough, for example on Arch with zfs-dkms. But there's also the issue of compatibility. Sometimes kernel updates will break ZFS. Even minor ones, 6.2.13 IIRC broke it, whereas 6.2.12 was fine. Right now, 6.3 seems to introduce major compatibility problems. --- edit: looking through the openzfs issues, I was likely thinking of 6.2.8 breaking it, where 6.2.7 was fine. Point stands, though. https://github.com/openzfs/zfs/issues/14658 https://github.com/openzfs/zfs/issues/14658 Regarding 6.3 support, it apparently is merged in the master branch, but no release as of yet. https://github.com/openzfs/zfs/issues/14622 https://github.com/openzfs/zfs/issues/14622
- bjoli 3y ago
  How is the legal situation of doing that? If I had a company I wouldn't want to get in trouble with any litigious companies.
  - vladvasiliu 3y ago
    The ArchZFS project distributes binary kernel images with ZFS integrated. I don't know what the legal situation is for that. In my case, the Arch package is more of a "recipe maker". It fetches the Linux headers and the zfs source code and compiles this for local use. As far as they are concerned, there is no distribution of the resulting artifact. IANAL, but I think if there's an issue with that, then OpenZFS is basically never usable under Linux. Other companies distributed kernels with zfs support directly, such as Ubuntu. I don't recall there being news of them being sued over this, but maybe they managed to work something out.
    - 5e92cb50239222b 3y ago
      archzfs does not distribute any kernel images, they only provide pre-built modules for the officially supported kernels.
  - dsr_ 3y ago
    IANAL. Oracle is very litigious. However, OpenZFS has been releasing code for more than a decade. Ubuntu shipped integrated ZFS/Linux in 2016. It's certain that Oracle knows all about it and has decided that being vague is more in their interests than actually settling the matter. On my list of potential legal worries, this is not a priority for me.
the8472 3y ago
> zfs is technically great It's only great due to the lack of competitors in the checksummed-CoW-raid category. It lacks a bunch of things: Defrag, Reflinks, On-Demand Dedup, Rebalance (online raid geometry change, device removal, device shrink). It also wastes RAM due to page cache + ARC.
- orra 3y ago
  Sure, there's lots of room for improvement. IIRC, rebalancing might be a WIP, finally? But credit where credit is due: for a long time, ZFS has been the only fit for purpose filesystem, if you care about the integrity of your data.
  - the8472 3y ago
    Afaik true rebalancing isn't in the works. Some limited add-device and remove-vdev features are in progress but AIUI they come with additional overhead and aren't as flexible. btrfs and bcachefs rebalance leave your pool as if you had created it from scratch with the existing data and the new layout.
- e12e 3y ago
  > [ZFS is] only great due to the lack of competitors in the checksummed-CoW-raid category. You forgot robust native encryption, network transparent dump/restore (ZFS send/receive) - and broad platform support (not so much anymore). For a while you could have a solid FS with encryption support for your USB hd that could be safely used with Linux, *BSD, Windows, Open/FOSS Solaris and MacOS.
  - josephg 3y ago
    Is it just the implementation of zfs which is owned by oracle now? I wonder how hard it would be to write a compatible clean room reimplementation of zfs in rust or something, from the spec. Even if it doesn’t implement every feature from the real zfs, it would still be handy for OS compatibility reasons.
    - 3y ago
ilyt 3y ago
> [1] It's been in development for a number of years. It now being proposed for inclusion in the mainline kernel is a major milestone. not a measure of quality in the slightest. btrfs had some serious bugs over the years despise being in mainline
- orra 3y ago
  True, but bcachefs gives the impression of being better designed, nor being rushed upstream. I think it helps that bcachefs evolved from bcache.
rektide 3y ago
Btrfs still highly recommends a raid1 mode for Metadata, but for data itself, the raid-5 is fine. I somewhat recall there being a little progress on trying to fix the remaining "write hole" issues, in the past year or two. But in general, I think there's very little pressure to do so because so very many many people run raid-5 for data already & it works great. Getting Metadata off raid1 is low priority, a nice to have.
- tremon 3y ago
  Still, even with raid1 for metadata and raid5 for data, the kernel still shouts at you about it being EXPERIMENTAL every time you mount such a filesystem. I understand that it's best to err on the side of caution, but that notice does a good job of persisting the idea that btrfs isn't ready for prime-time use. I use btrfs on most of my Linux systems now (though only one with raid5), except for backup disks and backups volumes: those I intend to keep on ext4 indefinitely.
- kiririn 3y ago
  Raid5 works ok until you scrub. Even scrubbing one device at a time is a barrage of random reads sustained for days at a time I’ll very happily move back from MD raid 5 when linear scrub for parity raid lands
  - rektide 3y ago
    Great point. I need to test this more & find out. I have seen a bunch of changes in the past year go by about how scrubbing is prioritized & what capacity it is allocated. On the one hand, unsettling. But also, good to see folks are prodding at this area.

by orra 3y ago

sedatk 3y ago

> btrfs still doesn't have reliable RAID5 Synology offers btrfs + RAID5 without warning the user. I wonder why they’re so confident with it.

bestham 3y ago
They are running brtrfs on top of DM. https://kb.synology.com/en-nz/DSM/tutorial/What_was_the_RAID_implementation_for_Btrfs_File_System_on_SynologyNAS https://kb.synology.com/en-nz/DSM/tutorial/What_was_the_RAID...
- sedatk 3y ago
  Thanks for the link!
sporkle-feet 3y ago
Synology doesn't use the btrfs raid - AIUI they layer non-raid btrfs over raid LVM
- sedatk 3y ago
  Thanks, I wonder what the advantage of using FS-based RAID is.

vladvasiliu 3y ago

> * zfs — zfs is technically great, but binary distribution of the zfs code is tricky, because the CDDL is GPL incompatible Building your own ZFS module is easy enough, for example on Arch with zfs-dkms. But there's also the issue of compatibility. Sometimes kernel updates will break ZFS. Even minor ones, 6.2.13 IIRC broke it, whereas 6.2.12 was fine. Right now, 6.3 seems to introduce major compatibility problems. --- edit: looking through the openzfs issues, I was likely thinking of 6.2.8 breaking it, where 6.2.7 was fine. Point stands, though. https://github.com/openzfs/zfs/issues/14658 https://github.com/openzfs/zfs/issues/14658 Regarding 6.3 support, it apparently is merged in the master branch, but no release as of yet. https://github.com/openzfs/zfs/issues/14622 https://github.com/openzfs/zfs/issues/14622

bjoli 3y ago
How is the legal situation of doing that? If I had a company I wouldn't want to get in trouble with any litigious companies.
- vladvasiliu 3y ago
  The ArchZFS project distributes binary kernel images with ZFS integrated. I don't know what the legal situation is for that. In my case, the Arch package is more of a "recipe maker". It fetches the Linux headers and the zfs source code and compiles this for local use. As far as they are concerned, there is no distribution of the resulting artifact. IANAL, but I think if there's an issue with that, then OpenZFS is basically never usable under Linux. Other companies distributed kernels with zfs support directly, such as Ubuntu. I don't recall there being news of them being sued over this, but maybe they managed to work something out.
  - 5e92cb50239222b 3y ago
    archzfs does not distribute any kernel images, they only provide pre-built modules for the officially supported kernels.
- dsr_ 3y ago
  IANAL. Oracle is very litigious. However, OpenZFS has been releasing code for more than a decade. Ubuntu shipped integrated ZFS/Linux in 2016. It's certain that Oracle knows all about it and has decided that being vague is more in their interests than actually settling the matter. On my list of potential legal worries, this is not a priority for me.

the8472 3y ago

> zfs is technically great It's only great due to the lack of competitors in the checksummed-CoW-raid category. It lacks a bunch of things: Defrag, Reflinks, On-Demand Dedup, Rebalance (online raid geometry change, device removal, device shrink). It also wastes RAM due to page cache + ARC.

orra 3y ago
Sure, there's lots of room for improvement. IIRC, rebalancing might be a WIP, finally? But credit where credit is due: for a long time, ZFS has been the only fit for purpose filesystem, if you care about the integrity of your data.
- the8472 3y ago
  Afaik true rebalancing isn't in the works. Some limited add-device and remove-vdev features are in progress but AIUI they come with additional overhead and aren't as flexible. btrfs and bcachefs rebalance leave your pool as if you had created it from scratch with the existing data and the new layout.
e12e 3y ago
> [ZFS is] only great due to the lack of competitors in the checksummed-CoW-raid category. You forgot robust native encryption, network transparent dump/restore (ZFS send/receive) - and broad platform support (not so much anymore). For a while you could have a solid FS with encryption support for your USB hd that could be safely used with Linux, *BSD, Windows, Open/FOSS Solaris and MacOS.
- josephg 3y ago
  Is it just the implementation of zfs which is owned by oracle now? I wonder how hard it would be to write a compatible clean room reimplementation of zfs in rust or something, from the spec. Even if it doesn’t implement every feature from the real zfs, it would still be handy for OS compatibility reasons.
  - 3y ago

ilyt 3y ago

> [1] It's been in development for a number of years. It now being proposed for inclusion in the mainline kernel is a major milestone. not a measure of quality in the slightest. btrfs had some serious bugs over the years despise being in mainline

orra 3y ago
True, but bcachefs gives the impression of being better designed, nor being rushed upstream. I think it helps that bcachefs evolved from bcache.

rektide 3y ago

Btrfs still highly recommends a raid1 mode for Metadata, but for data itself, the raid-5 is fine. I somewhat recall there being a little progress on trying to fix the remaining "write hole" issues, in the past year or two. But in general, I think there's very little pressure to do so because so very many many people run raid-5 for data already & it works great. Getting Metadata off raid1 is low priority, a nice to have.

tremon 3y ago
Still, even with raid1 for metadata and raid5 for data, the kernel still shouts at you about it being EXPERIMENTAL every time you mount such a filesystem. I understand that it's best to err on the side of caution, but that notice does a good job of persisting the idea that btrfs isn't ready for prime-time use. I use btrfs on most of my Linux systems now (though only one with raid5), except for backup disks and backups volumes: those I intend to keep on ext4 indefinitely.
kiririn 3y ago
Raid5 works ok until you scrub. Even scrubbing one device at a time is a barrage of random reads sustained for days at a time I’ll very happily move back from MD raid 5 when linear scrub for parity raid lands
- rektide 3y ago
  Great point. I need to test this more & find out. I have seen a bunch of changes in the past year go by about how scrubbing is prioritized & what capacity it is allocated. On the one hand, unsettling. But also, good to see folks are prodding at this area.