5 ms·

What filesystem would you suggest that has data checksums, efficient snapshots, and doesn't require compiling an out of tree kernel module?

freeone3000 2y ago
zfs is licensed too freely to be in-tree, but it’s still an excellent choice.
- do_not_redeem 2y ago
  Yeah the situation is unfortunate. There's a decent chance I'd be using ZFS if not for the licensing issues, but as a practical matter I'm getting too old to be futzing with kernel modules on my daily driver.
  - telgareith 2y ago
    DKMS solved these "licensing issues." Dell is mum on official motivation- but it provides a licensing demarcation point, and a way for kernels to update without breaking modules- so it's easier for companies to develop for Linux. _Windows Drivers work the same way and nobody huffs and puffs about that_ I'd love to have an intelligent discussion on how one person's opinion on licensing issues stacks up against the legal teams of half the fortune 50's. Licensing doesn't work on "well, I didn't mean it THAT way."
    - do_not_redeem 2y ago
      I admit I'm not fully up to date on whether it's actually "license issues" or something else. I'm not a lawyer. As a layman here's what I know. I go to the Arch wiki (https://wiki.archlinux.org/title/ZFS https://wiki.archlinux.org/title/ZFS) and I see this warning under the DKMS section (as you advised): > Warning: Occasionally, the dkms package might not compile against the newest kernel packages in Arch. Using the linux-lts kernel may provide better compatibility with out-of-tree kernel modules, otherwise zfs-dkms-staging-gitAUR backports compatibility patches and fixes for the latest kernel package in Arch on top of the stable zfs branch So... my system might fail to boot after updates. If I use linux-lts, it might break less often. Or I can use zfs-dkms-staging-git, and my system might break even less often... or more often, because it looks like that's installing kernel modules directly from the master branch of some repo. As a practical matter I could care less if my system fails to boot because of "license issues" or some other reason, I just want the lawyers to sort their shit out so I don't have to risk my system becoming unbootable at some random inopportune time. Until then, I've never hit a btrfs bug, so I'm going to keep on using it for every new build.
      - MrDrMcCoy 2y ago
        I've been bitten by kernel module incompatibility making my data unavailable enough times that I no longer consider ZFS to be viable under Linux. Using an LTS kernel only delays the issue until the next release is LTS. I really hope that bcachefs goes stable soon.
      - telgareith 2y ago
        Its. Not. The. Lawyers.
    - arp242 2y ago
      I used ZFS with DKMS on CentOS back in the day, and I found it a pain. It took a long time to compile, and I had some issues with upgrades as well (it's been a few years, so I have forgotten what the exact issues were). When it comes to filesystems, I very much appreciate the "it just works" experience – not having a working filesystem is not having a working system and it a pain to solve. Again, all of this has been a while. Maybe it's better now and I'm not opposed to trying, but I consider "having to use DKMS" to be a downside of ZFS.
  - asyx 2y ago
    I think ZFS on Debian is pretty safe. Debian is conservative enough that I don’t expect the DKMS build to fail on me.
    - mort96 2y ago
      I would never use a DKMS filesystem for / though.
- riffraff 2y ago
  I may be wrong but I don't think it's just "excessive freeness", the CDDL also has restrictions the GPL does not have (stuff about patents), it's mutual incompatibility. Apache v2 and GPLv3 were made explicitly compatible while providing different kinds of freedom.
  - boomboomsubban 2y ago
    The CDDL is more permissive, it's a weak copyleft license while the GPL is strong copyleft, and that makes the two incompatible. Calling it "excessive freeness" is inflammatory, but they're broadly correct.
    - starspangled 2y ago
      > The CDDL is more permissive, it's a weak copyleft license while the GPL is strong copyleft, and that makes the two incompatible. Calling it "excessive freeness" is inflammatory, but they're broadly correct. It's not really. Many aspects of the license are free-er, but that's not what causes the incompatibility. The GPL does not have any kind of clause saying code that is distributed under too permissive of a license may not be incorporated into a derived/combined work. It's not that it's weak copyleft, it is that it contains particular restrictions that makes it incompatible with GPL's restrictions. BSD licenses do not have that incompatible restriction (= are freer than CDDL, in that aspect) and can be compatible with the GPL.
      - arp242 2y ago
        The basic problem is that both licenses have a "you must redistribute under this license, and you can't impose additional restrictions"-type clause. There are some other differences between the licences, but that's what the incompatibility is about. Some have argued that this may not actually constitute an incompatibility, but not many are keen to "fuck around and find out" with Oracle's lawyers. So here we are.
        ChocolateGod 2y ago
- deleted 2y ago
  [deleted]
- chasil 2y ago
  In the Linux tree. In Windows, Satya would need to write Larry a check. It would probably be hefty. Edit: there was a time that this was planned for MacOS. https://arstechnica.com/gadgets/2016/06/zfs-the-other-new-apple-file-system-that-almost-was-until-it-wasnt/ https://arstechnica.com/gadgets/2016/06/zfs-the-other-new-ap...
  - gjvc 2y ago
    Edit: there was a time that this was planned for MacOS. That was a joyous prospect. A single volume manager/filesystem across all UNIX platforms would be wonderful. We had the UNIX wars of the 1990s. Since Linux won, they have been replaced by the filesystem wars.
    - chasil 2y ago
      For what it's worth, I upvoted you. I see no reason for the down votes.
    - mort96 2y ago
      It wouldn't have been a candidate for "the standard UNIX filesystem" even if it was in macOS because SUN made an intentionally GPL-incompatible license for it.
      - gjvc 2y ago
        You make a fair point, but it does not detract from the concept of having a single integrated volume manager and filesystem. Interoperability is most important.
JackSlateur 2y ago
Xfs ? Bcachefs ? Whatever you like because those features may not be implemented at the filesystem layer ?
- do_not_redeem 2y ago
  xfs doesn't have data checksumming. > Note: Unlike Btrfs and ZFS, the CRC32 checksum only applies to the metadata and not actual data. https://wiki.archlinux.org/title/XFS https://wiki.archlinux.org/title/XFS --- bcachefs isn't stable enough to daily drive. > Nobody sane uses bcachefs and expects it to be stable —Linus Torvalds (2024) https://lore.kernel.org/lkml/CAHk-%3Dwj1Oo9-g-yuwWuHQZU8v%3DVAsBceWCRLhWxy7_-QnSa1Ng@mail.gmail.com/ https://lore.kernel.org/lkml/CAHk-%3Dwj1Oo9-g-yuwWuHQZU8v%3D...
  - JackSlateur 2y ago
    [flagged]
  - koverstreet 2y ago
    Linus is not a filesystem guy
    - starspangled 2y ago
      You don't have to be to see bug reports, bug fix patches, or test it yourself, of course. "Filesystem guys" also tend to wear rose colored glasses when it comes to their filesystem, at times (ext2, xfs, btrfs, etc.)
      - koverstreet 2y ago
        Linus absolutely does jump in on VFS level bugs when necessary (and major respect for that; he doesn't rest on laurels and he's always got his priorities in the right place) - but there's only so much he can keep track of, and the complexity of a modern filesystem tends to dwarf other subsystems. The people at the top explicitly don't and can't keep track of everything, there's a lot of stuff (e.g. testing) that they leave to other people - and I do fault them for that a bit; we badly need to get a bit more organized on test infrastructure. And I wouldn't say that filesystem people in general wear rose colored glasses; I would categorize Dave Chinner and Ted T'so more in the hard nosed realistic category, and myself as well. I'd say it's just the btrfs folks who've had that fault in the past, and I think Josef has had more than enough experience at this point to learn that lesson.
toast0 2y ago
Zfs is in tree if you use a different kernel. :p
- johnny22 2y ago
  no, that is not in in tree. it's in build. big difference.
chasil 2y ago
There is an OpenZFS port to Windows, but I'm not sure how to find it from here: https://github.com/openzfsonwindows/ZFSin https://github.com/openzfsonwindows/ZFSin There is also Microsoft's own ReFS: https://en.m.wikipedia.org/wiki/ReFS https://en.m.wikipedia.org/wiki/ReFS
badgersnake 2y ago
If you artificially tailor your criteria such that the only answer you your question is btrfs then that is the answer you will get.
- homebrewer 2y ago
  There's nothing "artificial" in his requirements, data checksums and efficient snapshots are required for some workloads (for example, we use them for end-to-end testing on copies of the real production database that are created and thrown away in seconds), and building your own kernel modules is a stupid idea in many cases outside of two extremes of the home desktop or a well-funded behemoth like Facebook.
  - arp242 2y ago
    Data checksums in particular is 90% of the reason I want to use a newer filesystem than XFS or Ext4. This is useful for almost any usage. Snapshots a bit less so, but even for my laptop this would be useful, mostly for backups (that is: create snapshot, backup that, and then delete the snapshot – this is how I did things on FreeBSD back in the day). A second use case would be safer system updates and easier rollbacks – not useful that often, but when you need it, it's pretty handy.
ComputerGuru 2y ago
ZFS… using a BSD kernel :)
mappu 2y ago
NILFS2 is upstream, stores checksums for all data, and has the most efficient snapshot system bar none. Although I don't think i'd push the recommendation, solely because there are fewer eyes on it.
- ahartmetz 2y ago
  Heh, NILFS2. The first flash-friendly filesystem in Linux. When Flash was still expensive, I used it on an SD card to keep my most important source trees to speed up builds and things like svn (yes yes!) diff. I thought it had languished due to better resourced and more sophisticated efforts like F2FS.

freeone3000 2y ago

zfs is licensed too freely to be in-tree, but it’s still an excellent choice.

do_not_redeem 2y ago
Yeah the situation is unfortunate. There's a decent chance I'd be using ZFS if not for the licensing issues, but as a practical matter I'm getting too old to be futzing with kernel modules on my daily driver.
- telgareith 2y ago
  DKMS solved these "licensing issues." Dell is mum on official motivation- but it provides a licensing demarcation point, and a way for kernels to update without breaking modules- so it's easier for companies to develop for Linux. _Windows Drivers work the same way and nobody huffs and puffs about that_ I'd love to have an intelligent discussion on how one person's opinion on licensing issues stacks up against the legal teams of half the fortune 50's. Licensing doesn't work on "well, I didn't mean it THAT way."
  - do_not_redeem 2y ago
    I admit I'm not fully up to date on whether it's actually "license issues" or something else. I'm not a lawyer. As a layman here's what I know. I go to the Arch wiki (https://wiki.archlinux.org/title/ZFS https://wiki.archlinux.org/title/ZFS) and I see this warning under the DKMS section (as you advised): > Warning: Occasionally, the dkms package might not compile against the newest kernel packages in Arch. Using the linux-lts kernel may provide better compatibility with out-of-tree kernel modules, otherwise zfs-dkms-staging-gitAUR backports compatibility patches and fixes for the latest kernel package in Arch on top of the stable zfs branch So... my system might fail to boot after updates. If I use linux-lts, it might break less often. Or I can use zfs-dkms-staging-git, and my system might break even less often... or more often, because it looks like that's installing kernel modules directly from the master branch of some repo. As a practical matter I could care less if my system fails to boot because of "license issues" or some other reason, I just want the lawyers to sort their shit out so I don't have to risk my system becoming unbootable at some random inopportune time. Until then, I've never hit a btrfs bug, so I'm going to keep on using it for every new build.
    - MrDrMcCoy 2y ago
      I've been bitten by kernel module incompatibility making my data unavailable enough times that I no longer consider ZFS to be viable under Linux. Using an LTS kernel only delays the issue until the next release is LTS. I really hope that bcachefs goes stable soon.
    - telgareith 2y ago
      Its. Not. The. Lawyers.
  - arp242 2y ago
    I used ZFS with DKMS on CentOS back in the day, and I found it a pain. It took a long time to compile, and I had some issues with upgrades as well (it's been a few years, so I have forgotten what the exact issues were). When it comes to filesystems, I very much appreciate the "it just works" experience – not having a working filesystem is not having a working system and it a pain to solve. Again, all of this has been a while. Maybe it's better now and I'm not opposed to trying, but I consider "having to use DKMS" to be a downside of ZFS.
- asyx 2y ago
  I think ZFS on Debian is pretty safe. Debian is conservative enough that I don’t expect the DKMS build to fail on me.
  - mort96 2y ago
    I would never use a DKMS filesystem for / though.
riffraff 2y ago
I may be wrong but I don't think it's just "excessive freeness", the CDDL also has restrictions the GPL does not have (stuff about patents), it's mutual incompatibility. Apache v2 and GPLv3 were made explicitly compatible while providing different kinds of freedom.
- boomboomsubban 2y ago
  The CDDL is more permissive, it's a weak copyleft license while the GPL is strong copyleft, and that makes the two incompatible. Calling it "excessive freeness" is inflammatory, but they're broadly correct.
  - starspangled 2y ago
    > The CDDL is more permissive, it's a weak copyleft license while the GPL is strong copyleft, and that makes the two incompatible. Calling it "excessive freeness" is inflammatory, but they're broadly correct. It's not really. Many aspects of the license are free-er, but that's not what causes the incompatibility. The GPL does not have any kind of clause saying code that is distributed under too permissive of a license may not be incorporated into a derived/combined work. It's not that it's weak copyleft, it is that it contains particular restrictions that makes it incompatible with GPL's restrictions. BSD licenses do not have that incompatible restriction (= are freer than CDDL, in that aspect) and can be compatible with the GPL.
    - arp242 2y ago
      The basic problem is that both licenses have a "you must redistribute under this license, and you can't impose additional restrictions"-type clause. There are some other differences between the licences, but that's what the incompatibility is about. Some have argued that this may not actually constitute an incompatibility, but not many are keen to "fuck around and find out" with Oracle's lawyers. So here we are.
      - ChocolateGod 2y ago
deleted 2y ago
[deleted]
chasil 2y ago
In the Linux tree. In Windows, Satya would need to write Larry a check. It would probably be hefty. Edit: there was a time that this was planned for MacOS. https://arstechnica.com/gadgets/2016/06/zfs-the-other-new-apple-file-system-that-almost-was-until-it-wasnt/ https://arstechnica.com/gadgets/2016/06/zfs-the-other-new-ap...
- gjvc 2y ago
  Edit: there was a time that this was planned for MacOS. That was a joyous prospect. A single volume manager/filesystem across all UNIX platforms would be wonderful. We had the UNIX wars of the 1990s. Since Linux won, they have been replaced by the filesystem wars.
  - chasil 2y ago
    For what it's worth, I upvoted you. I see no reason for the down votes.
  - mort96 2y ago
    It wouldn't have been a candidate for "the standard UNIX filesystem" even if it was in macOS because SUN made an intentionally GPL-incompatible license for it.
    - gjvc 2y ago
      You make a fair point, but it does not detract from the concept of having a single integrated volume manager and filesystem. Interoperability is most important.

JackSlateur 2y ago

Xfs ? Bcachefs ? Whatever you like because those features may not be implemented at the filesystem layer ?

do_not_redeem 2y ago
xfs doesn't have data checksumming. > Note: Unlike Btrfs and ZFS, the CRC32 checksum only applies to the metadata and not actual data. https://wiki.archlinux.org/title/XFS https://wiki.archlinux.org/title/XFS --- bcachefs isn't stable enough to daily drive. > Nobody sane uses bcachefs and expects it to be stable —Linus Torvalds (2024) https://lore.kernel.org/lkml/CAHk-%3Dwj1Oo9-g-yuwWuHQZU8v%3DVAsBceWCRLhWxy7_-QnSa1Ng@mail.gmail.com/ https://lore.kernel.org/lkml/CAHk-%3Dwj1Oo9-g-yuwWuHQZU8v%3D...
- JackSlateur 2y ago
  [flagged]
- koverstreet 2y ago
  Linus is not a filesystem guy
  - starspangled 2y ago
    You don't have to be to see bug reports, bug fix patches, or test it yourself, of course. "Filesystem guys" also tend to wear rose colored glasses when it comes to their filesystem, at times (ext2, xfs, btrfs, etc.)
    - koverstreet 2y ago
      Linus absolutely does jump in on VFS level bugs when necessary (and major respect for that; he doesn't rest on laurels and he's always got his priorities in the right place) - but there's only so much he can keep track of, and the complexity of a modern filesystem tends to dwarf other subsystems. The people at the top explicitly don't and can't keep track of everything, there's a lot of stuff (e.g. testing) that they leave to other people - and I do fault them for that a bit; we badly need to get a bit more organized on test infrastructure. And I wouldn't say that filesystem people in general wear rose colored glasses; I would categorize Dave Chinner and Ted T'so more in the hard nosed realistic category, and myself as well. I'd say it's just the btrfs folks who've had that fault in the past, and I think Josef has had more than enough experience at this point to learn that lesson.

toast0 2y ago

Zfs is in tree if you use a different kernel. :p

johnny22 2y ago
no, that is not in in tree. it's in build. big difference.

chasil 2y ago

There is an OpenZFS port to Windows, but I'm not sure how to find it from here: https://github.com/openzfsonwindows/ZFSin https://github.com/openzfsonwindows/ZFSin There is also Microsoft's own ReFS: https://en.m.wikipedia.org/wiki/ReFS https://en.m.wikipedia.org/wiki/ReFS

badgersnake 2y ago

If you artificially tailor your criteria such that the only answer you your question is btrfs then that is the answer you will get.

homebrewer 2y ago
There's nothing "artificial" in his requirements, data checksums and efficient snapshots are required for some workloads (for example, we use them for end-to-end testing on copies of the real production database that are created and thrown away in seconds), and building your own kernel modules is a stupid idea in many cases outside of two extremes of the home desktop or a well-funded behemoth like Facebook.
- arp242 2y ago
  Data checksums in particular is 90% of the reason I want to use a newer filesystem than XFS or Ext4. This is useful for almost any usage. Snapshots a bit less so, but even for my laptop this would be useful, mostly for backups (that is: create snapshot, backup that, and then delete the snapshot – this is how I did things on FreeBSD back in the day). A second use case would be safer system updates and easier rollbacks – not useful that often, but when you need it, it's pretty handy.

ComputerGuru 2y ago

ZFS… using a BSD kernel :)

mappu 2y ago

NILFS2 is upstream, stores checksums for all data, and has the most efficient snapshot system bar none. Although I don't think i'd push the recommendation, solely because there are fewer eyes on it.

ahartmetz 2y ago
Heh, NILFS2. The first flash-friendly filesystem in Linux. When Flash was still expensive, I used it on an SD card to keep my most important source trees to speed up builds and things like svn (yes yes!) diff. I thought it had languished due to better resourced and more sophisticated efforts like F2FS.