6 ms·

>ZFS on Linux its been said in the thread already but this was always a non-starter. Torvalds even said so himself. CDDL was the last poison pill of a dying

>ZFS on Linux its been said in the thread already but this was always a non-starter. Torvalds even said so himself. CDDL was the last poison pill of a dying giant who couldnt pull its foot from the well. What we, er, the linux community, chose instead, was BTRFS. It isnt ZFS, but its made incredible strides. for most use cases, it is a reasonable and working replacement for ZFS.

_hyn3 5y ago
I lost data and had to restore from 12 hour old backups one too many times with BTRFS. XFS + Ext4 for me from here on out, but that's one of the great things about Linux: lots of choices.
- nix23 5y ago
  XFS all the way down...or ZFS (on FreeBSD)
- acomjean 5y ago
  Lamentably, for those of us that just use linux, the lots of choices seem weird and frankly a little scary (Stories of data loss with BTRFS I have heard before.) I use the default Ext4? I think? But as a believer in open source, I really would love it if the choices we had were so much superior to the proprietary stuff that was out there that it made using an open source OS a no brainer.
nix23 5y ago
>Torvalds even said so himself. Torvald's comment about ZFS was as uninformed as it gets...and he calls himself an FS-Guy ;(
- arid_eden 5y ago
  His comment was more on the wisdom (or otherwise) of running an out of tree filesystem. I think its hard to disagree with him. He went on to say you would never be able to merge the ZFS tree with Linux. Again he's the one who would know what code gets in Linux. His only actual comment against ZFS was that benchmarks didn't look great - which is unsurprising given all the extra work ZFS is doing wrt data integrity than other filesystems in production use. https://www.realworldtech.com/forum/?threadid=189711&curpostid=189841 https://www.realworldtech.com/forum/?threadid=189711&curpost...
  - wang_li 5y ago
    From the link: >[ZFS] was always more of a buzzword than anything else, I feel, This is deeply ignorant. I feel that Linux has been handicapped by the fact that many developers have never done any serious enterprise administration and thus not having clear understanding of the needs of a set of their users.
  - nix23 5y ago
    Not everything needs to be in the Linux Kernel (and honestly i don't care if it is), looking at the past "linux-sound-system-tragedy" i would say, to make something outside linux is often much better (not 1000's different peoples who thinks it's better the other way around, and you are full of sh* anyway). >His only actual comment against ZFS was that benchmarks didn't look great What benchmark? Mines are looking pretty good, with a preheated Arc and with L2 especially...actually much better than any HW-Raid. Compared to Linus, there are Institutes with a bit more than a single 3GB git repository, and the crazy stuff...they need verified backups. https://computing.llnl.gov/projects/zfs-lustre https://computing.llnl.gov/projects/zfs-lustre If he ever has to improve a Lustre Filesystem of 55 petabyte with ZFS, he can come back, otherwise Linus...shut-up* and be happy with your ext4 (nothing against that one). * A homage to the old linus-style of having a discussion.
josteink 5y ago
> What we, er, the linux community, chose instead, was BTRFS. It isnt ZFS Speak for yourself. As a part of “the Linux community” I gave btrfs a fair chance, but stopped using it because it constantly failed on me in ways no other fs had done before and didn’t protect my data. ZFS is rock solid and I’ve never had any of the issues I had with btrfs. So as a member of “the Linux community” you claim to unilaterally represent I put such petty license-politics aside and choose the file system which serves my needs best, and that is ZFS.
guenthert 5y ago
BTRFS is the only fs I recall having lost data to (rather it corrupts data, i.e. data of one file is found in another) as late as 2018.
ahl 5y ago
What is the latest on RAID-5/6 support for BTRFS. RAID-Z has its issues (saying this as the triple- and double-parity author), but it's been stable.
- simcop2387 5y ago
  As far as Ive heard it's still pretty iffy. Since you were involved with the zfs side of things what are your thoughts on the upcoming zfs draid bits? I don't have specific need for them myself but they look really attractive for building a new pool to replace my aging drives.
  - ahl 5y ago
    Unfortunately I've been completely out of the loop on the draid stuff.
jjav 5y ago
When filesystem integrity matters, the filesystem matters more than the OS. While I mostly use Linux these days, for file servers it must be ZFS, which means whichever OS has first-class support for ZFS. I'm still on Illumos but perhaps will move to FreeBSD at some point.
- arid_eden 5y ago
  This is why FreeBSD rebasing its ZFS fork on ZFS-on-Linux made me so scared for the future of FreeBSD. Their one major advantage over Linux and they didn't have the developers to maintain their fork themselves.
  - laumars 5y ago
    ZFS will always be a smoother experience on FreeBSD as opposed to Linux because FreeBSD endorses it. Thus the user land and documentation is written assuming you’re running ZFS. As opposed to Linux where some distros might ship pre-compiled binaries but everything is written assuming you’re not running ZFS. Thus everything takes that extra couple of steps to set up, fix, and maintain. For example, if you want to use ZFS as a storage for containers on Linux, you have to spend hours hunting around for some poorly maintained 3rd party shell scripts or build some tooling yourself. Whereas on FreeBSD all the tooling around Jails is built with ZFS in mind. This is why platforms like FreeBSD feel more harmonious than Linux. Not because Linux can’t do the job but because there are so many different contributors with their own unique preferences that Linux is essentially loose Lego pieces with no instructions. Whereas FreeBSD has the same org who manage the kernel, user land and who also push ZFS. And I say this as someone who loves Linux. There’s room for both Linux and FreeBSD in this world :)
    - mappu 5y ago
      I think "smoother experience on FreeBSD" is a myth - The standard volume manager on FreeBSD is vinum/geom; ZFS ships its entire separate volume manager to the host OS, so you can't use mount/umount to control mounting a ZFS volume. Maybe it would be okay to move entirely over to ZFS's volume manager but it only supports ZFS's own filesystem, you can't use the ZFS volume manager with a normal FreeBSD UFS2 partition. In both Linux and FreeBSD, ZFS's bolt-on ARC competes with the kernel's actual page cache for resources instead of properly integrating with it. It's an out-of-tree filesystem for both OSes. Sure FreeBSD periodically imports it into master from OpenZFS (née ZoL), but all development happens elsewhere, and the SPL is still trying to emulate a Solaris interface on top of both OSes. Is there any more concrete example of how ZFS is actually better integrated on FreeBSD compared to Linux, say Ubuntu? It takes ZFS snapshots automatically during apt upgrades, root-on-ZFS is a default installer option, etc.
guardiangod 5y ago
>What we, er, the linux community, chose instead, was BTRFS. Isn't that putting politic before technical excellence, something the Linux crowd is proud of? Other than in place volume expansion, there is no technical reason to choose BTRFS over ZFS (for now.) I don't really see a killer feature from BTRFS that would persuade me to take a chance with it.
- blihp 5y ago
  It's being pragmatic. Linux has typically placed freedom ahead of pretty much everything else.[1] All else being equal, sure you want the best technical solution. But if it doesn't fit the definition of freedom that Linux requires, how otherwise good a solution is doesn't matter. So the main 'killer' feature of BTRFS is that it fits the licensing requirements for integration into Linux. Linux has a great many problems, but being sticklers for a particular type of license isn't one of them IMO. [1] This isn't just idealism. See Oracle v. Google for an example of what happens if you play fast and loose with licenses and a malicious actor. Google eventually won, but how many millions of dollars did that victory cost them? Oracle would love Linux developers to blunder their way into the receiving end of a lawsuit.
- rnd0 5y ago
  >Isn't that putting politic before technical excellence, something the Linux crowd is proud of? It's not unprecedented. The adoption of systemd was forced on distros through political pressure, and not for technical reasons. If you want a truly non-political OS community these days, I think you're basically stuck with OpenBSD. No CoC, no systemd, no political BS at all -just pure tech. (there's other problems with OpenBSD -performance, mostly; that's why I use windows and Ubuntu instead. But the way they run things is admirable IMO. Blatant BS isn't tolerated.)
  - dale_glass 5y ago
    systemd wasn't "forced" on distros. The distros adopted it because they liked it. The thing is that systemd did something quite clever -- it sold itself to the people actually building distributions, which are the people that actually matter the most in regards what system software gets used. It made their jobs easier and less annoying in many ways. As somebody who's done a lot of packaging and writing of SysV scripts, I can tell you that it's a tiresome and annoying task even for a small amount of software, let alone a whole distro. At that point the unix philosophy loses its luster quite a bit.
laumars 5y ago
> for most use cases, it is a reasonable and working replacement for ZFS. That is a huuuuge overstatement for the current state of Btrfs. In some specific domains it is a working replacement. But for most domains it still falls far behind ZFS in terms of stability, resiliency or even ease of use. By all means if you want to use btrfs then go for it. But the favourable comparisons people make when comparing btrfs to ZFS is a combination of wishful thinking and not having really bullied their fs into those extreme edge cases where the cracks begin to show. And frankly I’d rather depend on something that has had those assurances for 10+yrs already than have the hassle of explaining downtime to restore data on production systems.
bombcar 5y ago
The Linux Community is working on BTRFS but if ZFS emerged with a GPLv2 compatible license tomorrow BTRFS would likely be moribund.
- znpy 5y ago
  Yup. If openzfs was gpl-ed tomorrow morning, btrfs would be dead by tomorrow at lunch time.
arid_eden 5y ago
My main issue with ZFS is the integrated nature - like systemd for filesystems. My 'alternative' for ZFS isn't BTRFS (awful performance characteristics for my workloads) but LVM coupled with ext4 and mdraid. I get snapshots, reliability, performance and a 'real UNIX' composable toolchain. I miss out on data checksums.
- riku_iki 5y ago
  No compression..
  - bombcar 5y ago
    You could use VDO (I never have) https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/deduplicating_and_compressing_logical_volumes_on_rhel/con_introduction-to-vdo-on-lvm_deduplicating-and-compressing-logical-volumes https://access.redhat.com/documentation/en-us/red_hat_enterp...
- e12e 5y ago
  In principle I dislike the coupling of volume manager, raid and filesystem. But I still think zfs gets most things right; I see the argument for a concistent system managing caching/logs, volumes, data integrity, discard support, compression, snapshots and encryption. The fact that it's the first serious, open, cross platform solution (Linux, bsd, Mac, winnt) that provides encryption, integrity and filesystem is a nice bonus. And the integration of snapshots and fs dumps via zfs send/receive is beautiful. I think zfs makes sense like one fat layer - networking can go below (drdb, iscsi) or on top (iscsi, nfs, cifs). Encryption need to be somewhat holistic - for making sane performance and data leaking tradeoffs.
- salmo 5y ago
  Having run all these thing in prod (except BTRFS, it ate a mirror on my desktop), I’ll say that even the LVM + + is so much more hacky than geom on FreeBSD which feels much more ‘Unix’ with a designed composable interface. Although, I do prefer the durability of XFS or ext4 (depending on workload) vs UFS, and the setup you described is totally maintainable.

_hyn3 5y ago

I lost data and had to restore from 12 hour old backups one too many times with BTRFS. XFS + Ext4 for me from here on out, but that's one of the great things about Linux: lots of choices.

nix23 5y ago
XFS all the way down...or ZFS (on FreeBSD)
acomjean 5y ago
Lamentably, for those of us that just use linux, the lots of choices seem weird and frankly a little scary (Stories of data loss with BTRFS I have heard before.) I use the default Ext4? I think? But as a believer in open source, I really would love it if the choices we had were so much superior to the proprietary stuff that was out there that it made using an open source OS a no brainer.

nix23 5y ago

>Torvalds even said so himself. Torvald's comment about ZFS was as uninformed as it gets...and he calls himself an FS-Guy ;(

arid_eden 5y ago
His comment was more on the wisdom (or otherwise) of running an out of tree filesystem. I think its hard to disagree with him. He went on to say you would never be able to merge the ZFS tree with Linux. Again he's the one who would know what code gets in Linux. His only actual comment against ZFS was that benchmarks didn't look great - which is unsurprising given all the extra work ZFS is doing wrt data integrity than other filesystems in production use. https://www.realworldtech.com/forum/?threadid=189711&curpostid=189841 https://www.realworldtech.com/forum/?threadid=189711&curpost...
- wang_li 5y ago
  From the link: >[ZFS] was always more of a buzzword than anything else, I feel, This is deeply ignorant. I feel that Linux has been handicapped by the fact that many developers have never done any serious enterprise administration and thus not having clear understanding of the needs of a set of their users.
- nix23 5y ago
  Not everything needs to be in the Linux Kernel (and honestly i don't care if it is), looking at the past "linux-sound-system-tragedy" i would say, to make something outside linux is often much better (not 1000's different peoples who thinks it's better the other way around, and you are full of sh* anyway). >His only actual comment against ZFS was that benchmarks didn't look great What benchmark? Mines are looking pretty good, with a preheated Arc and with L2 especially...actually much better than any HW-Raid. Compared to Linus, there are Institutes with a bit more than a single 3GB git repository, and the crazy stuff...they need verified backups. https://computing.llnl.gov/projects/zfs-lustre https://computing.llnl.gov/projects/zfs-lustre If he ever has to improve a Lustre Filesystem of 55 petabyte with ZFS, he can come back, otherwise Linus...shut-up* and be happy with your ext4 (nothing against that one). * A homage to the old linus-style of having a discussion.

josteink 5y ago

> What we, er, the linux community, chose instead, was BTRFS. It isnt ZFS Speak for yourself. As a part of “the Linux community” I gave btrfs a fair chance, but stopped using it because it constantly failed on me in ways no other fs had done before and didn’t protect my data. ZFS is rock solid and I’ve never had any of the issues I had with btrfs. So as a member of “the Linux community” you claim to unilaterally represent I put such petty license-politics aside and choose the file system which serves my needs best, and that is ZFS.

guenthert 5y ago

BTRFS is the only fs I recall having lost data to (rather it corrupts data, i.e. data of one file is found in another) as late as 2018.

ahl 5y ago

What is the latest on RAID-5/6 support for BTRFS. RAID-Z has its issues (saying this as the triple- and double-parity author), but it's been stable.

simcop2387 5y ago
As far as Ive heard it's still pretty iffy. Since you were involved with the zfs side of things what are your thoughts on the upcoming zfs draid bits? I don't have specific need for them myself but they look really attractive for building a new pool to replace my aging drives.
- ahl 5y ago
  Unfortunately I've been completely out of the loop on the draid stuff.

jjav 5y ago

When filesystem integrity matters, the filesystem matters more than the OS. While I mostly use Linux these days, for file servers it must be ZFS, which means whichever OS has first-class support for ZFS. I'm still on Illumos but perhaps will move to FreeBSD at some point.

arid_eden 5y ago
This is why FreeBSD rebasing its ZFS fork on ZFS-on-Linux made me so scared for the future of FreeBSD. Their one major advantage over Linux and they didn't have the developers to maintain their fork themselves.
- laumars 5y ago
  ZFS will always be a smoother experience on FreeBSD as opposed to Linux because FreeBSD endorses it. Thus the user land and documentation is written assuming you’re running ZFS. As opposed to Linux where some distros might ship pre-compiled binaries but everything is written assuming you’re not running ZFS. Thus everything takes that extra couple of steps to set up, fix, and maintain. For example, if you want to use ZFS as a storage for containers on Linux, you have to spend hours hunting around for some poorly maintained 3rd party shell scripts or build some tooling yourself. Whereas on FreeBSD all the tooling around Jails is built with ZFS in mind. This is why platforms like FreeBSD feel more harmonious than Linux. Not because Linux can’t do the job but because there are so many different contributors with their own unique preferences that Linux is essentially loose Lego pieces with no instructions. Whereas FreeBSD has the same org who manage the kernel, user land and who also push ZFS. And I say this as someone who loves Linux. There’s room for both Linux and FreeBSD in this world :)
  - mappu 5y ago
    I think "smoother experience on FreeBSD" is a myth - The standard volume manager on FreeBSD is vinum/geom; ZFS ships its entire separate volume manager to the host OS, so you can't use mount/umount to control mounting a ZFS volume. Maybe it would be okay to move entirely over to ZFS's volume manager but it only supports ZFS's own filesystem, you can't use the ZFS volume manager with a normal FreeBSD UFS2 partition. In both Linux and FreeBSD, ZFS's bolt-on ARC competes with the kernel's actual page cache for resources instead of properly integrating with it. It's an out-of-tree filesystem for both OSes. Sure FreeBSD periodically imports it into master from OpenZFS (née ZoL), but all development happens elsewhere, and the SPL is still trying to emulate a Solaris interface on top of both OSes. Is there any more concrete example of how ZFS is actually better integrated on FreeBSD compared to Linux, say Ubuntu? It takes ZFS snapshots automatically during apt upgrades, root-on-ZFS is a default installer option, etc.

guardiangod 5y ago

>What we, er, the linux community, chose instead, was BTRFS. Isn't that putting politic before technical excellence, something the Linux crowd is proud of? Other than in place volume expansion, there is no technical reason to choose BTRFS over ZFS (for now.) I don't really see a killer feature from BTRFS that would persuade me to take a chance with it.

blihp 5y ago
It's being pragmatic. Linux has typically placed freedom ahead of pretty much everything else.[1] All else being equal, sure you want the best technical solution. But if it doesn't fit the definition of freedom that Linux requires, how otherwise good a solution is doesn't matter. So the main 'killer' feature of BTRFS is that it fits the licensing requirements for integration into Linux. Linux has a great many problems, but being sticklers for a particular type of license isn't one of them IMO. [1] This isn't just idealism. See Oracle v. Google for an example of what happens if you play fast and loose with licenses and a malicious actor. Google eventually won, but how many millions of dollars did that victory cost them? Oracle would love Linux developers to blunder their way into the receiving end of a lawsuit.
rnd0 5y ago
>Isn't that putting politic before technical excellence, something the Linux crowd is proud of? It's not unprecedented. The adoption of systemd was forced on distros through political pressure, and not for technical reasons. If you want a truly non-political OS community these days, I think you're basically stuck with OpenBSD. No CoC, no systemd, no political BS at all -just pure tech. (there's other problems with OpenBSD -performance, mostly; that's why I use windows and Ubuntu instead. But the way they run things is admirable IMO. Blatant BS isn't tolerated.)
- dale_glass 5y ago
  systemd wasn't "forced" on distros. The distros adopted it because they liked it. The thing is that systemd did something quite clever -- it sold itself to the people actually building distributions, which are the people that actually matter the most in regards what system software gets used. It made their jobs easier and less annoying in many ways. As somebody who's done a lot of packaging and writing of SysV scripts, I can tell you that it's a tiresome and annoying task even for a small amount of software, let alone a whole distro. At that point the unix philosophy loses its luster quite a bit.

laumars 5y ago

> for most use cases, it is a reasonable and working replacement for ZFS. That is a huuuuge overstatement for the current state of Btrfs. In some specific domains it is a working replacement. But for most domains it still falls far behind ZFS in terms of stability, resiliency or even ease of use. By all means if you want to use btrfs then go for it. But the favourable comparisons people make when comparing btrfs to ZFS is a combination of wishful thinking and not having really bullied their fs into those extreme edge cases where the cracks begin to show. And frankly I’d rather depend on something that has had those assurances for 10+yrs already than have the hassle of explaining downtime to restore data on production systems.

bombcar 5y ago

The Linux Community is working on BTRFS but if ZFS emerged with a GPLv2 compatible license tomorrow BTRFS would likely be moribund.

znpy 5y ago
Yup. If openzfs was gpl-ed tomorrow morning, btrfs would be dead by tomorrow at lunch time.

arid_eden 5y ago

My main issue with ZFS is the integrated nature - like systemd for filesystems. My 'alternative' for ZFS isn't BTRFS (awful performance characteristics for my workloads) but LVM coupled with ext4 and mdraid. I get snapshots, reliability, performance and a 'real UNIX' composable toolchain. I miss out on data checksums.

riku_iki 5y ago
No compression..
- bombcar 5y ago
  You could use VDO (I never have) https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/deduplicating_and_compressing_logical_volumes_on_rhel/con_introduction-to-vdo-on-lvm_deduplicating-and-compressing-logical-volumes https://access.redhat.com/documentation/en-us/red_hat_enterp...
e12e 5y ago
In principle I dislike the coupling of volume manager, raid and filesystem. But I still think zfs gets most things right; I see the argument for a concistent system managing caching/logs, volumes, data integrity, discard support, compression, snapshots and encryption. The fact that it's the first serious, open, cross platform solution (Linux, bsd, Mac, winnt) that provides encryption, integrity and filesystem is a nice bonus. And the integration of snapshots and fs dumps via zfs send/receive is beautiful. I think zfs makes sense like one fat layer - networking can go below (drdb, iscsi) or on top (iscsi, nfs, cifs). Encryption need to be somewhat holistic - for making sane performance and data leaking tradeoffs.
salmo 5y ago
Having run all these thing in prod (except BTRFS, it ate a mirror on my desktop), I’ll say that even the LVM + + is so much more hacky than geom on FreeBSD which feels much more ‘Unix’ with a designed composable interface. Although, I do prefer the durability of XFS or ext4 (depending on workload) vs UFS, and the setup you described is totally maintainable.