7 ms·

With brtfs having all kind of crazy bugs and zfs never being finished, I guess it's back to ext4...

by panpanna 6y ago

flemhans 6y ago
Is ZFS not ready for use? I was going to use it in production next year.
- washadjeffmad 6y ago
  ZoL != OpenZFS != ZoF. A lot has changed recently, so you'll have to clarify the platform and version. ZFS on FUSE should be deprecated by now, and ZoL I believe merged with OpenZFS for the 2.0 release. FreeBSD ZFS is fine, and just check your versions if using it on Linux. Ie- don't use anything pre-2.0.
- NickNameNick 6y ago
  My home server is hardly production, but it has been running a ZFS raidz+1 for more than 5 years, and survived a disk failure. (Nb, avoid SMR hard drives - the rebuild took more than a week!)
  - StavrosK 6y ago
    Mine is RAIDZ too, and has become unusably slow since a few years back. Now I need to find a way to recreate the pool, but it's too much data to store on another disk... Not in love with ZFS so far.
    - magicalhippo 6y ago
      ZFS is great for storing data. It's very unlikely it will lose your data when using ZFS. It does however require a bit of care to maintain performance, and you need to know your expected workload going in. Otherwise you can find yourself in a situation with a very poorly performing pool where the only realistic route to recovery is a send/receive to a fresh pool and back. If you do not have a lot of sync workload (VMs, DBs) and don't have super-high performance needs, say a home NAS, then mainly you just need to think about not filling up the pool too much. A nice way to do this is to create a root dataset where you set a quota to say 75% of capacity, and then create all other datasets below this one. You should not go above 85% space usage, as ZFS switches allocation strategy then to one which can significantly increase fragmentation. If you do have a lot of sync writes, a SLOG device is basically mandatory. The SLOG device does not have to be large, it only stores about 5-10 seconds worth of writes, so 10-20GB can be plenty. I've partitioned up my SSDs and created a mirror out of two small partitions, using the remaining SSD space for other things. One thing to keep in mind is that while an L2ARC device sounds like a great thing, depending on your configuration you can actually slow things down with one. A bunch of disks has a lot more bandwidth than a single SATA SSD. An L2ARC device also requires some memory overhead, so reduces your primary ARC. Again depending on load this can be detrimental. And finally, don't ever think about using deduplication, unless you've read about the consequences, measured the performance benefits and ensured the memory overhead is acceptable. It sounds great on paper but has a lot of associated downsides that can ruin pool performance, and disabling it does not make it go away. At least that's what I've picked up so far.
      - StavrosK 6y ago
        This is a home NAS with 50% free space, I rarely write stuff and don't even read much, yet it still managed to get super slow :( I don't use deduplication or compression, and the good people over at #zfs have failed to find anything wrong multiple times. I think recreating the entire thing is about my only option at the moment.
        magicalhippo 6y ago
        That sucks, but also very weird. My pool, 2 vdevs each a 4-way RAID-Z1, has been used and abused for almost 7 years now. I've gone over the 85% mark but it still works fine, performance too. Some VM stuff but mostly media and similar. Do you mean slow IOPS or also sequential performance?
        StavrosK 6y ago
        It's mostly sequential performance, but it's also very odd. Deleting a single 2 GB file sometimes takes tens of seconds, or copying files from one volume to another will again be extremely slow.. .
        magicalhippo 6y ago
        It does sound like fragmentation, which can still happen with a pool that's not filled under the "right" circumstances (usually related to workload vs configuration). But yeah sadly that's the one area where ZFS is less stellar. Once it's fragmented it's hard to fix. Easiest is to send/receive to another pool, but as you note that is not always feasible.
        StavrosK 6y ago
        Is there no defragmentation tool? I do have lots of free space.
        magicalhippo 6y ago
- lmm 6y ago
  ZFS is great, but using it on Linux has difficulties.
  - betaby 6y ago
    'apt install zfs-dkms' is not difficult at all
- kim0 6y ago
  ZoL is very much production ready. Been using it for 5+ years. Rock solid
- zaarn 6y ago
  One thing Btrfs somewhat promised was mixed disk RAIDs. Ie, having a 2TB disk and a 1TB disk and being able to say "keep 2 copies of any data in this subvolume" and "keep only 1 copy in this subvolume". I'm currently on Bcachefs, which has erasure coding with similar promises and works better for me. ZFS is more rigid, I have to have all matched disks for best performance, plus RAM unless I like bad performance (I don't). This could be solved by striping disks into 1TB partitions, but ZFS doesn't like living on a partitioned disk nearly as much. Mixed disk has a bunch of cool properties for homelab users. Bcachefs also solves another problem; caching and speed. I could designate my 1TB NVMe as fast and my SATA 2TB SSDs as slow. I could also define that /tmp requiers no Erasure Coding and everything else requires 2 replicas. Then I would be able to combine the performance of my NVMe with the capacity of my SATA SSDs while gaining a simple redundance solution. My NAS has a dozen of mixed disks that I could combine more easily than ever. The issue with ZFS is not that it's not ready to use. The issue is that it's born from enterprise and requires a costly enterprise setup to run, rather than a cheap homelab setup.
  - kefyras 6y ago
    > but ZFS doesn't like living on a partitioned disk nearly as much. Don't know how true is this, since it's not even possible to create a zpool on the whole unpartitioned device on linux. It automatically creates GPT label with zfs and a small efi partition.
    - zaarn 6y ago
      That is true, but I suspect that only happens to protect the zpool against tools like the Windows Repair Install (which likes to attempt repairs on disks without GPT label). Using ZFS on a partition inside a GPT disk tends to lead to a few quirky behaviours, like for example ZFS not always reporting correct SMART data of the device.
      - fomine3 6y ago
        IIRC ZoL `zpool create /dev/disk/by-id/ata-XXX` makes gpt partition table and zfs partition inside it.
- cesarb 6y ago
  It can depend a lot on your definition of "ready to use". For instance, a common threshold for "ready to use" is "included in a released version of the upstream Linux kernel, and not under CONFIG_BROKEN or CONFIG_STAGING". Under that definition, neither ZFS nor bcachefs are "ready to use".
Schlaefer 6y ago
It's an edge case performance regression. There were also changes that improve performance in other cases. That happens all the time in software development.
ahachete 6y ago
Is any software ever finished? ZFS has been quite feature-full and production ready for more than a decade... why do you think it is not finished?
- MisterTea 6y ago
  > Is any software ever finished? Yes. When it does what it advertises, properly.
  - rincebrain 6y ago
    OpenZFS, in general, does what it advertises fairly well, and does not advertise features it does not possess. In what way has your experience differed?
    - MisterTea 6y ago
      My answer does not in any way reference the quality of ZFS. It only addressed the GP's question which I quoted.
      - rincebrain 6y ago
        My apologies, my question should have been directed more specifically at the GP who claimed ZFS is never finished.
tagrun 6y ago
Back to ext4 because btrfs had crazy bugs? ext4 had at least two critical data corruption bugs in the past 5 years in stable kernels (personal anecdote: one ruined the root filesystem on a server that I used, after which I stopped using ext4).
- kasabali 6y ago
  > ext4 had at least two critical data corruption bugs in the past 5 years in stable kernels No it hadn't. Those bugs were lower in the stack in block layer.
  - tagrun 6y ago
    Yes, it did. Here's the fix for one of the ext4 corruption bugs: https://lwn.net/Articles/645722/ https://lwn.net/Articles/645722/ (see also https://lwn.net/Articles/645720/ https://lwn.net/Articles/645720/). The one you're referring to is https://lwn.net/Articles/774440/ https://lwn.net/Articles/774440/ which is a different bug. I should add that as a user, what matters to me is the reliability of my data. If a bug exists outside of fs/ext4 in the kernel but affects only ext4 and not other filesystems (such as https://www.phoronix.com/scan.php?page=news_item&px=MTIxNDQ https://www.phoronix.com/scan.php?page=news_item&px=MTIxNDQ which was caused by an ext4-related commit, and similar others in prior years), this makes ext4 unreliable for me.
    - kasabali 6y ago
      You claimed there were two critical data corruption bugs in the past 5 years, one which was debunked as a not ext4 bug and the other one you linked is from April 2015, which happened more than 5 years ago. > which was caused by an ext4-related commit Now that's blatantly wrong. It wasn't caused by an ext4-related commit. The commit to blame was scheduler code in the block layer. Nothing filesystem specific.
      - tagrun 6y ago
        You're arguing that it happened 5 years and 7 months ago, which is more than 5 years, so all is well with ext4. Seriously? The point is, ext4 isn't a monument of reliability like you delude yourself with, and it had, and will continue having data corruption bugs. > Now that's blatantly wrong. It wasn't caused by an ext4-related commit. Yes it was. This is very clear from the context: https://patchwork.ozlabs.org/project/linux-ext4/patch/500F1C28.8010800@redhat.com/ https://patchwork.ozlabs.org/project/linux-ext4/patch/500F1C... You can try and weasel your way out with the semantics, say that it didn't change any files under ext4/ (which does happen with xfs/btrfs/... related patches too BTW, simply because fs/ contains common fs code), but the reality is, it was an ext4 related fix, it appeared in "linux-ext4" mailing list, and Ted Tso, the ext4 maintainer, signed off the patch. Lastly, idolizing a piece of code is nonsensical. Yes, btrfs had its data corruption bugs, but you can't pretend that ext4 and other filesystems didn't.

With brtfs having all kind of crazy bugs and zfs never being finished, I guess it's back to ext4...

flemhans 6y ago

Is ZFS not ready for use? I was going to use it in production next year.

washadjeffmad 6y ago
ZoL != OpenZFS != ZoF. A lot has changed recently, so you'll have to clarify the platform and version. ZFS on FUSE should be deprecated by now, and ZoL I believe merged with OpenZFS for the 2.0 release. FreeBSD ZFS is fine, and just check your versions if using it on Linux. Ie- don't use anything pre-2.0.
NickNameNick 6y ago
My home server is hardly production, but it has been running a ZFS raidz+1 for more than 5 years, and survived a disk failure. (Nb, avoid SMR hard drives - the rebuild took more than a week!)
- StavrosK 6y ago
  Mine is RAIDZ too, and has become unusably slow since a few years back. Now I need to find a way to recreate the pool, but it's too much data to store on another disk... Not in love with ZFS so far.
  - magicalhippo 6y ago
    ZFS is great for storing data. It's very unlikely it will lose your data when using ZFS. It does however require a bit of care to maintain performance, and you need to know your expected workload going in. Otherwise you can find yourself in a situation with a very poorly performing pool where the only realistic route to recovery is a send/receive to a fresh pool and back. If you do not have a lot of sync workload (VMs, DBs) and don't have super-high performance needs, say a home NAS, then mainly you just need to think about not filling up the pool too much. A nice way to do this is to create a root dataset where you set a quota to say 75% of capacity, and then create all other datasets below this one. You should not go above 85% space usage, as ZFS switches allocation strategy then to one which can significantly increase fragmentation. If you do have a lot of sync writes, a SLOG device is basically mandatory. The SLOG device does not have to be large, it only stores about 5-10 seconds worth of writes, so 10-20GB can be plenty. I've partitioned up my SSDs and created a mirror out of two small partitions, using the remaining SSD space for other things. One thing to keep in mind is that while an L2ARC device sounds like a great thing, depending on your configuration you can actually slow things down with one. A bunch of disks has a lot more bandwidth than a single SATA SSD. An L2ARC device also requires some memory overhead, so reduces your primary ARC. Again depending on load this can be detrimental. And finally, don't ever think about using deduplication, unless you've read about the consequences, measured the performance benefits and ensured the memory overhead is acceptable. It sounds great on paper but has a lot of associated downsides that can ruin pool performance, and disabling it does not make it go away. At least that's what I've picked up so far.
    - StavrosK 6y ago
      This is a home NAS with 50% free space, I rarely write stuff and don't even read much, yet it still managed to get super slow :( I don't use deduplication or compression, and the good people over at #zfs have failed to find anything wrong multiple times. I think recreating the entire thing is about my only option at the moment.
      - magicalhippo 6y ago
        That sucks, but also very weird. My pool, 2 vdevs each a 4-way RAID-Z1, has been used and abused for almost 7 years now. I've gone over the 85% mark but it still works fine, performance too. Some VM stuff but mostly media and similar. Do you mean slow IOPS or also sequential performance?
        StavrosK 6y ago
        It's mostly sequential performance, but it's also very odd. Deleting a single 2 GB file sometimes takes tens of seconds, or copying files from one volume to another will again be extremely slow.. .
        magicalhippo 6y ago
        It does sound like fragmentation, which can still happen with a pool that's not filled under the "right" circumstances (usually related to workload vs configuration). But yeah sadly that's the one area where ZFS is less stellar. Once it's fragmented it's hard to fix. Easiest is to send/receive to another pool, but as you note that is not always feasible.
        StavrosK 6y ago
        Is there no defragmentation tool? I do have lots of free space.
        magicalhippo 6y ago
lmm 6y ago
ZFS is great, but using it on Linux has difficulties.
- betaby 6y ago
  'apt install zfs-dkms' is not difficult at all
kim0 6y ago
ZoL is very much production ready. Been using it for 5+ years. Rock solid
zaarn 6y ago
One thing Btrfs somewhat promised was mixed disk RAIDs. Ie, having a 2TB disk and a 1TB disk and being able to say "keep 2 copies of any data in this subvolume" and "keep only 1 copy in this subvolume". I'm currently on Bcachefs, which has erasure coding with similar promises and works better for me. ZFS is more rigid, I have to have all matched disks for best performance, plus RAM unless I like bad performance (I don't). This could be solved by striping disks into 1TB partitions, but ZFS doesn't like living on a partitioned disk nearly as much. Mixed disk has a bunch of cool properties for homelab users. Bcachefs also solves another problem; caching and speed. I could designate my 1TB NVMe as fast and my SATA 2TB SSDs as slow. I could also define that /tmp requiers no Erasure Coding and everything else requires 2 replicas. Then I would be able to combine the performance of my NVMe with the capacity of my SATA SSDs while gaining a simple redundance solution. My NAS has a dozen of mixed disks that I could combine more easily than ever. The issue with ZFS is not that it's not ready to use. The issue is that it's born from enterprise and requires a costly enterprise setup to run, rather than a cheap homelab setup.
- kefyras 6y ago
  > but ZFS doesn't like living on a partitioned disk nearly as much. Don't know how true is this, since it's not even possible to create a zpool on the whole unpartitioned device on linux. It automatically creates GPT label with zfs and a small efi partition.
  - zaarn 6y ago
    That is true, but I suspect that only happens to protect the zpool against tools like the Windows Repair Install (which likes to attempt repairs on disks without GPT label). Using ZFS on a partition inside a GPT disk tends to lead to a few quirky behaviours, like for example ZFS not always reporting correct SMART data of the device.
    - fomine3 6y ago
      IIRC ZoL `zpool create /dev/disk/by-id/ata-XXX` makes gpt partition table and zfs partition inside it.
cesarb 6y ago
It can depend a lot on your definition of "ready to use". For instance, a common threshold for "ready to use" is "included in a released version of the upstream Linux kernel, and not under CONFIG_BROKEN or CONFIG_STAGING". Under that definition, neither ZFS nor bcachefs are "ready to use".

Schlaefer 6y ago

It's an edge case performance regression. There were also changes that improve performance in other cases. That happens all the time in software development.

ahachete 6y ago

Is any software ever finished? ZFS has been quite feature-full and production ready for more than a decade... why do you think it is not finished?

MisterTea 6y ago
> Is any software ever finished? Yes. When it does what it advertises, properly.
- rincebrain 6y ago
  OpenZFS, in general, does what it advertises fairly well, and does not advertise features it does not possess. In what way has your experience differed?
  - MisterTea 6y ago
    My answer does not in any way reference the quality of ZFS. It only addressed the GP's question which I quoted.
    - rincebrain 6y ago
      My apologies, my question should have been directed more specifically at the GP who claimed ZFS is never finished.

tagrun 6y ago

Back to ext4 because btrfs had crazy bugs? ext4 had at least two critical data corruption bugs in the past 5 years in stable kernels (personal anecdote: one ruined the root filesystem on a server that I used, after which I stopped using ext4).

kasabali 6y ago
> ext4 had at least two critical data corruption bugs in the past 5 years in stable kernels No it hadn't. Those bugs were lower in the stack in block layer.
- tagrun 6y ago
  Yes, it did. Here's the fix for one of the ext4 corruption bugs: https://lwn.net/Articles/645722/ https://lwn.net/Articles/645722/ (see also https://lwn.net/Articles/645720/ https://lwn.net/Articles/645720/). The one you're referring to is https://lwn.net/Articles/774440/ https://lwn.net/Articles/774440/ which is a different bug. I should add that as a user, what matters to me is the reliability of my data. If a bug exists outside of fs/ext4 in the kernel but affects only ext4 and not other filesystems (such as https://www.phoronix.com/scan.php?page=news_item&px=MTIxNDQ https://www.phoronix.com/scan.php?page=news_item&px=MTIxNDQ which was caused by an ext4-related commit, and similar others in prior years), this makes ext4 unreliable for me.
  - kasabali 6y ago
    You claimed there were two critical data corruption bugs in the past 5 years, one which was debunked as a not ext4 bug and the other one you linked is from April 2015, which happened more than 5 years ago. > which was caused by an ext4-related commit Now that's blatantly wrong. It wasn't caused by an ext4-related commit. The commit to blame was scheduler code in the block layer. Nothing filesystem specific.
    - tagrun 6y ago
      You're arguing that it happened 5 years and 7 months ago, which is more than 5 years, so all is well with ext4. Seriously? The point is, ext4 isn't a monument of reliability like you delude yourself with, and it had, and will continue having data corruption bugs. > Now that's blatantly wrong. It wasn't caused by an ext4-related commit. Yes it was. This is very clear from the context: https://patchwork.ozlabs.org/project/linux-ext4/patch/500F1C28.8010800@redhat.com/ https://patchwork.ozlabs.org/project/linux-ext4/patch/500F1C... You can try and weasel your way out with the semantics, say that it didn't change any files under ext4/ (which does happen with xfs/btrfs/... related patches too BTW, simply because fs/ contains common fs code), but the reality is, it was an ext4 related fix, it appeared in "linux-ext4" mailing list, and Ted Tso, the ext4 maintainer, signed off the patch. Lastly, idolizing a piece of code is nonsensical. Yes, btrfs had its data corruption bugs, but you can't pretend that ext4 and other filesystems didn't.