3 ms·

BTRFS is not as bad as it was. People often forget all the edge conditions ZFS has also. (Of which I got caught by many in my day.) That said, BTRFS is also

by ilc 3y ago

BTRFS is not as bad as it was. People often forget all the edge conditions ZFS has also. (Of which I got caught by many in my day.) That said, BTRFS is also not the rampant layering violation that ZFS is, for the good and the bad of that statement in so many ways. The way ARC and L2ARC is handled... is just crazy from a modern prospective. They may have cleaned it up since I last looked, but I tend to doubt it. It was pretty baked in. BTRFS / ZFS really share 0 with bittorrent other than checksumming for integrity. That's it. Otherwise they are so different they might as well come from different planets. ceph has a ways to go. It'll get there, but distributed filesystems are hard. As a wise man said to me: Normal file systems take ~10-15 years to mature. Distributed ones take even longer. So ceph... may be getting usable, but I'm not sure I'd bet the farm on cephfs. (Their S3 stuff maybe. But that's much simpler.) minio is a great example of a product not trying to be everything to everyone... and succeeding. Kudos to them and their team.

jtriangle 3y ago
Simple implementations are always better in general really, because you find yourself well within any reasonable test case. Sure you can do fantastically exotic things with all of the above, but, at that point you're on your own. I've always much preferred to keep things as "down the middle" as I can for all of those reasons. Never had a CEPH cluster fail, haven't had any issues with ZFS (recently), nor any issues with BTRFS (recently), and that's not magic, it's all just because all of that is by the book or not done at all. It's a constraint at times, to be sure, but, hard to scoff at the results. Also you're correct in assuming ARC/L2ARC is still.... weird. Sometimes you get wirespeed cache hits, sometimes you just don't, with no rhyme or reason outside of "well it just decided that file didn't belong in cache".
- ilc 3y ago
  It isn't no rhyme or reason. The issue has to do with layering caches over each other. If you layer caches over each other and they don't pass hit rates to each other bad things happen, especially for designs like ARC. The ZFS caches explicitly take hit rate into account... but with a page cache over them... they don't know the true hit rates. Heck they are a secondary cache ;). Then L2ARC has the same issue one layer down. And there's the issue. It isn't magical. It is just annoying. Honestly on linux: btrfs is probably better for your sanity than ZFS. As far as ceph goes, if you understand it, and can do the systems engineering.. yeah it is great stuff. My only complaint with ceph is just too complex for most mortals. (Despite that once you get to any real large scale... you can't avoid that complexity.) ceph you can do some crazy shit if you know what you are doing. Want to do a seamless cluster migration? We got ya. ;) Not as good as AFS probably. But it is pretty scary.
  - anonacct37 3y ago
    What you say makes lots of sense. I can see how hit rates not being passed down would cause issues. But anecdotally I think I'm in the same spot many others are. Btrfs let me down a couple times (once lots of truncated files and another time I ran out of inode space on a mostly empty disk iirc) and I'm in no hurry to give it another chance. zfs has just happily worked for me. I ran it as a root filesystem for years. All the snapshot/send/receive stuff just worked and stayed out of my way. Note that this was for personal use and with SSDs and later nvme so I wasn't comparing it head to head with anything. Which might be why I didn't notice issues like page caching behavior.
    - ilc 3y ago
      ZFS isn't what lets you down in the ZFS stack. It is memory fragmentation due to ARC, and the ARC not responding to memory pressure correctly. I'm sure there's parts of btrfs that still suck. Don't get me wrong. And you can do the running out of space, so you can't remove files trick with ZFS too ;). Personally, I only put btrfs into use on my laptop about 1.5 years ago. So... Take that for what you will. I'm pretty conservative.
  - jtriangle 3y ago
    True, ARC isn't magic, it is more that it makes poor decisions on occasion, with no real way to work around it reliably, aka no guaranteed way to ensure a cache hit. I would posit however, if you need to guarantee cache hits... you don't really need cache, you need to roll your own flash storage and find a way to intelligently manage what's present on it at a given time. I've done this, namely for an environment that had very, very clear, scheduled needs for hot data, warm data, and archive data. Fairly trivial to take files and just scoot them to a different storage array when they're X days old on a nightly basis. That system is completely agnostic to, well, everything, I've done it on windows hosts and, while it's more of a pain in the ass, also worked fine there. That was back in the hardware raid or nothing days, when having a terabyte of flash available was fairly exotic and expensive, but, as a model, programmatically structured storage isn't old by any means and probably fits many environments better than ZFS and "as much memory as accounting would approve". Also, I don't really 100% agree that ceph is too complex for mere mortals, it's more that, it very, very quickly becomes too complex for your average linux sysadmin to handle if you're not careful. I really should dip my toes deeper into BTRFS for production usage though. I've played with it in the lab, I run it at home almost exclusively, but, never have done any big boy britches production stuff with it and it very well might be a good tool to have in my pocket.

by ilc 3y ago

jtriangle 3y ago

Simple implementations are always better in general really, because you find yourself well within any reasonable test case. Sure you can do fantastically exotic things with all of the above, but, at that point you're on your own. I've always much preferred to keep things as "down the middle" as I can for all of those reasons. Never had a CEPH cluster fail, haven't had any issues with ZFS (recently), nor any issues with BTRFS (recently), and that's not magic, it's all just because all of that is by the book or not done at all. It's a constraint at times, to be sure, but, hard to scoff at the results. Also you're correct in assuming ARC/L2ARC is still.... weird. Sometimes you get wirespeed cache hits, sometimes you just don't, with no rhyme or reason outside of "well it just decided that file didn't belong in cache".

ilc 3y ago
It isn't no rhyme or reason. The issue has to do with layering caches over each other. If you layer caches over each other and they don't pass hit rates to each other bad things happen, especially for designs like ARC. The ZFS caches explicitly take hit rate into account... but with a page cache over them... they don't know the true hit rates. Heck they are a secondary cache ;). Then L2ARC has the same issue one layer down. And there's the issue. It isn't magical. It is just annoying. Honestly on linux: btrfs is probably better for your sanity than ZFS. As far as ceph goes, if you understand it, and can do the systems engineering.. yeah it is great stuff. My only complaint with ceph is just too complex for most mortals. (Despite that once you get to any real large scale... you can't avoid that complexity.) ceph you can do some crazy shit if you know what you are doing. Want to do a seamless cluster migration? We got ya. ;) Not as good as AFS probably. But it is pretty scary.
- anonacct37 3y ago
  What you say makes lots of sense. I can see how hit rates not being passed down would cause issues. But anecdotally I think I'm in the same spot many others are. Btrfs let me down a couple times (once lots of truncated files and another time I ran out of inode space on a mostly empty disk iirc) and I'm in no hurry to give it another chance. zfs has just happily worked for me. I ran it as a root filesystem for years. All the snapshot/send/receive stuff just worked and stayed out of my way. Note that this was for personal use and with SSDs and later nvme so I wasn't comparing it head to head with anything. Which might be why I didn't notice issues like page caching behavior.
  - ilc 3y ago
    ZFS isn't what lets you down in the ZFS stack. It is memory fragmentation due to ARC, and the ARC not responding to memory pressure correctly. I'm sure there's parts of btrfs that still suck. Don't get me wrong. And you can do the running out of space, so you can't remove files trick with ZFS too ;). Personally, I only put btrfs into use on my laptop about 1.5 years ago. So... Take that for what you will. I'm pretty conservative.
- jtriangle 3y ago
  True, ARC isn't magic, it is more that it makes poor decisions on occasion, with no real way to work around it reliably, aka no guaranteed way to ensure a cache hit. I would posit however, if you need to guarantee cache hits... you don't really need cache, you need to roll your own flash storage and find a way to intelligently manage what's present on it at a given time. I've done this, namely for an environment that had very, very clear, scheduled needs for hot data, warm data, and archive data. Fairly trivial to take files and just scoot them to a different storage array when they're X days old on a nightly basis. That system is completely agnostic to, well, everything, I've done it on windows hosts and, while it's more of a pain in the ass, also worked fine there. That was back in the hardware raid or nothing days, when having a terabyte of flash available was fairly exotic and expensive, but, as a model, programmatically structured storage isn't old by any means and probably fits many environments better than ZFS and "as much memory as accounting would approve". Also, I don't really 100% agree that ceph is too complex for mere mortals, it's more that, it very, very quickly becomes too complex for your average linux sysadmin to handle if you're not careful. I really should dip my toes deeper into BTRFS for production usage though. I've played with it in the lab, I run it at home almost exclusively, but, never have done any big boy britches production stuff with it and it very well might be a good tool to have in my pocket.