4 ms·

My dream Linux filesystem is a distributable ZFS with automatic tiering and advance notification of cache invalidation. I guess we have different dreams.

by dsr_ 3y ago

yjftsjthsd-h 3y ago
Well yeah, obviously different people have different dreams depending on what they value. Mine is ZFS but with the licensing situation fixed so it can be mainlined into Linux, or BTRFS but trustworthy and with better cross-platform support. Though I did also once sketch out a distributed filesystem that would combine the features of ZFS (ui, stability), BTRFS (heterogeneous pools), minio (distributed, erasure coding), ceph (distributed, FUSE+NFS+native kernel driver) to give me one ultimate filesystem that could scale from 1 disk in 1 drive to a single filesystem across dozens of machines and hundreds of drives. (Mostly, I noticed that at a high level, a lot of filesystems and sync tools boil down to "slice up your files into blocks, append a checksum, and then save/mirror/stream those blocks wherever you need to" - if you squint, BTRFS and bittorrent are really similar)
- ilc 3y ago
  BTRFS is not as bad as it was. People often forget all the edge conditions ZFS has also. (Of which I got caught by many in my day.) That said, BTRFS is also not the rampant layering violation that ZFS is, for the good and the bad of that statement in so many ways. The way ARC and L2ARC is handled... is just crazy from a modern prospective. They may have cleaned it up since I last looked, but I tend to doubt it. It was pretty baked in. BTRFS / ZFS really share 0 with bittorrent other than checksumming for integrity. That's it. Otherwise they are so different they might as well come from different planets. ceph has a ways to go. It'll get there, but distributed filesystems are hard. As a wise man said to me: Normal file systems take ~10-15 years to mature. Distributed ones take even longer. So ceph... may be getting usable, but I'm not sure I'd bet the farm on cephfs. (Their S3 stuff maybe. But that's much simpler.) minio is a great example of a product not trying to be everything to everyone... and succeeding. Kudos to them and their team.
  - jtriangle 3y ago
    Simple implementations are always better in general really, because you find yourself well within any reasonable test case. Sure you can do fantastically exotic things with all of the above, but, at that point you're on your own. I've always much preferred to keep things as "down the middle" as I can for all of those reasons. Never had a CEPH cluster fail, haven't had any issues with ZFS (recently), nor any issues with BTRFS (recently), and that's not magic, it's all just because all of that is by the book or not done at all. It's a constraint at times, to be sure, but, hard to scoff at the results. Also you're correct in assuming ARC/L2ARC is still.... weird. Sometimes you get wirespeed cache hits, sometimes you just don't, with no rhyme or reason outside of "well it just decided that file didn't belong in cache".
    - ilc 3y ago
      It isn't no rhyme or reason. The issue has to do with layering caches over each other. If you layer caches over each other and they don't pass hit rates to each other bad things happen, especially for designs like ARC. The ZFS caches explicitly take hit rate into account... but with a page cache over them... they don't know the true hit rates. Heck they are a secondary cache ;). Then L2ARC has the same issue one layer down. And there's the issue. It isn't magical. It is just annoying. Honestly on linux: btrfs is probably better for your sanity than ZFS. As far as ceph goes, if you understand it, and can do the systems engineering.. yeah it is great stuff. My only complaint with ceph is just too complex for most mortals. (Despite that once you get to any real large scale... you can't avoid that complexity.) ceph you can do some crazy shit if you know what you are doing. Want to do a seamless cluster migration? We got ya. ;) Not as good as AFS probably. But it is pretty scary.
      - anonacct37 3y ago
        What you say makes lots of sense. I can see how hit rates not being passed down would cause issues. But anecdotally I think I'm in the same spot many others are. Btrfs let me down a couple times (once lots of truncated files and another time I ran out of inode space on a mostly empty disk iirc) and I'm in no hurry to give it another chance. zfs has just happily worked for me. I ran it as a root filesystem for years. All the snapshot/send/receive stuff just worked and stayed out of my way. Note that this was for personal use and with SSDs and later nvme so I wasn't comparing it head to head with anything. Which might be why I didn't notice issues like page caching behavior.

My dream Linux filesystem is a distributable ZFS with automatic tiering and advance notification of cache invalidation. I guess we have different dreams.

by dsr_ 3y ago

My dream Linux filesystem is a distributable ZFS with automatic tiering and advance notification of cache invalidation. I guess we have different dreams.

yjftsjthsd-h 3y ago

Well yeah, obviously different people have different dreams depending on what they value. Mine is ZFS but with the licensing situation fixed so it can be mainlined into Linux, or BTRFS but trustworthy and with better cross-platform support. Though I did also once sketch out a distributed filesystem that would combine the features of ZFS (ui, stability), BTRFS (heterogeneous pools), minio (distributed, erasure coding), ceph (distributed, FUSE+NFS+native kernel driver) to give me one ultimate filesystem that could scale from 1 disk in 1 drive to a single filesystem across dozens of machines and hundreds of drives. (Mostly, I noticed that at a high level, a lot of filesystems and sync tools boil down to "slice up your files into blocks, append a checksum, and then save/mirror/stream those blocks wherever you need to" - if you squint, BTRFS and bittorrent are really similar)

ilc 3y ago
BTRFS is not as bad as it was. People often forget all the edge conditions ZFS has also. (Of which I got caught by many in my day.) That said, BTRFS is also not the rampant layering violation that ZFS is, for the good and the bad of that statement in so many ways. The way ARC and L2ARC is handled... is just crazy from a modern prospective. They may have cleaned it up since I last looked, but I tend to doubt it. It was pretty baked in. BTRFS / ZFS really share 0 with bittorrent other than checksumming for integrity. That's it. Otherwise they are so different they might as well come from different planets. ceph has a ways to go. It'll get there, but distributed filesystems are hard. As a wise man said to me: Normal file systems take ~10-15 years to mature. Distributed ones take even longer. So ceph... may be getting usable, but I'm not sure I'd bet the farm on cephfs. (Their S3 stuff maybe. But that's much simpler.) minio is a great example of a product not trying to be everything to everyone... and succeeding. Kudos to them and their team.
- jtriangle 3y ago
  Simple implementations are always better in general really, because you find yourself well within any reasonable test case. Sure you can do fantastically exotic things with all of the above, but, at that point you're on your own. I've always much preferred to keep things as "down the middle" as I can for all of those reasons. Never had a CEPH cluster fail, haven't had any issues with ZFS (recently), nor any issues with BTRFS (recently), and that's not magic, it's all just because all of that is by the book or not done at all. It's a constraint at times, to be sure, but, hard to scoff at the results. Also you're correct in assuming ARC/L2ARC is still.... weird. Sometimes you get wirespeed cache hits, sometimes you just don't, with no rhyme or reason outside of "well it just decided that file didn't belong in cache".
  - ilc 3y ago
    It isn't no rhyme or reason. The issue has to do with layering caches over each other. If you layer caches over each other and they don't pass hit rates to each other bad things happen, especially for designs like ARC. The ZFS caches explicitly take hit rate into account... but with a page cache over them... they don't know the true hit rates. Heck they are a secondary cache ;). Then L2ARC has the same issue one layer down. And there's the issue. It isn't magical. It is just annoying. Honestly on linux: btrfs is probably better for your sanity than ZFS. As far as ceph goes, if you understand it, and can do the systems engineering.. yeah it is great stuff. My only complaint with ceph is just too complex for most mortals. (Despite that once you get to any real large scale... you can't avoid that complexity.) ceph you can do some crazy shit if you know what you are doing. Want to do a seamless cluster migration? We got ya. ;) Not as good as AFS probably. But it is pretty scary.
    - anonacct37 3y ago
      What you say makes lots of sense. I can see how hit rates not being passed down would cause issues. But anecdotally I think I'm in the same spot many others are. Btrfs let me down a couple times (once lots of truncated files and another time I ran out of inode space on a mostly empty disk iirc) and I'm in no hurry to give it another chance. zfs has just happily worked for me. I ran it as a root filesystem for years. All the snapshot/send/receive stuff just worked and stayed out of my way. Note that this was for personal use and with SSDs and later nvme so I wasn't comparing it head to head with anything. Which might be why I didn't notice issues like page caching behavior.