5 ms·

This is one of the most laughably ignorant posts I've ever seen on the internet. > Btrfs has had no real work done on it since then, and anyone that is actuall

This is one of the most laughably ignorant posts I've ever seen on the internet. > Btrfs has had no real work done on it since then, and anyone that is actually serious about data storage has moved to ZFS, or has left POSIX filesystems entirely Btrfs is the most actively developed filesystem in the kernel, both by LOC changed and the number of contributors. Facebook publicly stated that they run BTRFS on every webserver years ago. ZFS is an antiquated joke. It is wed to the memory allocator in Solaris, and requires that you permanently reserve 1GB of RAM per 1TB of disk space, which is an impractical constraint in the modern world. ZFS is also full of cute little wasteful "innovations" like 128-bit file size support. I don't know what planet you're from, but it's not mine :)

uniqueuid 5y ago
Please correct me if I'm wrong, but the "1GB ram per TB" is for the de-duplication table. ZFS' memory cache (ARC) by default uses up to 50% of system ram and frees this under pressure. You can also set it lower.
- jcalvinowens 5y ago
  Yes, this is what I'm talking about. With respect to ARC, it gets along very poorly with the native page cache and memory allocator. I would add that BTRFS has no such requirements for deduplication.
  - db48x 5y ago
    That’s because the btrfs deduplicator is quite a different animal. You could just as easily run fdupes or something similar. Meanwhile ZFS is doing something much more ambitious. In any case, I don’t think that either of them are worth using in most cases. Deduplication is just too expensive to be worth it.
    - Dylan16807 5y ago
      > You could just as easily run fdupes or something similar. If you only deduplicate whole files and soft links or hard links are suitable for your situation.
      - db48x 5y ago
        That’s not the aspect of the deduplication process that I was referring to. Btrfs only supports offline deduplication, where you periodically run a process that searches for duplicate blocks and combines them. Meanwhile ZFS supports online deduplication, where every block written is checked against an index to see if it is a duplicate of some existing block. That index is what takes up the memory that jcalvinowens complained about. When deduplication is turned on, ZFS keeps that whole index in memory all of the time. Meanwhile btrfs only needs that index to be loaded while the deduplicator is running, so it doesn’t show up most of the time. Personally I don’t think that either is really worth the cost; the benefit is so small that the cost isn’t worth paying.
        Dylan16807 5y ago
        I know the ZFS deduplicator accomplishes more, I'm just pointing out that that if you want offline deduplication on ZFS, you can't have it.
mavhc 5y ago
What are they changing all the LOCs for if not to fix the existing massive data loss bugs?
dijit 5y ago
You’re not correct on a number of fronts here. ZFS is not tied to the memory allocator in Solaris, I’m very unclear how you could have made that mistake. 128bits might feel wasteful, but you’re likely never going to have the problems FAT32 has had- it’s designed for the next 100 years- which has drawbacks but prevent the kinds of ugly hacks that NTFS has gone through to maintain forward compatibility (rolling on disk upgrade). And yes, the 1G per TB is for the in memory block map, which is a requirement for deduplucation on the default block size.
- jcalvinowens 5y ago
  > ZFS is not tied to the memory allocator in Solaris, I’m very unclear how you could have made that mistake > And yes, the 1G per TB is for the in memory block map, which is a requirement for deduplucation on the default block size. You actually answered your own question: it's antiquated because of the assumption it is possible to get contiguous RAM for that mapping. On modern Linux, that is only possible if you reserve it pre-boot.
  - dijit 5y ago
    The Solaris memory allocator is not contiguous either though
    - jcalvinowens 5y ago
      This is a whole rabbit hole... but essentially, Linux uses a naive buddy allocator for physical RAM, and Solaris was more interesting.
  - detaro 5y ago
    > it's antiquated because of the assumption it is possible to get contiguous RAM for that mapping. source on that? (and even if that's the case, wouldn't that be an implementation detail? E.g. implementations can spill it to L2ARC, but somehow not to different memory locations?)
    - 5y ago
isatty 5y ago
Look you can’t start your post with > This is one of the most laughably ignorant posts I've ever seen on the internet. And say > requires that you permanently reserve 1GB of RAM per 1TB of disk If you want people to take you seriously. There are ways to debate on HN, this isn’t it.
deleted 5y ago
[deleted]

uniqueuid 5y ago

Please correct me if I'm wrong, but the "1GB ram per TB" is for the de-duplication table. ZFS' memory cache (ARC) by default uses up to 50% of system ram and frees this under pressure. You can also set it lower.

jcalvinowens 5y ago
Yes, this is what I'm talking about. With respect to ARC, it gets along very poorly with the native page cache and memory allocator. I would add that BTRFS has no such requirements for deduplication.
- db48x 5y ago
  That’s because the btrfs deduplicator is quite a different animal. You could just as easily run fdupes or something similar. Meanwhile ZFS is doing something much more ambitious. In any case, I don’t think that either of them are worth using in most cases. Deduplication is just too expensive to be worth it.
  - Dylan16807 5y ago
    > You could just as easily run fdupes or something similar. If you only deduplicate whole files and soft links or hard links are suitable for your situation.
    - db48x 5y ago
      That’s not the aspect of the deduplication process that I was referring to. Btrfs only supports offline deduplication, where you periodically run a process that searches for duplicate blocks and combines them. Meanwhile ZFS supports online deduplication, where every block written is checked against an index to see if it is a duplicate of some existing block. That index is what takes up the memory that jcalvinowens complained about. When deduplication is turned on, ZFS keeps that whole index in memory all of the time. Meanwhile btrfs only needs that index to be loaded while the deduplicator is running, so it doesn’t show up most of the time. Personally I don’t think that either is really worth the cost; the benefit is so small that the cost isn’t worth paying.
      - Dylan16807 5y ago
        I know the ZFS deduplicator accomplishes more, I'm just pointing out that that if you want offline deduplication on ZFS, you can't have it.

mavhc 5y ago

What are they changing all the LOCs for if not to fix the existing massive data loss bugs?

dijit 5y ago

You’re not correct on a number of fronts here. ZFS is not tied to the memory allocator in Solaris, I’m very unclear how you could have made that mistake. 128bits might feel wasteful, but you’re likely never going to have the problems FAT32 has had- it’s designed for the next 100 years- which has drawbacks but prevent the kinds of ugly hacks that NTFS has gone through to maintain forward compatibility (rolling on disk upgrade). And yes, the 1G per TB is for the in memory block map, which is a requirement for deduplucation on the default block size.

jcalvinowens 5y ago
> ZFS is not tied to the memory allocator in Solaris, I’m very unclear how you could have made that mistake > And yes, the 1G per TB is for the in memory block map, which is a requirement for deduplucation on the default block size. You actually answered your own question: it's antiquated because of the assumption it is possible to get contiguous RAM for that mapping. On modern Linux, that is only possible if you reserve it pre-boot.
- dijit 5y ago
  The Solaris memory allocator is not contiguous either though
  - jcalvinowens 5y ago
    This is a whole rabbit hole... but essentially, Linux uses a naive buddy allocator for physical RAM, and Solaris was more interesting.
- detaro 5y ago
  > it's antiquated because of the assumption it is possible to get contiguous RAM for that mapping. source on that? (and even if that's the case, wouldn't that be an implementation detail? E.g. implementations can spill it to L2ARC, but somehow not to different memory locations?)
  - 5y ago

isatty 5y ago

Look you can’t start your post with > This is one of the most laughably ignorant posts I've ever seen on the internet. And say > requires that you permanently reserve 1GB of RAM per 1TB of disk If you want people to take you seriously. There are ways to debate on HN, this isn’t it.

deleted 5y ago

[deleted]