5 ms·

> Data integrity is the natural expectation humans have from computers I've said it once, and I'll say it again: the only reason ZFS isn't the norm is because

> Data integrity is the natural expectation humans have from computers I've said it once, and I'll say it again: the only reason ZFS isn't the norm is because we all once lived through a primordial era when it didn't exist. No serious person designing a filesystem today would say it's okay to misplace your data. Not long ago, on this forum, someone told me that ZFS is only good because it had no competitors in its space. Which is kind of like saying the heavyweight champ is only good because no one else could compete.

hulitu 2y ago
> the only reason ZFS isn't the norm is because we all once lived through a primordial era when it didn't exist. There were good filesystems before ZFS. I would love to have a versioning filesystem like Apollo had.
wolrah 2y ago
To paraphrase, "ZFS is the worst filesystem, except for all those other filesystems that have been tried from time to time." It's far from perfect, but it has no peers. I spent many years stubbornly using btrfs and lost data multiple times. Never once did the redundancy I had supposedly configured actually do anything to help me. ZFS has identified corruption caused by bad memory and a bad CPU and let me know immediately which files were damaged.
Too 2y ago
The reason ZFS isn't the norm is because it historically was difficult to set up. Outside of NAS solutions, it's only since Ubuntu 20.04 it has been supported out of the box on any high profile customer facing OS. The reliability of the early versions was also questionable, with high zsys cpu usage and some times arcane commands needed to rebuild pools. Anecdotally, I've had to support lots of friends with zfs issues, never so with other file systems. The data always comes back, it's just that it needs petting. Earlier, there used to be lot of fears around the license, with Torvalds advising against its use, both for that reason and for lack of maintainers. Now i believe that has been mostly ironed out and should be less of an issue.
- hulitu 2y ago
  > The reason ZFS isn't the norm is because it historically was difficult to set up. Has this changed ? ZFS comes with a BSD view of the world (i.e slices). It also needed a sick amount of RAM to function properly.
- mustache_kimono 2y ago
  > The reason ZFS isn't the norm is because it historically was difficult to set up. Outside of NAS solutions, it's only since Ubuntu 20.04 it has been supported out of the box on any high profile customer facing OS. In this one very narrow sense, we are agreed, if we are talking about Linux on root. IMHO it should also have been virtually everywhere else. It should have been in MacOS, etc. However, I think your particular comment may miss the forest for the trees. Yes, ZFS was difficult to set up for Linux, because Linux people disfavored its use (which you do touch upon later). People sometimes imagine that purely technical considerations govern the technical choices of remote groups. However, I think when people say "all tech is political" in the cultural-war-ing American politics sense, they may be right, but they are absolutely right in the small ball open source politics sense. Linux communities were convinced not to include or build ZFS support. Because licensing was a problem. Because btrfs was coming and would be better. Because Linus said ZFS was mostly marketing. So they didn't care to build support. Of course, this was all BS or FUD or NIH, but it was what happened, not that ZFS had new and different recovery tool, or was less reliable in the arbitrary past. It was because the Linux community engaged in its own (successful) FUD campaign against another FOSS project.
- zvr 2y ago
  Was there any change in the license that made you believe that it should be less than a issue? Or do you think people simply stopped paying attention?
  - Too 2y ago
    Canonical took a team of lawyers to deeply review the license in 2016. It's beyond my legal skills to say if the conclusion made it more or less of an issue, at least the boundaries should now be more clear, for those who understand these matters more. https://canonical.com/blog/zfs-licensing-and-linux https://canonical.com/blog/zfs-licensing-and-linux https://softwarefreedom.org/resources/2016/linux-kernel-cddl.html https://softwarefreedom.org/resources/2016/linux-kernel-cddl...
KMag 2y ago
> No serious person designing a filesystem today would say it's okay to misplace your data. Former LimeWire developer here... the LimeWire splash screen at startup was due to experiences with silent data corruption. We got some impossible bug reports, so we created a stub executable that would show a splash screen while computing the SHA-1 checksums of the actual application DLLs and JARs. Once everything checked out, that stub would use Java reflection to start the actual application. After moving to that, those impossible bug reports stopped happening. With 60 million simultaneous users, there were always some of them with silent disk corruption that they would blame on LimeWire. When Microsoft was offering free Win7 pre-release install ISOs for download, I was having install issues. I didn't want to get my ISO illegally, so I found a torrent of the ISO, and wrote a Python script to download the ISO from Microsoft, but use the torrent file to verify chunks and re-download any corrupted chunks. Something was very wrong on some device between my desktop and Microsoft's servers, but it eventually got a non-corrupted ISO. It annoys me to no end that ECC isn't the norm for all devices with more than 1 GB of RAM. Silent bit flips are just not okay. Edit: side note: it's interesting to see the number of complaints I still see from people who blame hard drive failures on LimeWire stressing their drives. From very early on, LimeWire allowed bandwidth limiting, which I used to keep heat down on machines that didn't cool their drives properly. Beyond heat issues that I would blame on machine vendors, failures from write volume I would lay at the feet of drive manufacturers. Though, I'm biased. Any blame for drive wear that didn't fall on either the drive manufacturers or the filesystem implementers not dealing well with random writes would probably fall at my feet. I'm the one who implemented randomized chunk order downloading in order to rapidly increase availability of rare content, which would increase the number of hard drive head seeks on non-log-based filesystems. I always intended to go back and (1) use sequential downloads if tens of copies of the file were in the swarm, to reduce hard drive seeks and (2) implement randomized downloading of rarest chunks first, rather than the naive randomization in the initial implementation. I say naive, but the initial implementation did have some logic to randomize chunk download order in a way to reduce the size of the messages that swarms used to advertise which peers had which chunks. As it turns out, there were always more pressing things to implement and the initial implementation was good enough. (Though, really, all read-write filesystems should be copy-on-write log-based, at least for recent writes, maybe having some background process using a count-min-sketch to estimate locality for frequently read data and optimize read locality for rarely changing data that's also frequently read.) Edit: Also, it's really a shame that TCP over IPv6 doesn't use CRC-32C (to intentionally use a different CRC polynomial than Ethernet, to catch more error patterns) to end-to-end checksum data in each packet. Yes, it's a layering abstraction violation, but IPv6 was a convenient point to introduce a needed change. On the gripping hand, it's probably best in the big picture to raise flow control, corruption/loss detection, retransmission (and add forward error correction) in libraries at the application layer (a la QUIC, etc.) and move everything to UDP. I was working on Google's indexing system infra when they switched transatlantic search index distribution from multiple parallel transatlantic TCP streams to reserving dedicated bandwidth from the routers and blasting UDP using rateless forward error codes. Provided that everyone is implementing responsible (read TCP-compatible) flow control, it's really good to have the rapid evolution possible by just using UDP and raising other concerns to libraries at the application layer. (N parallel TCP streams are useful because they typically don't simultaneously hit exponential backoff, so for long-fat networks, you get both higher utilization and lower variance than a single TCP stream at N times the bandwidth.)
KMag 2y ago
How are the memory overheads of ZFS these days? In the old days, I remember balking at the extra memory required to run ZFS on the little ARM board I was using for a NAS.
- doublepg23 2y ago
  That was always FUD more or less. ZFS uses RAM as its primary cache…like every other filesystem, so it if you have very little RAM for caching the performance will degrade…like every other filesystem.
  - magicalhippo 2y ago
    > That was always FUD more or less. To give some context. ZFS support de-duplication, and until fairly recently, the de-duplication data structures had to be resident in memory. So if you used de-duplication earlier, then yes, you absolutely did need a certain amount of memory per byte stored. However, there is absolutely no requirement to use de-duplication, and without it the memory requirements are just a small, fairly fixed amount. It'll store writes in memory until it commits them in a so-called transaction group, so you need to have room for that. But the limits on a transaction group is configurable, so you can lower the defaults.
    - doublepg23 2y ago
      I don’t think I came across anyone suggesting zfs dedupe without insisting that it was effectively broken except for very specific workloads.
  - KMag 2y ago
    But if you have a single board computer with 1 GB of RAM and several TB of ZFS, will it just be slow, or actually not run? Granted, my use case was abnormal, and I was evaluating in the early days when there were both license and quality concerns with ZFS on Linux. However, my understanding at the time was that it wouldn't actually work to have several TB in a ZFS pool with 1 GB of RAM. My understanding is that ZFS has its own cache apart from the page cache, and the minimum cache size scales with the storage size. Did I misundertand/is my information outdated?

hulitu 2y ago

> the only reason ZFS isn't the norm is because we all once lived through a primordial era when it didn't exist. There were good filesystems before ZFS. I would love to have a versioning filesystem like Apollo had.

wolrah 2y ago

To paraphrase, "ZFS is the worst filesystem, except for all those other filesystems that have been tried from time to time." It's far from perfect, but it has no peers. I spent many years stubbornly using btrfs and lost data multiple times. Never once did the redundancy I had supposedly configured actually do anything to help me. ZFS has identified corruption caused by bad memory and a bad CPU and let me know immediately which files were damaged.

Too 2y ago

The reason ZFS isn't the norm is because it historically was difficult to set up. Outside of NAS solutions, it's only since Ubuntu 20.04 it has been supported out of the box on any high profile customer facing OS. The reliability of the early versions was also questionable, with high zsys cpu usage and some times arcane commands needed to rebuild pools. Anecdotally, I've had to support lots of friends with zfs issues, never so with other file systems. The data always comes back, it's just that it needs petting. Earlier, there used to be lot of fears around the license, with Torvalds advising against its use, both for that reason and for lack of maintainers. Now i believe that has been mostly ironed out and should be less of an issue.

hulitu 2y ago
> The reason ZFS isn't the norm is because it historically was difficult to set up. Has this changed ? ZFS comes with a BSD view of the world (i.e slices). It also needed a sick amount of RAM to function properly.
mustache_kimono 2y ago
> The reason ZFS isn't the norm is because it historically was difficult to set up. Outside of NAS solutions, it's only since Ubuntu 20.04 it has been supported out of the box on any high profile customer facing OS. In this one very narrow sense, we are agreed, if we are talking about Linux on root. IMHO it should also have been virtually everywhere else. It should have been in MacOS, etc. However, I think your particular comment may miss the forest for the trees. Yes, ZFS was difficult to set up for Linux, because Linux people disfavored its use (which you do touch upon later). People sometimes imagine that purely technical considerations govern the technical choices of remote groups. However, I think when people say "all tech is political" in the cultural-war-ing American politics sense, they may be right, but they are absolutely right in the small ball open source politics sense. Linux communities were convinced not to include or build ZFS support. Because licensing was a problem. Because btrfs was coming and would be better. Because Linus said ZFS was mostly marketing. So they didn't care to build support. Of course, this was all BS or FUD or NIH, but it was what happened, not that ZFS had new and different recovery tool, or was less reliable in the arbitrary past. It was because the Linux community engaged in its own (successful) FUD campaign against another FOSS project.
zvr 2y ago
Was there any change in the license that made you believe that it should be less than a issue? Or do you think people simply stopped paying attention?
- Too 2y ago
  Canonical took a team of lawyers to deeply review the license in 2016. It's beyond my legal skills to say if the conclusion made it more or less of an issue, at least the boundaries should now be more clear, for those who understand these matters more. https://canonical.com/blog/zfs-licensing-and-linux https://canonical.com/blog/zfs-licensing-and-linux https://softwarefreedom.org/resources/2016/linux-kernel-cddl.html https://softwarefreedom.org/resources/2016/linux-kernel-cddl...

KMag 2y ago

> No serious person designing a filesystem today would say it's okay to misplace your data. Former LimeWire developer here... the LimeWire splash screen at startup was due to experiences with silent data corruption. We got some impossible bug reports, so we created a stub executable that would show a splash screen while computing the SHA-1 checksums of the actual application DLLs and JARs. Once everything checked out, that stub would use Java reflection to start the actual application. After moving to that, those impossible bug reports stopped happening. With 60 million simultaneous users, there were always some of them with silent disk corruption that they would blame on LimeWire. When Microsoft was offering free Win7 pre-release install ISOs for download, I was having install issues. I didn't want to get my ISO illegally, so I found a torrent of the ISO, and wrote a Python script to download the ISO from Microsoft, but use the torrent file to verify chunks and re-download any corrupted chunks. Something was very wrong on some device between my desktop and Microsoft's servers, but it eventually got a non-corrupted ISO. It annoys me to no end that ECC isn't the norm for all devices with more than 1 GB of RAM. Silent bit flips are just not okay. Edit: side note: it's interesting to see the number of complaints I still see from people who blame hard drive failures on LimeWire stressing their drives. From very early on, LimeWire allowed bandwidth limiting, which I used to keep heat down on machines that didn't cool their drives properly. Beyond heat issues that I would blame on machine vendors, failures from write volume I would lay at the feet of drive manufacturers. Though, I'm biased. Any blame for drive wear that didn't fall on either the drive manufacturers or the filesystem implementers not dealing well with random writes would probably fall at my feet. I'm the one who implemented randomized chunk order downloading in order to rapidly increase availability of rare content, which would increase the number of hard drive head seeks on non-log-based filesystems. I always intended to go back and (1) use sequential downloads if tens of copies of the file were in the swarm, to reduce hard drive seeks and (2) implement randomized downloading of rarest chunks first, rather than the naive randomization in the initial implementation. I say naive, but the initial implementation did have some logic to randomize chunk download order in a way to reduce the size of the messages that swarms used to advertise which peers had which chunks. As it turns out, there were always more pressing things to implement and the initial implementation was good enough. (Though, really, all read-write filesystems should be copy-on-write log-based, at least for recent writes, maybe having some background process using a count-min-sketch to estimate locality for frequently read data and optimize read locality for rarely changing data that's also frequently read.) Edit: Also, it's really a shame that TCP over IPv6 doesn't use CRC-32C (to intentionally use a different CRC polynomial than Ethernet, to catch more error patterns) to end-to-end checksum data in each packet. Yes, it's a layering abstraction violation, but IPv6 was a convenient point to introduce a needed change. On the gripping hand, it's probably best in the big picture to raise flow control, corruption/loss detection, retransmission (and add forward error correction) in libraries at the application layer (a la QUIC, etc.) and move everything to UDP. I was working on Google's indexing system infra when they switched transatlantic search index distribution from multiple parallel transatlantic TCP streams to reserving dedicated bandwidth from the routers and blasting UDP using rateless forward error codes. Provided that everyone is implementing responsible (read TCP-compatible) flow control, it's really good to have the rapid evolution possible by just using UDP and raising other concerns to libraries at the application layer. (N parallel TCP streams are useful because they typically don't simultaneously hit exponential backoff, so for long-fat networks, you get both higher utilization and lower variance than a single TCP stream at N times the bandwidth.)

KMag 2y ago

How are the memory overheads of ZFS these days? In the old days, I remember balking at the extra memory required to run ZFS on the little ARM board I was using for a NAS.

doublepg23 2y ago
That was always FUD more or less. ZFS uses RAM as its primary cache…like every other filesystem, so it if you have very little RAM for caching the performance will degrade…like every other filesystem.
- magicalhippo 2y ago
  > That was always FUD more or less. To give some context. ZFS support de-duplication, and until fairly recently, the de-duplication data structures had to be resident in memory. So if you used de-duplication earlier, then yes, you absolutely did need a certain amount of memory per byte stored. However, there is absolutely no requirement to use de-duplication, and without it the memory requirements are just a small, fairly fixed amount. It'll store writes in memory until it commits them in a so-called transaction group, so you need to have room for that. But the limits on a transaction group is configurable, so you can lower the defaults.
  - doublepg23 2y ago
    I don’t think I came across anyone suggesting zfs dedupe without insisting that it was effectively broken except for very specific workloads.
- KMag 2y ago
  But if you have a single board computer with 1 GB of RAM and several TB of ZFS, will it just be slow, or actually not run? Granted, my use case was abnormal, and I was evaluating in the early days when there were both license and quality concerns with ZFS on Linux. However, my understanding at the time was that it wouldn't actually work to have several TB in a ZFS pool with 1 GB of RAM. My understanding is that ZFS has its own cache apart from the page cache, and the minimum cache size scales with the storage size. Did I misundertand/is my information outdated?