5 ms·

I wonder what ZFS in the iPhone would've looked like. As far as I recall, the iPhone didn't have error correcting memory, and ZFS is notorious for corrupting it

I wonder what ZFS in the iPhone would've looked like. As far as I recall, the iPhone didn't have error correcting memory, and ZFS is notorious for corrupting itself when bit flips hit it and break the checksum on disk. ZFS' RAM-hungry nature would've also forced Apple to add more memory to their phone.

deleted 2y ago
[deleted]
Dylan16807 2y ago
> ZFS is notorious for corrupting itself when bit flips hit it and break the checksum on disk I don't think it is. I've never heard of that happening, or seen any evidence ZFS is more likely to break than any random filesystem. I've only seen people spreading paranoid rumors based on a couple pages saying ECC memory is important to fully get the benefits of ZFS.
- thfuran 2y ago
  They also insist that you need about 10 TB RAM per TB disk space or something like that.
  - yjftsjthsd-h 2y ago
    There is a rule of thumb that you should have at least 1 GB of RAM per TB of disk when using deduplication. That's.... Different.
    - williamstein 2y ago
      Fortunately, this has significantly improved since dedup was rewritten as part of the new ZFS 2.3 release. Search for zfs “fast dedup”.
    - thfuran 2y ago
      So you've never seen the people saying you should steer clear of ZFS unless you're going to have an enormous ARC even when talking about personal media servers?
      - amarshall 2y ago
        Even then you obviously need L2ARC as well!! /s
        thfuran 2y ago
        But on optane. Because obviously you need an all flash main array for streaming a movie.
      - toast0 2y ago
        People, especially those on the Internet, say a lot of things. Some of the things they say aren't credible, even if they're said often. You don't need an enormous amount of ram to run zfs unless you have dedupe enabled. A lot of people thought they wanted dedupe enabled though. (2024's fast dedupe may help, but probably the right answer for most people is not to use dedupe) It's the same thing with the "need" for ECC. If your ram is bad, you're going to end up with bad data in your filesystem. With ZFS, you're likely to find out your filesystem is corrupt (although, if the data is corrupted before the checksum is calculated, then the checksum doesn't help); with a non-checksumming filesystem, you may get lucky and not have meta data get corrupted and the OS keeps going, just some of your files are wrong. Having ECC would be better, but there's tradeoffs so it never made sense for me to use it at home; zfs still works and is protecting me from disk contents changing, even if what was written could be wrong.
      - yjftsjthsd-h 2y ago
        Not that I recall? And it's worked fine for me...
      - ryao 2y ago
        I have seen people say such things, and none of it was based on reality. They just misinterpreted the performance cliff that data deduplication had to mean you must have absurd amounts of memory even though data deduplication is off by default. I suspect few of the people peddling this nonsense even used ZFS and the few who did, had not looked very deeply into it.
  - amarshall 2y ago
    It’s unfortunate some folks are missing the tongue-in-cheek nature of your comment.
amarshall 2y ago
> ZFS is notorious for corrupting itself when bit flips hit it and break the checksum on disk ZFS does not need or benefit from ECC memory any more than any other FS. The bitflip corrupted the data, regardless of ZFS. Any other FS is just oblivious, ZFS will at least tell you your data is corrupt but happily keep operating. > ZFS' RAM-hungry nature ZFS is not really RAM-hungry, unless one uses deduplication (which is not enabled by default, nor generally recommended). It can often seem RAM hungry on Linux because the ARC is not counted as “cache” like the page cache is. --- ZFS docs say as much as well: https://openzfs.github.io/openzfs-docs/Project%20and%20Community/FAQ.html#hardware-requirements https://openzfs.github.io/openzfs-docs/Project%20and%20Commu...
- williamstein 2y ago
  And even dedup was finally rewritten to be significantly more memory efficient, as of the new 2.3 release of ZFS: https://github.com/openzfs/zfs/discussions/15896 https://github.com/openzfs/zfs/discussions/15896
mrkeen 2y ago
> ZFS is notorious for corrupting itself when bit flips hit it and break the checksum on disk. What's a bit flip?
- zie 2y ago
  Basically it's that memory changes out from under you. As we know, computers use Binary, so everything boils down to it being a 0 or a 1. A bit flip is changing what was say a 0 into a 1. Usually attributed to "cosmic rays", but really can happen for any number of less exciting sounding reasons. Basically, there is zero double checking in your computer for almost everything except stuff that goes across the network. Memory and disks are not checked for correctness, basically ever on any machine anywhere. Many servers(but certainly not all) are the rare exception when it comes to memory safety. They usually have ECC(Error Correction Code) Memory, basically a checksum on the memory to ensure that if memory is corrupted, it's noticed and fixed. Essentially every filesystem everywhere does zero data integrity checking: MacOS APFS: Nope Windows NTFS: Nope Linux EXT4: Nope BSD's UFS: Nope Your mobile phone: Nope ZFS is the rare exception for file systems that actually double check the data you save to it is the data you get back from it. Every other filesystem is just a big ball of unknown data. You probably get back what you put it, but there is zero promises or guarantees.
  - crazygringo 2y ago
    > disks are not checked for correctness, basically ever on any machine anywhere. I'm not sure that's really accurate -- all modern hard drives and SSD's use error-correcting codes, as far as I know. That's different from implementing additional integrity checking at the filesystem level. But it's definitely there to begin with.
    - tpetry 2y ago
      But SSDs (to my knowledge) only implement checksum for the data transfer. Its a requirement of the protocol. So you can be sure that the Stuff in memory and checksum computed by the CPU arrives exactly like that in the SSD driver. In the past this was a common error source with hardware raid which was faulty. But there is ABSOLUTELY NO checksum for the bits stored on a SSD. So bit rot at the cells of the SSDs are undetected.
terlisimo 2y ago
> ZFS is notorious for corrupting itself when bit flips That is a notorious myth. https://jrs-s.net/2015/02/03/will-zfs-and-non-ecc-ram-kill-your-data/ https://jrs-s.net/2015/02/03/will-zfs-and-non-ecc-ram-kill-y...
ahl 2y ago
It's very amusing that this kind of legend has persisted! ZFS is notorious for *noticing* when bits flip, something APFS designers claimed was rare given the robustness of Apple hardware.[1][2] What would ZFS on iPhone have looked like? Hard to know, and that certainly wasn't the design center. Neither here nor there, but DTrace was ported to iPhone--it was shown to me in hushed tones in the back of an auditorium once... [1]: https://arstechnica.com/gadgets/2016/06/a-zfs-developers-analysis-of-the-good-and-bad-in-apples-new-apfs-file-system/#toc-h10 https://arstechnica.com/gadgets/2016/06/a-zfs-developers-ana... [2]: https://ahl.dtrace.org/2016/06/19/apfs-part5/#checksums https://ahl.dtrace.org/2016/06/19/apfs-part5/#checksums
- ryao 2y ago
  I did early ZFSOnLinux development on hardware that did not have ECC memory. I once had a situation where a bit flip happened in the ARC buffer for libpython.so and all python software started crashing. Initially, I thought I had hit some sort of blizzard bug in ZFS, so I started debugging. At that time, opening a ZFS snapshot would fetch a duplicate from disk into a redundant ARC buffer, so while debugging, I ran cmp on libpython.so between the live copy and a snapshot copy. It showed the exact bit that had flipped. After seeing that and convincing myself the bitflip was not actually on stable storage, I did a reboot, and all was well. Soon afterward, I got a new development machine that had ECC so that I would not waste my time chasing phantom bugs caused by bit flips.
  - redeeman 2y ago
    erhm, isnt zfs supposed to store the checksum of records stored in the arc, and verify on read? are you sure this is what happened?
Modified3019 2y ago
ZFS detects corruption. A very long ago someone named cyberjock was a prolific and opinionated proponent of ZFS, who wrote many things about ZFS during a time when the hobbyist community was tiny and not very familiar with how to use it and how it worked. Unfortunately, some of their most misguided and/or outdated thoughts still haunt modern consciousness like an egregore. What you are probably thinking of is the proposed doomsday scenario where bad ram could theoretically kill a ZFS pool during a scrub. This article does a good job of explaining how that might happen, and why being concerned about it is tilting at windmills: https://jrs-s.net/2015/02/03/will-zfs-and-non-ecc-ram-kill-your-data/ https://jrs-s.net/2015/02/03/will-zfs-and-non-ecc-ram-kill-y... I have never once heard of this happening in real life. Hell, I’ve never even had bad ram. I have had bad sata/sas cables, and a bad disk though. ZFS faithfully informed me there was a problem, which no other file system would have done. I’ve seen other people that start getting corruption when sata/sas controllers go bad or overheat, which again is detected by ZFS. What actually destroys pools is user error, followed very distantly by plain old fashioned ZFS bugs that someone with an unlucky edge case ran into.
- tmoertel 2y ago
  > Hell, I’ve never even had bad ram. To what degree can you separate this claim from "I've never noticed RAM failures"?
  - wtallis 2y ago
    It isn't hard to run memtest on all your computers, and that will catch the kind of bad RAM that the aforementioned doomsday scenario requires.
  - Modified3019 2y ago
    You can take that as meaning “I’ve never had a noticed issue that was detected by extensive ram testing, or solved by replacing ram”. I got into overclocking both regular and ECC DDR4 ram for a while when AMD’s 1st gen ryzen stuff came out, thanks to asrock’s x399 motherboard which unofficially supporting ECC, allowing both it’s function and reporting of errors (produced when overlocking) Based on my own testing and issues seen from others, regular memory has quite a bit of leeway before it becomes unstable, and memory that’s generating errors tends to constantly crash the system, or do so under certain workloads. Of course, without ECC you can’t prove every single operation has been fault free, but as some point you call it close enough. I am of the opinion that ECC memory is the best memory to overclock, precisely because you can prove stability simply by using the system. All that said, as things become smaller with tighter specifications to squeeze out faster performance, I do grow more leery of intermittent single errors that occur on the order of weeks or months in newer generations of hardware. I was once able to overclock my memory to the edge of what I thought was stability as it passed all tests for days, but about every month or two there’d be a few corrected errors show up in my logs. Typically, any sort of stability is caught by manual tests within minutes or the hour.
E39M5S62 2y ago
Not this myth again. ZFS does not need ECC RAM. Stop propagating this falsehood.
cryptonector 2y ago
> ZFS is notorious for corrupting itself when bit flips hit it and break the checksum on disk. If you have no mirrors and no raidz and no ditto blocks then errors cause problems, yes. Early on they would cause panics. But this isn't ZFS "corrupting itself", rather, it's ZFS saving itself and you from corruption, and the price you pay for that is that you need to add redundancy (mirrors, raidz, or ditto blocks). It's not a bad deal. Some prefer not to know.

deleted 2y ago

[deleted]

Dylan16807 2y ago

> ZFS is notorious for corrupting itself when bit flips hit it and break the checksum on disk I don't think it is. I've never heard of that happening, or seen any evidence ZFS is more likely to break than any random filesystem. I've only seen people spreading paranoid rumors based on a couple pages saying ECC memory is important to fully get the benefits of ZFS.

thfuran 2y ago
They also insist that you need about 10 TB RAM per TB disk space or something like that.
- yjftsjthsd-h 2y ago
  There is a rule of thumb that you should have at least 1 GB of RAM per TB of disk when using deduplication. That's.... Different.
  - williamstein 2y ago
    Fortunately, this has significantly improved since dedup was rewritten as part of the new ZFS 2.3 release. Search for zfs “fast dedup”.
  - thfuran 2y ago
    So you've never seen the people saying you should steer clear of ZFS unless you're going to have an enormous ARC even when talking about personal media servers?
    - amarshall 2y ago
      Even then you obviously need L2ARC as well!! /s
      - thfuran 2y ago
        But on optane. Because obviously you need an all flash main array for streaming a movie.
    - toast0 2y ago
      People, especially those on the Internet, say a lot of things. Some of the things they say aren't credible, even if they're said often. You don't need an enormous amount of ram to run zfs unless you have dedupe enabled. A lot of people thought they wanted dedupe enabled though. (2024's fast dedupe may help, but probably the right answer for most people is not to use dedupe) It's the same thing with the "need" for ECC. If your ram is bad, you're going to end up with bad data in your filesystem. With ZFS, you're likely to find out your filesystem is corrupt (although, if the data is corrupted before the checksum is calculated, then the checksum doesn't help); with a non-checksumming filesystem, you may get lucky and not have meta data get corrupted and the OS keeps going, just some of your files are wrong. Having ECC would be better, but there's tradeoffs so it never made sense for me to use it at home; zfs still works and is protecting me from disk contents changing, even if what was written could be wrong.
    - yjftsjthsd-h 2y ago
      Not that I recall? And it's worked fine for me...
    - ryao 2y ago
      I have seen people say such things, and none of it was based on reality. They just misinterpreted the performance cliff that data deduplication had to mean you must have absurd amounts of memory even though data deduplication is off by default. I suspect few of the people peddling this nonsense even used ZFS and the few who did, had not looked very deeply into it.
- amarshall 2y ago
  It’s unfortunate some folks are missing the tongue-in-cheek nature of your comment.

amarshall 2y ago

> ZFS is notorious for corrupting itself when bit flips hit it and break the checksum on disk ZFS does not need or benefit from ECC memory any more than any other FS. The bitflip corrupted the data, regardless of ZFS. Any other FS is just oblivious, ZFS will at least tell you your data is corrupt but happily keep operating. > ZFS' RAM-hungry nature ZFS is not really RAM-hungry, unless one uses deduplication (which is not enabled by default, nor generally recommended). It can often seem RAM hungry on Linux because the ARC is not counted as “cache” like the page cache is. --- ZFS docs say as much as well: https://openzfs.github.io/openzfs-docs/Project%20and%20Community/FAQ.html#hardware-requirements https://openzfs.github.io/openzfs-docs/Project%20and%20Commu...

williamstein 2y ago
And even dedup was finally rewritten to be significantly more memory efficient, as of the new 2.3 release of ZFS: https://github.com/openzfs/zfs/discussions/15896 https://github.com/openzfs/zfs/discussions/15896

mrkeen 2y ago

> ZFS is notorious for corrupting itself when bit flips hit it and break the checksum on disk. What's a bit flip?

zie 2y ago
Basically it's that memory changes out from under you. As we know, computers use Binary, so everything boils down to it being a 0 or a 1. A bit flip is changing what was say a 0 into a 1. Usually attributed to "cosmic rays", but really can happen for any number of less exciting sounding reasons. Basically, there is zero double checking in your computer for almost everything except stuff that goes across the network. Memory and disks are not checked for correctness, basically ever on any machine anywhere. Many servers(but certainly not all) are the rare exception when it comes to memory safety. They usually have ECC(Error Correction Code) Memory, basically a checksum on the memory to ensure that if memory is corrupted, it's noticed and fixed. Essentially every filesystem everywhere does zero data integrity checking: MacOS APFS: Nope Windows NTFS: Nope Linux EXT4: Nope BSD's UFS: Nope Your mobile phone: Nope ZFS is the rare exception for file systems that actually double check the data you save to it is the data you get back from it. Every other filesystem is just a big ball of unknown data. You probably get back what you put it, but there is zero promises or guarantees.
- crazygringo 2y ago
  > disks are not checked for correctness, basically ever on any machine anywhere. I'm not sure that's really accurate -- all modern hard drives and SSD's use error-correcting codes, as far as I know. That's different from implementing additional integrity checking at the filesystem level. But it's definitely there to begin with.
  - tpetry 2y ago
    But SSDs (to my knowledge) only implement checksum for the data transfer. Its a requirement of the protocol. So you can be sure that the Stuff in memory and checksum computed by the CPU arrives exactly like that in the SSD driver. In the past this was a common error source with hardware raid which was faulty. But there is ABSOLUTELY NO checksum for the bits stored on a SSD. So bit rot at the cells of the SSDs are undetected.

terlisimo 2y ago

> ZFS is notorious for corrupting itself when bit flips That is a notorious myth. https://jrs-s.net/2015/02/03/will-zfs-and-non-ecc-ram-kill-your-data/ https://jrs-s.net/2015/02/03/will-zfs-and-non-ecc-ram-kill-y...

ahl 2y ago

It's very amusing that this kind of legend has persisted! ZFS is notorious for *noticing* when bits flip, something APFS designers claimed was rare given the robustness of Apple hardware.[1][2] What would ZFS on iPhone have looked like? Hard to know, and that certainly wasn't the design center. Neither here nor there, but DTrace was ported to iPhone--it was shown to me in hushed tones in the back of an auditorium once... [1]: https://arstechnica.com/gadgets/2016/06/a-zfs-developers-analysis-of-the-good-and-bad-in-apples-new-apfs-file-system/#toc-h10 https://arstechnica.com/gadgets/2016/06/a-zfs-developers-ana... [2]: https://ahl.dtrace.org/2016/06/19/apfs-part5/#checksums https://ahl.dtrace.org/2016/06/19/apfs-part5/#checksums

ryao 2y ago
I did early ZFSOnLinux development on hardware that did not have ECC memory. I once had a situation where a bit flip happened in the ARC buffer for libpython.so and all python software started crashing. Initially, I thought I had hit some sort of blizzard bug in ZFS, so I started debugging. At that time, opening a ZFS snapshot would fetch a duplicate from disk into a redundant ARC buffer, so while debugging, I ran cmp on libpython.so between the live copy and a snapshot copy. It showed the exact bit that had flipped. After seeing that and convincing myself the bitflip was not actually on stable storage, I did a reboot, and all was well. Soon afterward, I got a new development machine that had ECC so that I would not waste my time chasing phantom bugs caused by bit flips.
- redeeman 2y ago
  erhm, isnt zfs supposed to store the checksum of records stored in the arc, and verify on read? are you sure this is what happened?

Modified3019 2y ago

ZFS detects corruption. A very long ago someone named cyberjock was a prolific and opinionated proponent of ZFS, who wrote many things about ZFS during a time when the hobbyist community was tiny and not very familiar with how to use it and how it worked. Unfortunately, some of their most misguided and/or outdated thoughts still haunt modern consciousness like an egregore. What you are probably thinking of is the proposed doomsday scenario where bad ram could theoretically kill a ZFS pool during a scrub. This article does a good job of explaining how that might happen, and why being concerned about it is tilting at windmills: https://jrs-s.net/2015/02/03/will-zfs-and-non-ecc-ram-kill-your-data/ https://jrs-s.net/2015/02/03/will-zfs-and-non-ecc-ram-kill-y... I have never once heard of this happening in real life. Hell, I’ve never even had bad ram. I have had bad sata/sas cables, and a bad disk though. ZFS faithfully informed me there was a problem, which no other file system would have done. I’ve seen other people that start getting corruption when sata/sas controllers go bad or overheat, which again is detected by ZFS. What actually destroys pools is user error, followed very distantly by plain old fashioned ZFS bugs that someone with an unlucky edge case ran into.

tmoertel 2y ago
> Hell, I’ve never even had bad ram. To what degree can you separate this claim from "I've never noticed RAM failures"?
- wtallis 2y ago
  It isn't hard to run memtest on all your computers, and that will catch the kind of bad RAM that the aforementioned doomsday scenario requires.
- Modified3019 2y ago
  You can take that as meaning “I’ve never had a noticed issue that was detected by extensive ram testing, or solved by replacing ram”. I got into overclocking both regular and ECC DDR4 ram for a while when AMD’s 1st gen ryzen stuff came out, thanks to asrock’s x399 motherboard which unofficially supporting ECC, allowing both it’s function and reporting of errors (produced when overlocking) Based on my own testing and issues seen from others, regular memory has quite a bit of leeway before it becomes unstable, and memory that’s generating errors tends to constantly crash the system, or do so under certain workloads. Of course, without ECC you can’t prove every single operation has been fault free, but as some point you call it close enough. I am of the opinion that ECC memory is the best memory to overclock, precisely because you can prove stability simply by using the system. All that said, as things become smaller with tighter specifications to squeeze out faster performance, I do grow more leery of intermittent single errors that occur on the order of weeks or months in newer generations of hardware. I was once able to overclock my memory to the edge of what I thought was stability as it passed all tests for days, but about every month or two there’d be a few corrected errors show up in my logs. Typically, any sort of stability is caught by manual tests within minutes or the hour.

E39M5S62 2y ago

Not this myth again. ZFS does not need ECC RAM. Stop propagating this falsehood.

cryptonector 2y ago

> ZFS is notorious for corrupting itself when bit flips hit it and break the checksum on disk. If you have no mirrors and no raidz and no ditto blocks then errors cause problems, yes. Early on they would cause panics. But this isn't ZFS "corrupting itself", rather, it's ZFS saving itself and you from corruption, and the price you pay for that is that you need to add redundancy (mirrors, raidz, or ditto blocks). It's not a bad deal. Some prefer not to know.