5 ms·

Can people here please stop posting that ZFS needs ECC memory. Every filesystem, with any name like FAT, NTFS, EXT4 runs more safe with ECC memory. ZFS is actua

by olavgg 9y ago

Can people here please stop posting that ZFS needs ECC memory. Every filesystem, with any name like FAT, NTFS, EXT4 runs more safe with ECC memory. ZFS is actually one of the few that can still be safer if you don't run with ECC memory. Source: Matthew Ahrens himself: https://arstechnica.com/civis/viewtopic.php?f=2&t=1235679&p=26303271#p26303271 https://arstechnica.com/civis/viewtopic.php?f=2&t=1235679&p=...

lomnakkus 9y ago
Indeed. It's true that the data may be corrupted before hitting any disk[1], but once it has hit the disks (>1), it's extremely unlikely that you'll ever hit a similar bit error where it'll mistakenly choose the wrong disk block to recover from. The main point of e.g. ZFS or Btrfs checksumming is that a) at least it isn't getting worse, and b) I can tell if it's getting worse. [1] ... but if the bits are not generated by the machine that actually saving to disk, how do you know they weren't corrupted along the way? The number of people who religiously check PGP signatures/SHA256sums or whatever is miniscule.
- ruleabidinguser 9y ago
  What are you doing where you're actually checking checksums periodically and detcting when things get worse? That seems like a lot of work to set up.
  - boomboomsubban 9y ago
    They are using ZFS, scrubbing is one command.
  - lomnakkus 9y ago
    zpool scrub (This may be a myth: It's not something you should actually do that often because actually reading the media may dregrade it.)
    - derefr 9y ago
      Either way, it'd be fine on SSDs, then?
      - lomnakkus 9y ago
        Hm? As long as the error correction technology on your chosen SSD of choice stands up, I guess... yes? What, exactly, are you asking?
        derefr 9y ago
        My thought was "reading SSDs doesn't degrade them, so there's no disadvantage to constant scrubbing." Unless I've misunderstood what you mean by "reading."
        lomnakkus 9y ago
        Ah, right, fair point. AFAIUI SSD storage does degrade a tiny(!) amount when reading, but magnetic storage degrades quite a bit more. (That's where that came from, draw your own conlusions :).)
    - kscz 9y ago
      or for btrfs users out there: btrfs scrub start /mnt/volume_name
- derefr 9y ago
  > The number of people who religiously check PGP signatures/SHA256sums or whatever is miniscule. • If you transfer things around using BitTorrent, it'll ensure you always end up with a file that hashes correctly to the sum it originally had when the .torrent file was constructed. • Many archive formats (zip, rar, and 7z, at least) contain checksums, and archival utilities validate those checksums during extraction, refusing to extract broken files. "Self-extracting archive" executables that use these formats inherit this property. • Some common disk-image formats (dmg, wim) embed a checksum that checks the whole disk-image during mount, and will refuse to mount a bad one. (I believe you can then try to "repair" the disk image with your OS's disk-repair utility, if you have no other copies.) • Web pages increasingly use Sub-Resource Integrity attributes on things like .css and .js files, protecting them (though not the page itself) from errors. • ISO files don't embed checks, but all the common package formats (Windows .cab and .msi; Linux .deb and .rpm; macOS .pkg) on installer ISOs embed their own checksums and often signatures. • git repos are 'protected' insofar as you won't be able to sync mis-hashed objects from a remote, so they won't spread. Really, looking over all that, it's only 1. plain binary executables, and 2. "media files" (images, audio, video)—and only when retrieved over a "dumb" protocol, rather than a pre-baked-manifest protocol like BitTorrent or zsync—that are "risky" and in need of explicit checksum comparison.
  - lomnakkus 9y ago
    True. Still mostly immaterial in the grand scheme of things, I think? EDIT: Should say, as a point of interest: Even though .zip's were protected back in the Good Old Days, that didn't really matter because we all got corrupted (expanded from .zip) .mp3's because of those fucking RTL3xxsomething cards that would just transmit things perfectly and then corrupt the checksum (or whichever way 'round). Ugh. One of the few times I've actually hated engineers. (Don't get me wrong. I really do want more of these checks to be pervasive. We start with our local file systems.)
  - comex 9y ago
    Both macOS/iOS and Windows use code signing for executables, which should guard against most types of corruption. Web pages (and anything else) transmitted over HTTPS are protected from corruption in transit by TLS's hashing (which is vastly stronger than the checksums at lower levels of the network stack), though that doesn't help if the server has faulty memory or storage. PNG has built-in checksums, though other image formats don't (JPEG). Not sure about video.
X86BSD 9y ago
It's not the that it NEEDS it, it's that if you DONT use it you are introducing potential data errors into an otherwise checksummed data path. Which would completely negate the rest of the path.
gbrown_ 9y ago
I'm so glad to see this comment high up.
kurlberg 9y ago
In an old discussion regarding ECC/ZFS (in particular, whether hitting bad RAM while scrubbing could corrupt more and more data), user XorNot kindly took a look the ZFS source and wrote "In fact I'm looking at the RAID-Z code right now. This scenario would be literally impossible because the code keeps everything read from the disk in memory in separate buffers - i.e. reconstructed data and bad data do not occupy or reuse the same memory space, and are concurrently allocated. The parity data is itself checksummed, as ZFS assumes it might be reading bad parity by default." His full comment can be found here: https://news.ycombinator.com/item?id=8294434 https://news.ycombinator.com/item?id=8294434
koffiezet 9y ago
ZFS is not the cheap option, it was never intended to be - so why bother skimping on ECC? If you're worried about ECC prices - ZFS is probably not for you. It's not that you need ECC for ZFS, but when you're at a point where you're willing to throw money at a storage system where ZFS makes sense, the extra cost of ECC is minuscule. The most expensive hardware requirement of ZFS is that you need disks of the same size anyway, which means you're not just throwing a random amount of disks together, and if you want to expand, you need to add another full zpool, or replace disks one by one. On my home NAS, the difference was about 120 EUR for 32GB (80eur/dimm vs 50eur/dimm), on a grand total of over 2500 EUR. One of the reasons for choosing ZFS was storage reliability, and then skimping out on ECC is a imho a bit silly.
- GuB-42 9y ago
  You can have disks of differing sizes with ZFS, though you are making things difficult for yourself so your point still stands. However the cost of ECC is not negligible because you need your CPU and motherboard to support it. The total budget for my NAS was 1000 EUR in 2013, 500 for the 5 3TB disks, 350 for the motherboard+CPU+8GB ECC RAM, and 150 for the case, PSU, system SSD, and accessories. In reality I salvaged 2 3TB disks, lowering the cost to 800. By using non-ECC I could have used a cheaper motherboard and CPU, in addition to cheaper RAM. In fact I would probably have used hardware from an older desktop PC. It would have been a 15-20% saving, or 45% if I take reuse into account. Not negligible. My previous NAS, running linux soft-RAID, entirely made of salvaged parts except for some of the disks had a few corruption problems. One of them caused by a defective disk. ZFS would have caught it, so even on cheap systems, ZFS has its use. I also had defective DRAM, rebuilds not going smoothly, etc... That system caused me too many scares, so I decided that the next system would be cheap but not too cheap as to endanger my sanity. I also got a proper backup solution.
tjoff 9y ago
No. ZFS is in much greater need for ECC than most other filesystems. 1. ZFS doesn't come with any disk repair tools and the ones that exist are not nearly as capable as for other filesystems (the ZFS motto is that it is too costly to repair filesystems, just recover from tape instead (here we can sense the intended audience of ZFS)). If the wrong bit get's flipped your entire pool might be gone (you can of course spend months of your spare time to debug it yourself if you want to). This is not the case (to the same extent) for FAT, NTFS or EXT. 2. The more you use memory the more likely you are about to get hit. I'd argue that ZFS is a quite resource heavy filesystem and is thus more likely to actually attract bit flips. This is similar as to when using an encrypted filesystem on an overclocked CPU. There is nothing inherently more risky with encrypting your filesystem with an overclocked CPU - but overclocking your CPU increases the risk for miscalculations. And enabling encryption increases the CPU usage when accessing the filesystem by several orders of magnitude. So, in practice you quickly notice how filesystem data on encrypted drives get corrupted but not on regular drives on an ever so slightly too overclocked machine. So, if you care about your filesystem, then yes - saying that ZFS needs ECC is quite sensible. (if you care about your data you should have backups regardless)
- cyberpunk 9y ago
  Well; I don't think 'lack of repair tools' for ZFS is the reason that ECC is the suggested good practice; but I can sort of agree that recovering from a badly farked pool isn't fun having been down that rabbithole... Regardless of ZFS, this is why we architect our storage to cope with such potential borkage (which has, incidentially, only happened to me _ONCE_ in ~8 years of zfs in prod, had nothing to do with silent md corruption in ram and had everything to do with a nasty hw sas hba bug) -- If losing a single storage node/pool causes you problems then you're "doing it wrong" (sorry) and it makes no difference if you're using XFS, ZFS, BTRFS or whatever else... So... I'm not sure about the ECC stuff, to me ECC really matters not at all for the simple reason that any significant deployment is using ECC anyway: even deploying a cheapo 10k pair of jbods and a tiny 1U head to run NFS or something, you'll be unlikely to even have the option of non ECC from whoever you're buying the kit from (dell? hp? bla) right? Yes, it might work without it. It might work better somehow with it.. What does it matter when even the cheap gear comes with it anyway? I've built a reasonable slew of ZFS backed storage (well, a 10 PB prod or so anyway, nothing compared to what some of the folks who comment here have done) and besides some hardware compat issues if you're building storage this is currently your best option to back your objstore/dfs. ZFS as the storage backend for your DC/Cloud? Good pick. ZFS as the 'local' FS's in your VMs? I wouldn't bother, unless you need some features (it works pretty well with docker, as it goes, but I prefer to run apps on plain ext backed by zvols instead of 'zfs-on-zfs')....

by olavgg 9y ago

lomnakkus 9y ago

Indeed. It's true that the data may be corrupted before hitting any disk[1], but once it has hit the disks (>1), it's extremely unlikely that you'll ever hit a similar bit error where it'll mistakenly choose the wrong disk block to recover from. The main point of e.g. ZFS or Btrfs checksumming is that a) at least it isn't getting worse, and b) I can tell if it's getting worse. [1] ... but if the bits are not generated by the machine that actually saving to disk, how do you know they weren't corrupted along the way? The number of people who religiously check PGP signatures/SHA256sums or whatever is miniscule.

ruleabidinguser 9y ago
What are you doing where you're actually checking checksums periodically and detcting when things get worse? That seems like a lot of work to set up.
- boomboomsubban 9y ago
  They are using ZFS, scrubbing is one command.
- lomnakkus 9y ago
  zpool scrub (This may be a myth: It's not something you should actually do that often because actually reading the media may dregrade it.)
  - derefr 9y ago
    Either way, it'd be fine on SSDs, then?
    - lomnakkus 9y ago
      Hm? As long as the error correction technology on your chosen SSD of choice stands up, I guess... yes? What, exactly, are you asking?
      - derefr 9y ago
        My thought was "reading SSDs doesn't degrade them, so there's no disadvantage to constant scrubbing." Unless I've misunderstood what you mean by "reading."
        lomnakkus 9y ago
        Ah, right, fair point. AFAIUI SSD storage does degrade a tiny(!) amount when reading, but magnetic storage degrades quite a bit more. (That's where that came from, draw your own conlusions :).)
  - kscz 9y ago
    or for btrfs users out there: btrfs scrub start /mnt/volume_name
derefr 9y ago
> The number of people who religiously check PGP signatures/SHA256sums or whatever is miniscule. • If you transfer things around using BitTorrent, it'll ensure you always end up with a file that hashes correctly to the sum it originally had when the .torrent file was constructed. • Many archive formats (zip, rar, and 7z, at least) contain checksums, and archival utilities validate those checksums during extraction, refusing to extract broken files. "Self-extracting archive" executables that use these formats inherit this property. • Some common disk-image formats (dmg, wim) embed a checksum that checks the whole disk-image during mount, and will refuse to mount a bad one. (I believe you can then try to "repair" the disk image with your OS's disk-repair utility, if you have no other copies.) • Web pages increasingly use Sub-Resource Integrity attributes on things like .css and .js files, protecting them (though not the page itself) from errors. • ISO files don't embed checks, but all the common package formats (Windows .cab and .msi; Linux .deb and .rpm; macOS .pkg) on installer ISOs embed their own checksums and often signatures. • git repos are 'protected' insofar as you won't be able to sync mis-hashed objects from a remote, so they won't spread. Really, looking over all that, it's only 1. plain binary executables, and 2. "media files" (images, audio, video)—and only when retrieved over a "dumb" protocol, rather than a pre-baked-manifest protocol like BitTorrent or zsync—that are "risky" and in need of explicit checksum comparison.
- lomnakkus 9y ago
  True. Still mostly immaterial in the grand scheme of things, I think? EDIT: Should say, as a point of interest: Even though .zip's were protected back in the Good Old Days, that didn't really matter because we all got corrupted (expanded from .zip) .mp3's because of those fucking RTL3xxsomething cards that would just transmit things perfectly and then corrupt the checksum (or whichever way 'round). Ugh. One of the few times I've actually hated engineers. (Don't get me wrong. I really do want more of these checks to be pervasive. We start with our local file systems.)
- comex 9y ago
  Both macOS/iOS and Windows use code signing for executables, which should guard against most types of corruption. Web pages (and anything else) transmitted over HTTPS are protected from corruption in transit by TLS's hashing (which is vastly stronger than the checksums at lower levels of the network stack), though that doesn't help if the server has faulty memory or storage. PNG has built-in checksums, though other image formats don't (JPEG). Not sure about video.

X86BSD 9y ago

It's not the that it NEEDS it, it's that if you DONT use it you are introducing potential data errors into an otherwise checksummed data path. Which would completely negate the rest of the path.

gbrown_ 9y ago

I'm so glad to see this comment high up.

kurlberg 9y ago

In an old discussion regarding ECC/ZFS (in particular, whether hitting bad RAM while scrubbing could corrupt more and more data), user XorNot kindly took a look the ZFS source and wrote "In fact I'm looking at the RAID-Z code right now. This scenario would be literally impossible because the code keeps everything read from the disk in memory in separate buffers - i.e. reconstructed data and bad data do not occupy or reuse the same memory space, and are concurrently allocated. The parity data is itself checksummed, as ZFS assumes it might be reading bad parity by default." His full comment can be found here: https://news.ycombinator.com/item?id=8294434 https://news.ycombinator.com/item?id=8294434

koffiezet 9y ago

ZFS is not the cheap option, it was never intended to be - so why bother skimping on ECC? If you're worried about ECC prices - ZFS is probably not for you. It's not that you need ECC for ZFS, but when you're at a point where you're willing to throw money at a storage system where ZFS makes sense, the extra cost of ECC is minuscule. The most expensive hardware requirement of ZFS is that you need disks of the same size anyway, which means you're not just throwing a random amount of disks together, and if you want to expand, you need to add another full zpool, or replace disks one by one. On my home NAS, the difference was about 120 EUR for 32GB (80eur/dimm vs 50eur/dimm), on a grand total of over 2500 EUR. One of the reasons for choosing ZFS was storage reliability, and then skimping out on ECC is a imho a bit silly.

GuB-42 9y ago
You can have disks of differing sizes with ZFS, though you are making things difficult for yourself so your point still stands. However the cost of ECC is not negligible because you need your CPU and motherboard to support it. The total budget for my NAS was 1000 EUR in 2013, 500 for the 5 3TB disks, 350 for the motherboard+CPU+8GB ECC RAM, and 150 for the case, PSU, system SSD, and accessories. In reality I salvaged 2 3TB disks, lowering the cost to 800. By using non-ECC I could have used a cheaper motherboard and CPU, in addition to cheaper RAM. In fact I would probably have used hardware from an older desktop PC. It would have been a 15-20% saving, or 45% if I take reuse into account. Not negligible. My previous NAS, running linux soft-RAID, entirely made of salvaged parts except for some of the disks had a few corruption problems. One of them caused by a defective disk. ZFS would have caught it, so even on cheap systems, ZFS has its use. I also had defective DRAM, rebuilds not going smoothly, etc... That system caused me too many scares, so I decided that the next system would be cheap but not too cheap as to endanger my sanity. I also got a proper backup solution.

tjoff 9y ago

No. ZFS is in much greater need for ECC than most other filesystems. 1. ZFS doesn't come with any disk repair tools and the ones that exist are not nearly as capable as for other filesystems (the ZFS motto is that it is too costly to repair filesystems, just recover from tape instead (here we can sense the intended audience of ZFS)). If the wrong bit get's flipped your entire pool might be gone (you can of course spend months of your spare time to debug it yourself if you want to). This is not the case (to the same extent) for FAT, NTFS or EXT. 2. The more you use memory the more likely you are about to get hit. I'd argue that ZFS is a quite resource heavy filesystem and is thus more likely to actually attract bit flips. This is similar as to when using an encrypted filesystem on an overclocked CPU. There is nothing inherently more risky with encrypting your filesystem with an overclocked CPU - but overclocking your CPU increases the risk for miscalculations. And enabling encryption increases the CPU usage when accessing the filesystem by several orders of magnitude. So, in practice you quickly notice how filesystem data on encrypted drives get corrupted but not on regular drives on an ever so slightly too overclocked machine. So, if you care about your filesystem, then yes - saying that ZFS needs ECC is quite sensible. (if you care about your data you should have backups regardless)

cyberpunk 9y ago
Well; I don't think 'lack of repair tools' for ZFS is the reason that ECC is the suggested good practice; but I can sort of agree that recovering from a badly farked pool isn't fun having been down that rabbithole... Regardless of ZFS, this is why we architect our storage to cope with such potential borkage (which has, incidentially, only happened to me _ONCE_ in ~8 years of zfs in prod, had nothing to do with silent md corruption in ram and had everything to do with a nasty hw sas hba bug) -- If losing a single storage node/pool causes you problems then you're "doing it wrong" (sorry) and it makes no difference if you're using XFS, ZFS, BTRFS or whatever else... So... I'm not sure about the ECC stuff, to me ECC really matters not at all for the simple reason that any significant deployment is using ECC anyway: even deploying a cheapo 10k pair of jbods and a tiny 1U head to run NFS or something, you'll be unlikely to even have the option of non ECC from whoever you're buying the kit from (dell? hp? bla) right? Yes, it might work without it. It might work better somehow with it.. What does it matter when even the cheap gear comes with it anyway? I've built a reasonable slew of ZFS backed storage (well, a 10 PB prod or so anyway, nothing compared to what some of the folks who comment here have done) and besides some hardware compat issues if you're building storage this is currently your best option to back your objstore/dfs. ZFS as the storage backend for your DC/Cloud? Good pick. ZFS as the 'local' FS's in your VMs? I wouldn't bother, unless you need some features (it works pretty well with docker, as it goes, but I prefer to run apps on plain ext backed by zvols instead of 'zfs-on-zfs')....