13 ms·
Please Use ZFS with ECC Memory (2014)
- jqpabc123 4y agoSounds like the article should have been titled, "Why you shouldn't use ZFS". ZFS does not have such [recovery] tools, if the pool is corrupt, all data must be considered lost, there is no option for recovery. In other words, this marvelous piece of over engineered technology is fragile. When it works, it's great. When it fails, it's a complete disaster. Everything fails eventually. How do you prefer your failure served? In small manageable increments or one spectacular, complete, overwhelming, unrecoverable helping.
- Dork1234 4y agoActually this is a good thing. If you have a failure you want to fix the faulty device and restore from backup. The biggest problem is having a failure, and then overwriting the good backup with failed data. I have had this happen to me, and have since switched to ZFS for storage I want to keep, with multiple backups.
- jqpabc123 4y agowith multiple backups Multiple rotating backups would have solved your problem before ZFS.
- Wowfunhappy 4y agoUnless you can somehow make backups every few seconds, restoring from a backup still means loosing data. It's just highly preferable to loosing all data. Also, on a non-checksummed filesystem, how will you know whether a file needs to be restored from the backup?
- freemint 4y ago> Also, on a non-checksummed filesystem, how will you know whether a file needs to be restored from the backup? Has same edit epoch? If yes skip. If no compute checksum of both and compare, copy over in case they differ. I am fully aware of the flaws of this.
- mustache_kimono 4y agoI think it's important to note this wasn't written by an expert on ZFS. And I'm pretty certain this author is actually wrong about this particular point, as ZFS does allow a number of extraordinary measures to recover a pool from a corrupted state, such as: zpool import -FX mypool, where: 1) -F Attempt rewind if necessary. 2) -X Turn on extreme rewind. 3) -T Specify a starting txg to use for import. There are a few other points you might consider: First, what kind of failure might cause an unrecoverable corrupted pool, and how useful are other filesystems data recovery tools actually, if corrupted in a similar fashion? Is there an apples to apples comparison that you can share? I believe the author and you are sharing speculation. Second, how do you know you have corruption with another filesystem until you read back the data? And do you even know you have corruption then?
- Avamander 4y agoSure, those options are nice, but they use the same code paths as regular reading/mounting does. If you've got some corruption that causes a crash, *all zfs tools are useless*. You must whip out some idiotic 13yo Python script to destroy uberblocks and you might make it work. Usually bug => corruption => module crashes => no zfs own tools work.
- mustache_kimono 4y agoFirst, the author is wrong by your own terms. If you think a 13yo Python script is the state of the art for ZFS recovery, that still doesn't comport with what the author said ("all data must be considered lost, there is no option for recovery"). The author is facially wrong. If you had said "I once had a corrupt pool I couldn't recover", that's a useful data point. But instead you tied your wagon to someone who obviously doesn't know much about what they are talking about. Second, you managed to ignore my two other original points -- 1) Is the state of other filesystems any better re: recovery in similar circumstances (show your work, when XFS is afflicted with the exact same type of corruption, how is it better)?, and 2) is it possible are you better positioned to deal with corruption with ZFS (because it's usually not silent)? I'm not saying ZFS is better for you. You are obviously not predisposed. What I am saying is -- your argument has to be better supported for it to make any sense.
- getcrunk 4y agoSeriously outside of very specific use cases I think there is a major bandwagoning with zfs. Using snapraid for larger unchanging data or normal raid without striping for vms or dbs allows your recovery story to be more flexible/robust
- mustache_kimono 4y ago> I think there is a major bandwagoning with zfs And I think there is a kooky caucus for half ass solutions. The Linux community has chosen to beclown itself by pretending ZFS isn't absolutely amazing (and free!), and the result is the strangest collection of system level kludges anywhere. You forgot Stratis and unraid!
- matheusmoreira 4y ago> there is a major bandwagoning with zfs Well, there is a huge number of reasons for that. Just recently ZFS got RAID-Z expansion capabilities. Before that, I would have had to make plans for a storage server and buy all the storage devices upfront. Now I can expand the storage pool as needed, one device at a time. The ability to do this is the only reason I even bothered with btrfs. Now that ZFS has flexible storage pool expansion, it is essentially perfect as far as I'm concerned.
- Wowfunhappy 4y agoIf you're at the point of using NTFS data recovery tools, you've already lost. With an enormous amount of time, effort, and/or money, you may be able to recover something, although without checksums you'll never be able to validate whatever-it-is you pull out. I wouldn't call it a "manageable" failure. The goal is to never reach this point, via a combination of redundancy and backups. ZFS helps enormously with the first part.
- crest 4y agosigh It still checksums everything it touches and can read damaged file systems in most cases. More importantly it can tell you which parts have been corrupted and which haven't. Because all live data on disk is checksummed and the checksum is part of the reference (except for the ring buffers of superblocks which carry their own checksums) you can find detect data corruption and recover to a sane state. If you detect and guestimate a fix for corruption on traditional file system by your own logic you would have to consider all file system data structures as well as all file content suspect with no recover path short of wiping the file system and restoring the data to a fresh file system. Oh wait your backup was made from an untrustworthy source... It's not as black and white as you or the blog post makes it out to be. I've had to recover damaged ZFS pools while traveling (there are neither affordable nor travel compatible no laptops with ECC RAM support). ZFS scrub told me which 4 files have been corrupted and overwriting them with good copies from an other machine solved the problem. Without ZFS (or a similar file system) I wouldn't have noticed the data corruption as quickly and wouldn't have known what to restore. Also ZFS is no one trick pony and has more to offer than "just" end to end checksumming e.g. pooled storage for multiple file systems and sparse block devices, fast consistent snapshots (good enough to backup a running RDBMs), incremental backups and replication (no more dreaded full backups), ease of administration, easy to grow capacity (as long as you do it in large enough increments), transparent compression and if you really need it block level deduplication (you probably don't want online block level dedup).
- hamandcheese 4y agoYou should have an offsite backup. If you don’t, you are asking for data loss, regardless of what filesystem you use. I personally am glad that ZFS doesn’t have “easy to partially recover a corrupted FS” on their list of priorities. Designing for that might take resources away from making the system as a whole more robust, and at best it’s a half-measure that will never be a substitute for an offsite backup solution.
- Avamander 4y ago> If you don’t, you are asking for data loss, regardless of what filesystem you use. Backups have a delay. A filesystem that fails catastrophically is significantly worse than the one that doesn't.
- hamandcheese 4y agoSuppose your FS has a 50% chance of partially recoverable failure in a given time range. Is that always better than the catastrophic alternative, even if the odds of catastrophic failure are 0.01%?
- Avamander 4y agoThat risk assessment depends on your use-case. Those percentages also aren't very correct. Assuming that new files are being constantly created and they're not of vital long-term integrity, the chance of preventing silent bit rot (ZFS is supposed to protect against) is worth fuck all if it's much more likely that a bug takes your pool irrecoverably with hours of data.
- rhinoceraptor 4y agoIt's just not true that if ZFS metadata is corrupted, then the entire pool is gone. ZFS is a transactional file system, so the pool will be put in a faulted state, and it gives you the option to roll back to before the faulty transaction. I really do not see how anyone could possibly think that is worse than your data being silently corrupted on disk, which then propagates to your backups, corrupting them too. People seem to think ZFS is worse because it tells you when your data is corrupted, rather than not telling you? It doesn't make any sense.
- louwrentius 4y agoIf you checksum garbled data from memory, that checksum is useless. Garbage in, garbage out.
- Wowfunhappy 4y agoYes, the checksum is useless, because the checksummed file is corrupt. By contrast, without checksumming... the file would still be corrupt! In the best case, when good data was written to disk, checksumming preserves the good data. In the worst case, when bad data was written to disk, checksumming does no harm. Checksumming is good.
- rhinoceraptor 4y agoNo one is saying that you shouldn't use ECC, obviously you should if at all possible. But even if you don't have it, that is no reason not to use ZFS.
- Avamander 4y ago> It's just not true that if ZFS metadata is corrupted, then the entire pool is gone. ZFS is a transactional file system, so the pool will be put in a faulted state, and it gives you the option to roll back to before the faulty transaction. It is true because ZFS on Linux crashes very quickly when metadata is corrupt. It can't roll back transactions because it'll crash almost instantly even looking at your pool, not to mention when scrubbing it.
- anonuser123456 4y agoYour file system is not a backup technique. All file systems will fail. The only time I’ve had ZFS fail, was when both drives in a mirror pool died within 8 minutes of another. No filesystem would save you there. Since ZFS has a replication protocol (ZFS send) recovery was basically a few ZFS revc calls into a new pool.
- yarg 4y agoAssuming that you bought two of the same device? https://news.ycombinator.com/item?id=32048148 https://news.ycombinator.com/item?id=32048148
- louwrentius 4y agoAuthor here: I think ZFS is totally fine, but it's about understanding risk. And running ZFS on your laptop without ECC, that's ok. Running ZFS on a home server as a NAS but without ECC memory, you are addressing one risk, but you are forgetting another (IMHO bigger) risk: faulty memory causing bitrot. It's as if you lock your back-door and leave the front-door unlocked.
- GekkePrutser 4y agoUnderstandable, but by telling people to just forget ZFS and use EXT4 instead, you're getting them to leave both doors open. Also, bitrot is not like burglars. Burglars try one door and if they don't succeed they will look for further vulnerabilities. Memory and disk corruption is random, it's not trying to attack you. In many cases ECC is just not possible without buying expensive new hardware (with possibly higher power consumption) due to intel being so precious with ECC as a 'server feature only'. Data protection is not absolute, protecting against one class of corruption is better than none. E.g. one of my NASs is a low-power NUC specifically chosen for its power consumption. Of course both is even better but not everyone has the financial resources for perfection.
- louwrentius 4y agoI would never tell people to forget ZFS, it’s understanding that ZFS alone just doesn’t reduce all bitrot risk. Groeten! In the end people worry too much, all consumer NAS gear doesn’t have any bitrot protection… (to put things in perspective).
- Avlin67 4y agoActually everything should be ECCed but it is a consumer/professional segmentation maintained for years
- thanatos519 4y agoI didn't really appreciate this until I got an ECC workstation. It's noticeably more stable than my previous machines, even under abusive levels of load. Noticeably more stable, as in, "never ever crashes" as opposed to "almost never crashes" without ECC. Thanks Linux! :D
- PragmaticPulp 4y agoDid you actually check the ECC error counters? Memory errors in data centers tend to be concentrated in a small number of bad sticks of RAM rather than evenly distributed across all memory. If you have a machine crashing regularly due to memory errors, it’s likely a bad stick of RAM, not random errors due to lack of ECC.
- freemint 4y agoThe ECC error counters are not really honest anymore as repaired faults are often not reported from what i heard.
- dannyw 4y agoOn hard drives and SSDs, yes, but has this been happening to RAM too?
- undersuit 4y agoI guess you could say it has begun with DDR5 on-die ECC. I don't think you can query those counters.
- ChuckMcM 4y agoThis depends on the BIOS and the OS. Correctable (and corrected) errors are typically logged into the baseboard controller (the BMC) and the OS "should" periodically dump the logs from the controller to maintain a record of those errors longer term. That said, eventually they just "fall off" the list of errors the BMC is holding because it stores them round robin. Uncorrectable errors will cause a machine check, unless the BIOS disables it. Which some do.
- g0xA52A2A 4y agoI hate this headline and wince every time I see it, even the article quotes > There's nothing special about ZFS that requires/encourages the use of ECC RAM more so than any other filesystem. If you use UFS, EXT, NTFS, btrfs, etc without ECC RAM, you are just as much at risk as if you used ZFS without ECC RAM. I would simply say: if you love your data, use ECC RAM. Additionally, use a filesystem that checksums your data, such as ZFS. And goes on to say > I have nothing to substantiate this, but my thinking is that since ZFS is a way more advanced and complex file system, it may be more susceptible to the adverse effects of bad memory, compared to legacy file systems.
- PragmaticPulp 4y agoI’m actually fascinated by how much this post became accepted canon over the years. The truth is that everyone should want to use ECC wherever data integrity is a priority, regardless of whether or not they’re using ZFS. The conjecture about ZFS being uniquely vulnerable has been debunked over and over again. Maybe there was something different back in 2014 when this was written, but even if that was the case it’s not a good idea to use such old blog posts full of admitted conjecture to anchor your technology understandings.
- wmf 4y agoPeople love clever contrarian takes. A thing that sounds good is actually bad!
- colechristensen 4y agoEh, the chance that data I care about will get corrupted in a way that matters is very small. ECC also does not eliminate errors, just reduces them by some constant proportion. Something quite rare becoming say, 10 times less likely doesn't really effect my decisions much. If you're using ECC you can make the exact same argument that you need another bit of parity with the same justifications of making something rare, rarer. Of the things than can go wrong with my data, small corruptions from memory bit flips are far down on the risks to worry about. If you're Google (you're not Google) then the scale of errors becomes a choice of optimization.
- getcrunk 4y agoDdr5 has some form of built in ecc. Finally bringing it by default to the masses Edit: sorry, this is misleading. See below or ignore. Ddr5 "ecc" that I am referring to is not the same as what one would think when saying ecc normally
- wmf 4y agoIt's not clear whether the resulting error rate is any better than DDR4, and I assume you can't monitor on-die ECC either.
- Teever 4y agoThis isn't accurate. It has onchip ECC to mitigate the fact that we're pushing the limits of hardware to eek out more performance from DDR5, but there isn't ECC between the ram and cpu, by default anyways. So this is one more thing that will confuse the consumer.
- toast0 4y agoThe masses had parity ram back in the day, then there was fake parity ram, then there was no parity. Internal correction codes have been normal for disk storage for a long time, but it's not really enough unless there's reporting and monitoring. What does consumer DDR5 do when there's a correctable error, what does it do when there's an uncorrectable error?
- nix23 4y agoPlease always use ECC because your Filesystem-cache (for example xfs) is the same as your ARC (on ZFS)...it's really the same. But you can activate Arc check-summing at least. That's a better article: https://jrs-s.net/2015/02/03/will-zfs-and-non-ecc-ram-kill-your-data/ https://jrs-s.net/2015/02/03/will-zfs-and-non-ecc-ram-kill-y...
- dannyw 4y agoI have 10 SATA HDDs (14-18TB each). I would like to build a TrueNAS box with ECC memory on a budget of less than $700 (used hardware is perfectly fine). Any recommendations for what to look on ebay?
- rhinoceraptor 4y agoI'd recommend keeping an eye on Craigslist, or unixsurplus.com is also great, but they usually stock newer gear out of that price range. My NAS is a 2U Dell R510 (Dual E5649, 32gb ECC memory), I think it cost me about $250 on Craigslist, and I've had zero issues. It will likely be difficult to find a machine that can take 10 drives, mine can take 8. If a machine has 3.5" SAS bays, it likely can take SATA drives as well, but be sure to do your own research. Also, in a lot of these older machines the PCIe card that connects the SAS backplane is hardware RAID only, which means you will have to find a new card, and possibly flash it with a new firmware to enable JBOD mode, so that your disks are passed directly to the host OS. I had to do this for my NAS but it wasn't difficult. One other tip I highly recommend, you can buy PCIe cards that fit one or two 2.5" SATA SSDs in. I run my TrueNAS off of a mirror of two small SATA SSDs this way, which frees up space for more hard drives in the front.
- undersuit 4y agoYou could use an older AMD Zen CPU with a consumer motherboard that has ECC support. Gigabyte and Asrock are manufacturers I know that list ECC support on their product pages. The ECC memory is going to eat up most of the budget I would say. Make sure you don't buy Registered ECC memory under this plan.
- alexklarjr 4y agoLTT channel recently lost 1 PB of videos on ZFS on super expensive server with ECC RAM.
- Avamander 4y agoThat was mostly misuse, but ZFS does have its fair share of bugs. My favourite aspect about ZFS is the 13yo Python script for reverting transactions by destroying uberblocks being the to-go even now.
- _abox 4y agoI have to disagree with this guy. Sure, the corruption can occur, but if the memory is not faulty, ZFS will detect data corruption in-situ during a scrub. It's just that ECC errors can introduce their own corruption that is separate, during write operations only. I believe if you have no choice but to use non-ECC (e.g. existing hardware that is limited by Intel's stupid design choice of "ECC is only for servers"), ZFS is still much better than using ext4. It still protects against a different class of errors which is HDD degradation. When used in RAID-Z it can even recover for them. For perfect protection ECC is necessary. But this is not always possible financially. I think it's a bit of a wild statement to say that if you can't afford a server with ECC you should forget about ZFS entirely.
- louwrentius 4y agoThe article tells you that if you don’t need ECC memory, bitrot as a risk isn’t that important to you. It is clear that you don’t want to pay for that kind of safety so. So if bitrot isn’t that important, you can go with any other file system, or ZFS but don’t pretend you’re covered. I would rather run XFS or EXT4 with ECC memory than ZFS without ECC because silent bitrot is extremely rare because drives and protocols are full of error correction stuff. Memory is the real weak spot from a bitrot perspective. ECC first. ZFS second.