9 ms·
ZFS Gotchas
- brunoqc 12y agoIs there a similar article but about BTRFS?
- theandrewbailey 12y agoArs Technica sorta went over BTRFS exclusive features. http://arstechnica.com/information-technology/2014/01/bitrot-and-atomic-cows-inside-next-gen-filesystems/3/ http://arstechnica.com/information-technology/2014/01/bitrot...
- pronoiac 12y agoI'll say that reading about the ext3 to btrfs conversion made me laugh - it's a mad scientist thing that builds all the btrfs pointers in the ext3 free space, pointing to the data blocks as stored by ext3, and it keeps the filesystem readable as ext3 until it's completed and mounted as btrfs.
- seunosewa 12y agoWhat's funny about it?
- fiatmoney 12y agoI'd add one more: it is impossible to remove a vdev from a pool. That means in a small server / home scenario, you have less flexibility to throw in an extra mirror vdev of old disks you had lying around, or gradually changing your storage architecture if HDD sizes increase faster than your data consumption. However, none of these diminish what I think is the main use case of ZFS: very robustly protecting against corruption below the total-disk-failure level.
- voltagex_ 12y agoDoes this mean I couldn't start with 4x3TB drives and move to 4TB drives as they become cost-effective? This might kill my dreams of a ZFS NAS...
- herf 12y agoYou can remove each disk and replace with a bigger one (wait for rebuild each time), and if you set autoexpand, the pool will resize afterwards.
- voltagex_ 12y agoDoing some more reading, this sounds like it "degrades" the array each time and could be risky. Other replies to my comment seem to suggest it can't be done.
- takeda 12y agoI believe there are no problems with your scenario, there's zpool replace command which does this: zpool replace [-f] pool device [new_device] Replaces old_device with new_device. This is equivalent to attaching new_device, waiting for it to resilver, and then detaching old_device. The size of new_device must be greater than or equal to the minimum size of all the devices in a mirror or raidz configuration. new_device is required if the pool is not redundant. If new_device is not specified, it defaults to old_device. This form of replacement is useful after an existing disk has failed and has been physically replaced. In this case, the new disk may have the same /dev path as the old device, even though it is actually a different disk. ZFS recognizes this. -f Forces use of new_device, even if its appears to be in use. Not all devices can be overridden in this manner.
- mosselman 12y agoThanks for pointing this out. I still don't know how to see all of this in a home NAS context however. It would be really great if someone could explain. Lets say I have 6 SATA ports. I have 4 drives that I collected from various computers that I now want to unify in a home built NAS: A. 1TB B. 2TB C. 4TB D. 1TB E. -empty- F. -empty- Now all my drives are full and I want to either add a disk or replace a disk; How do I: 1. replace disk A. with a 4TB disk 2. add a 4TB disk on slot E.
- dang 12y agoChanged to document title to avoid the linkbait in "Things Nobody Told You About ZFS".
- rosser 12y agoWhile I don't disagree that the original title is perhaps mildly link-baity, I don't think "Read me first" is very apt, either; this article assumes a nontrivial degree of familiarity with ZFS and its concepts.
- dang 12y agoFair enough; if you or anyone would like to suggest a neutral, accurate title, we'd be happy to change it.
- rosser 12y agoSomething along the lines of "ZFS gotchas" or "ZFS caveats", perhaps? Unrelated: I really don't think dang's post here warranted downvoting. I expect that it's being done by folks who really want HN's moderators to know they're unhappy about titles being changed as capriciously as they sometimes are. This post is an actual moderator soliciting actual input on how to make at least one instance of that problem better, however, and I think that should be encouraged, not dinged.
- dang 12y agoOk, we changed it to "ZFS Gotchas" because the body of the article more or less uses that word. It's ok for dang to get dinged. But "capricious"? No. I realize it sometimes seems that way to people paying sporadic attention, but every change we make is in keeping with the HN guidelines, and those are hardly secret or unclear. We make mistakes, of course, and are always open to improvement. The helpful way to criticize an HN title is to suggest a better one.
- lmm 12y ago
- colechristensen 12y agoOverall, this is quite good. Some of it is oversold as "things nobody told you about" and some of it could really benefit from real data. A few things are a little too close to seemingly unsupported folk-wisdom rather than sound advice.
- voltagex_ 12y agoAll this article tells me is that I'm not smart enough to run a ZFS NAS. But what are my alternatives?
- dap 12y agoI wouldn't take that lesson from it. Many of these considerations don't have anything to do ZFS per se, but come up in designing any non-trivial storage system. These include most of the comments about IOPS capacity, the comments about characterizing your workload to understand if it would benefit from separate intent logs devices and SSD read caches, the notes about quality hardware components (like ECC RAM), and most of the notes about pool design, which come largely from the physics of disks and how most RAID-like storage systems use them. Several of the points are just notes about good things that you could ignore if you want to, but are also easy to understand, like "compression is good." "Snapshots are not backups" is true, but that's missing the point that constant-time snapshots are incredibly useful even if they don't also solve the backup problem. Many of the caveats are highly configuration specific: hot spares are good choices in many configurations; the 128GB DRAM limit is completely bogus in my experience; and the "zfs destroy" problem has been largely fixed for a long time now.
- voltagex_ 12y agoThere was a cheesy "CSI" video with some Sun engineers building a zfs pool from a crapton of USB sticks. I think I might do that to have a play with it before I spend ~$1500AUD building a NAS.
- phil21 12y agoI actually did this with 64x32GB eBay special (read: godawful slow) MicroSD's. More to make a joke about my "enterprise flash storage system" as we were in the middle of a bunch of storage vendor trials, and sick of dealing with buzzword sales guys. It was... slow. But the blinky lights were pretty awesome.
- dspillett 12y agoWe did a similar experiment to http://macguild.org/raid.html http://macguild.org/raid.html (RAID 5 over USB floppy drives) with a bunch of small slow USB sticks we'd collected between us (~64..256Mb IIRC, using just 60M on each to treat them as same sized devices) and various Linux RAID arrangements. It was a good way to play with the tools in an environment where we can break things artificially, as well as an interesting little play thing. Another good place to practise is in VMs of course. Both options are far enough away from a real physical NAS that any performance tests you do are largely irrelevant and some of the failure modes aren't as easy to properly simulate as you might think, but still a useful learning or confidence building exercise if you are new to working with the relevant tools.
- rdtsc 12y agoArticle mentions clustered file systems. Is there any reason why one might choose that for a small office / home office setup instead of ZFS? Would it make setup/expansion easier. Taking it in another direction. I've been following this LeoFS project which basically replicates Amazon's S3 storage (so any S3 clients can work with it). http://leo-project.net/leofs/ http://leo-project.net/leofs/ Its claim is one can add nodes to the cluster to expand its storage capability. Given enough physical space it might be cheaper to find 3-4 machines (towers) with 4 drive bays than building one new server with 12+ bays. Then use linux's fuse client to access it.
- buster 12y agoWell, performance. If you need IOPS, every network based storage will be far slower. And a whole lot more gotchas in terms of setup.
- dmpk2k 12y agoAnd a whole lot more gotchas when things go wrong. Networks cause all sorts of fun byzantine failures; local is a simpler problem to reason about.
- lmm 12y agoI've never seen a clustered filesystem that was friendly or simple enough for the small office / home use case. With ZFS I pile a bunch of disks in a server, run samba on it, and there you go. Still, best of luck if you try it.
- e12e 12y agoOnly problem with ZFS for budget/home use as I see it, is the inability to grow zpools (add a disk to get more space on an existing filesystem). Basically the the design is more "buy what you need (eg: 4x2TB), then replace (eg: 4x4TB as prices come down) -- and throw away your old disks".
- rdtsc 12y ago
- kevin1024 12y ago> Do not use raidz1 for disks 1TB or greater in size. Oops, I'm doing that on my home NAS. Does anyone know why this is bad?
- edude03 12y agoI am as well (5x 3TB in Raidz1). I'm pretty sure it's because of the likelihood of having an unreadable bit/byte/sector on one of the non failed disks gets higher as the capacity increases and thus there is a good chance that you'll lose some data. This article discusses the theory. http://www.zdnet.com/blog/storage/why-raid-5-stops-working-in-2009/16 http://www.zdnet.com/blog/storage/why-raid-5-stops-working-i...
- voltagex_ 12y agoIs there a way to check statistics of failed read/writes?
- dap 12y ago"zpool status" will show if there have been errors reading data from individual devices. If a drive experiences enough failures, at least on illumos and Solaris-based systems, it will be marked degraded or faulted and removed from service. You can view individual failures on these systems with "fmdump -e". Here's a made-up worked example: https://blogs.oracle.com/bobn/entry/zfs_and_fma_two_great https://blogs.oracle.com/bobn/entry/zfs_and_fma_two_great
- anon4 12y agocorrected link: http://www.zdnet.com/blog/storage/why-raid-5-stops-working-in-2009/162 http://www.zdnet.com/blog/storage/why-raid-5-stops-working-i...
- dap 12y agoI wish the author had provided an explanation. The main issue I'm familiar with is that throughput and IOPS capacity generally don't increase linearly with storage capacity, so the time to recover from a drive failure increases significantly with larger drives. The author may be saying that you should use raidz2 or raidz3 with 1TB drives because the time to resilver 1TB is long enough that the odds of sustaining another drive failure with raidz1 are too high, or alternatively that you should use 750GB or smaller drives with raidz1 to keep the resilver times lower in order to reduce the odds of a second failure during resilvering).
- javajosh 12y agoWhat I would like to see is a Feynman-style "Introduction to Filesystems" geared toward programmers. And when I say Feynman-style, I mean lucid, devoid of jargon, and dense with meaningful concepts. Granted, most application programmers don't deal with the file system in any meaningful way - most interaction is deferred to other processes (e.g. the datastore). But I for one would be interested in certain silly things like, oh, writing a program that can (empirically) tell the difference between a spinning disk and a solid state disk, or what file system it's running on, just from performance characteristics. Other fun things would be to determine just how fast you can write data to disk, and what parameters make this rate faster or slower.
- mhw 12y ago> Examples of things you don't want to do if you want to keep your data intact include using non-ECC RAM I'm planning to build a (Linux) development workstation soon and was intending to run ZFS on at least a proportion of the disks in the system, if not the whole thing. But building a workstation that supports ECC appears to be ridiculously hard and expensive - most of the Haswell CPUs don't support ECC. Can anyone point me at a sensible CPU/motherboard combo that's comparable to something like an i7-4770k/Gigabyte GA-Z87MX-D3H? (As an indication of the state of the component market, pcpartpicker.com doesn't support searching CPUs or motherboards by ECC support, but it does allow you to search by what colour the motherboard is!)
- rockdoe 12y agoYou need a Xeon v3 for the ECC support, basically. Also obviously getting ECC ram for stability and getting a K series CPU for overclocking is just an incompatible mindset.
- moletown 12y agoYou can use ECC on some i3 CPUs [1] as long as the motherboard supports it. Supermicro has a X10 micro-atx board [2] which would work. [1] http://ark.intel.com/search/advanced?s=t&FamilyText=4th%20Generation%20Intel%C2%AE%20Core%E2%84%A2%20i3%20Processors&MarketSegment=DT&ECCMemory=true http://ark.intel.com/search/advanced?s=t&FamilyText=4th%20Ge... [2] http://www.supermicro.com/products/motherboard/Xeon/C220/X10SLM-F.cfm http://www.supermicro.com/products/motherboard/Xeon/C220/X10...
- voltagex_ 12y agoI wonder if an i3 has enough grunt for ZFS. I definitely notice the difference between a middling i3 and my i7 2600k.
- voltagex_ 12y agoFiddling with the filters on that page, it seems some Atom processors support ECC, too - http://ark.intel.com/products/81270/Intel-Atom-Processor-C2508-2M-Cache-1_25-GHz http://ark.intel.com/products/81270/Intel-Atom-Processor-C25...
- brongondwana 12y agoI don't see any mention of "for god's sake don't let your pool get more than 90% full or the molasses gremlins will ooze out of your disk controllers and gum up the plates" in that list...
- Touche 12y agoSomething I never see discussed, what is ZFS's effect on battery life? I have not put it on laptops for this concern.
- flyinghamster 12y agoAnother gotcha: If you intend to physically move a pool of disks from one system to another, or even move them to different SATA or SAS ports on the same system (or different drive bays in a JBOD), you should EXPORT THE POOL FIRST. I came within a hair's breadth of losing a pool when I didn't do that, and import attempts resulted in "corrupted data" messages. I managed to straighten that out and recover the pool, but it was a close call. When it is not exported, the pool stores the device names of its component disks, and if the wrong disks end up on the wrong devices, you get the "corrupted data" problem even though the data really isn't corrupted.
- flyinghamster 12y agoAlso note, if you're using an SAS HBA: the LSI 1068E-R based HBAs, when flashed with IT firmware, have a persistent drive mapping setting. When it is active, it will always give a specific physical disk the same device name, even if it gets moved to a different physical port. Let's say you a drive with serial number ABC123 on port 0 as c4t0d0 and serial number XYZ987 on port 1 as c4t1d0. You could swap the two drives and ABC123 would still be c4t0d0 and XYZ987 would still be c4t1d0. If all ports on the controller were in use, and you yanked c4t7d0 and slid in a new drive, it would become c4t8d0 (unless you used lsiutil to remove the persistent mapping). Other HBAs may have similar features.
- namewithhe1d 12y agoOk, say you implemented Dedep accidentally and now your table is huge. I'm sure it is a slog to go back, but HOW DO YOU DO IT. I've tried the method here: https://groups.google.com/a/zfsonlinux.org/forum/#!topic/zfs-discuss/at3Cb08d21A https://groups.google.com/a/zfsonlinux.org/forum/#!topic/zfs... and didn't get it fixed. Any more thoughts? Please don't recommend I make a new pool and send all data over, I don't have the space at 70% capacity.
- IgorPartola 12y agoNon-ZFS Gotchas: Other file systems and RAID setups do not checksum your data. If you have a mirror of two disks (RAID1) and during a read two blocks differ from one another, most (all?) RAID controllers (hardware and software) will simply choose the lower numbered disk as canonical and silently "repair" the bad block. This leads to loads of silent data corruption on what we might consider a reliable storage solution. By contrast, ZFS will store both the block and its checksum and will use the block with the correct checksum as canonical. In other words, if you care about your data at all, use ZFS. I am frankly surprised it's not the standard file system for most situations as it is the only production filesystem that can actually be trusted with data. P.S.: I have been told that at least on Linux if you have more than two drives, the Linux software RAID controller will try to choose the version of the block that is agreed upon by most drives, if that's possible. This is no guarantee, but it's better than randomly choosing one version. P.P.S: BTRFS and friends seem to not yet be as production ready as ZFS. Conversely, ZFS works beautifully on Linux thanks to the ZFS on Linux project.
- voltagex_ 12y agoThe last release of zfs on Linux was Aug 23 2013, should I be worried?
- voltagex_ 12y agoSorry about the number of posts, but https://groups.google.com/a/zfsonlinux.org/forum/#!topic/zfs-devel/RUBQHIaUugE https://groups.google.com/a/zfsonlinux.org/forum/#!topic/zfs... makes me very nervous to use ZFS on Linux.
- IgorPartola 12y agoI can only speak from my own limited experience, but for me ZFS on Ubuntu works as advertised. I was trying to put together a long term storage solution without becoming a storage expert. ZFS fit the bill. If you are nervous, go with a BSD: they are great choices.
- tjoff 12y ago