26 ms·
ZFS Is the Best Filesystem For Now
- Mic92 9y agoThe article does not mention bcachefs as a future alternative: http://bcachefs.org/ http://bcachefs.org/
- sfoskett 9y agoI had never heard of bcachefs. Thanks for the pointer!
- jeremyw 9y agoHN: Please consider subscribing to make this alternative real. https://www.patreon.com/bcachefs https://www.patreon.com/bcachefs
- specto 9y agoWe can all hope...
- reacharavindh 9y agoI’m certainly an excited audience for modern alternative file system options. Quick look at the homepage of bcachefs reads - “Not quite finished - it's safe to enable, but there's some work left related to copy GC before we can enable free space accounting based on compressed size: right now, enabling compression won't actually let you store any more data in your filesystem than if the data was uncompressed.” What is the point of having compression enabled if you can’t store more data than you could if it was uncompressed? Shouldn’t they just say “compression mechanism works but not useful yet. as of now it is just extra overhead”..
- wtallis 9y agoCompression can give you a net performance improvement on slow disks if the CPU time required to compress is less than the time saved by writing less data to the disk.
- std_throwaway 9y agoI see they use another approach to file systems. They sound very confident. A bit too confident to my taste. Are they experienced enough in building file systems that they won't run into huge problems arising when it actually goes into the dirty details?
- wtallis 9y agoThe developer of bcachefs already has the experience of creating bcache, a general-purpose block-level caching system that is optimized for the characteristics of flash-based SSDs as the cache devices. The original bcache essentially serves as the proof of concept and practice implementation for all the lower-level portions of bcachefs (not that it was planned that way; they just realized while working on bcache they had implemented most of a filesystem).
- alyandon 9y ago"Once you build a ZFS volume, it’s pretty much fixed for life." The ease of growing/shrinking existing volumes and adding/removing storage is why I made the decision to go with btrfs when I rebuilt my home file server.
- ThatPlayer 9y agoI'm also using Btrfs for my home file server, but Btrfs's current RAID5/6 status leaves a lot to be desired. I'm running Btrfs on top of mdadm right now because of that.
- sheepdestroyer 9y agoShould be "mostly" resolved from kernel 4.12 : https://www.phoronix.com/scan.php?page=news_item&px=Linux-4.12-Btrfs-RAID-Mostly-OK https://www.phoronix.com/scan.php?page=news_item&px=Linux-4....
- pantalaimon 9y agoIt used to be "usable for most purposes" at some point already, before those scary data corruption bugs were discovered. https://btrfs.wiki.kernel.org/index.php?title=RAID56&diff=30445&oldid=29769 https://btrfs.wiki.kernel.org/index.php?title=RAID56&diff=30...
- alyandon 9y agoMy last file server used mdadm raid 5 across 4 drives and I just couldn't take the write performance loss anymore so I switched to btrfs raid1(ish) block duplication.
- phil21 9y agoThis is only difficult with ZFS if you care about performance. If you are a typical home file server user, you can add vdevs in a rather ad-hoc fashion as they fill and it works pretty well. There is a huge performance penalty of course, as all the majority of new data will reside on the latest vdevs - but it generally works. I do agree this is one of the largest drawbacks of ZFS, but very few filesystems get it right.
- floatboth 9y ago> ZFS never really adapted to today’s world of widely-available flash storage: Although flash can be used to support the ZIL and L2ARC caches, these are of dubious value in a system with sufficient RAM, and ZFS has no true hybrid storage capability. How is L2ARC not "true hybrid"? > And no one is talking about NVMe even though it’s everywhere in performance PC’s. Why should a filesystem care about NVMe? It's a different layer. ZFS generally doesn't care if it's IDE, SATA, NVMe or a microSD card. > can be a pain to use (except in FreeBSD, Solaris, and purpose-built appliances) I think it's just a package install away on many Linux distros? Also installable on macOS — I had a ZFS USB disk I shared between Mac and FreeBSD. Also it's interesting that these two sentences appear in the same article: > best level of data protection in a small office/home office (SOHO) environment. > It’s laughable that the ZFS documentation obsesses over a few GB of SLC flash when multi-TB 3D NAND drives are on the market Who has enough money to get a mutli-TB SSD for SOHO?!
- edude03 9y ago> How is L2ARC not "true hybrid"? L2ARC in my understanding is only for reads, whereas ZIL is the write ahead log. Ideally, ZFS would "combine" the two into a MRU "write through cache" such that data is written to the SSD first, then asynchronously written to the disk after (ZIL does this already) but then, when the data is read back, it's read back from the SSD.
- philsnow 9y agoThis makes sense; if you were building a service that served bytes off of disk, with an in-memory LRU cache, you wouldn't have two separate pools of memory, with one for writes and one for reads. Did the ZIL and L2ARC concepts come up before SSD was widely available? Especially the ZIL seems very much optimized for crazy enterprise 15k rpm spinning rust. Memory and SSD access characteristics are so different from spinning disks; I don't know why ZFS separates ZIL and L2ARC.
- koffiezet 9y agoBecause the ZIL is a journal log, not a cache. It is intended to increase data security without sacrificing too much performance. Many people also confuse ZIL and SLOG devices though... By default, the ZIL is written on the same disks as where the data will be stored, but an external device (aka SLOG) can be added. From that point on, your write IOPS will be limited by this SLOG device, so normally you add a more expensive fast disk as SLOG device to increase your write IOPS.
- Cieplak 9y agoOn my current laptop, I'm seeing a 20% reduction in disk usage relative to the filesystem size because of ZFS's built-in compression.
- floatboth 9y ago20%? That's weak :P I have 3.32x refcompressratio on my /home partition in my dev VM (using gzip-7 here).
- loeg 9y agoYou might look into using xz(lzma) or zstd in place of gzip. Gzip offers pretty poor compression per CPU time performance compared to these newer options. https://clearlinux.org/blogs/linux-os-data-compression-options-comparing-behavior https://clearlinux.org/blogs/linux-os-data-compression-optio...
- mastax 9y agoZFS doesn't support either yet: http://open-zfs.org/wiki/Performance_tuning http://open-zfs.org/wiki/Performance_tuning Though it looks like zstd is coming, which is exciting: https://reviews.freebsd.org/D11124 https://reviews.freebsd.org/D11124
- simcop2387 9y agozfs doesn't support either of those yet unfortunately. I'd love for zstd to be available given it's benefits in speed and compression ratio.
- loeg 9y agohttps://reviews.freebsd.org/D11124 https://reviews.freebsd.org/D11124 :) Or use lz4 while you wait.
- snakeanus 9y agoI am really excited for bcachefs. It is also the only fs that has support for chacha20-poly1305 encryption.
- foepys 9y agoI wonder if a non-hardware accelerated encryption algorithm is good enough for a FS that also has checksums. The CPU is already busy with checksimming, so doesn't this considerably slow down writes?
- wmf 9y agoSomewhat counterintuitively, Chacha-Poly is as fast as AES-NI. You could probably go even faster using QuickAssist but most people don't have that.
- conductor 9y agoBoth Chacha20 and Poly1305 are optimized (by design) for running on general purpose CPUs. AES-GCM using AES-NI instructions is still faster, but not that much [0]. [0] https://community.qualys.com/thread/16005 https://community.qualys.com/thread/16005
- fulafel 9y agoDepends on your CPU. Desktop/server CPUs are plenty fast enough, but a home NAS might slow down.
- carlob 9y ago> Many remain skeptical of deduplication, which hogs expensive RAM in the best-case scenario. And I do mean expensive: Pretty much every ZFS FAQ flatly declares that ECC RAM is a must-have and 8 GB is the bare minimum. In my own experience with FreeNAS, 32 GB is a nice amount for an active small ZFS server, and this costs $200-$300 even at today’s prices. I use nas4free with much less ram…
- eriknstr 9y agoThe massive amounts of RAM recommendation is if you need to do deduplication. Are you doing that? If not then you don't need a lot of RAM. Between the low cost of storage, and alternative solutions for deduplicating data I personally don't use the built-in deduplication functionality of ZFS for my zpools. Might come down to what sorts of data you are storing, though.
- sfoskett 9y agoYup. This. Now that I have 32 GB of RAM in my FreeNAS box I decided I really didn't need dedupe after all. I just don't have that much duplicated data, and I've got 60 TB of HDDs in the box. So I use the RAM for VMs instead!
- agumonkey 9y agoYeah dedup is mostly useful for massively multiusers setups, say mail, file sharing, etc etc. For small setups you can do a `fdupes` pass once a day and fix it yourself I suppose.
- 293984j29384 9y agoI remember that time I turned on dedup on our 45drive box with 32GB of ram and 140TB~ of data..
- mioelnir 9y ago...and nothing happened because it does not retroactively dedup existing data? Over time it would become horrendously slow, I agree, since you have too little RAM by a factor of at least 8.
- moonbug22 9y agoI'll stick with GPFS, thanks.
- fnord123 9y agoPorque no los dos? http://files.gpfsug.org/presentations/2016/anl-june/LANL_GPFS_ZFS.pdf http://files.gpfsug.org/presentations/2016/anl-june/LANL_GPF...
- Terribledactyl 9y agowow thank you, I had toyed with this idea a few months ago but didn't have time to investigate it.
- eriknstr 9y agoOP is not suggesting ZFS above all else. > And every enterprise system has already moved way past what ZFS can do, including enterprise-class offerings based on ZFS from Sun, Nexenta, and iXsystems. However, OP is naturally suggesting ZFS over NTFS, HFS+, ext3/4 and even ReFS and APFS. > Still, ZFS is way better than legacy storage SOHO filesystems. The lack of integrity checking, redundancy, and error recovery makes NTFS (Windows), HFS+ (macOS), and ext3/4 (Linux) wholly inappropriate for use as a long-term storage platform. And even ReFS and APFS, lacking data integrity checking, aren’t appropriate where data loss cannot be tolerated.
- X86BSD 9y agoInteresting. I'd like to know what the "enterprise" has moved on to from ZFS.
- sfoskett 9y agoEnterprise Storage == Real Storage Arrays Basically every storage array is years beyond ZFS in terms of features and capabilities. Even the ones based on ZFS! And I don't think anyone would argue that. After all, Dell EMC, HPE, Pure Storage, HDS, etc spend many millions a year developing advanced storage technologies.
- gulikoza 9y agoThe thing I'm struggling with is 4K sector support. It's horribly inefficient with ZFS. RAIDZ2 wastes a ton of space when pool is made with ashift=12. And everybody knows 512e on AF disks is horribly slow...so ZFS is either very slow or wastes 10% of total space...Or both (ZVOL :D) According to some bug reports, nobody has touched this since 2011...
- iooi 9y agoCan you elaborate on how ZFS wastes 10% of total space? I recently set up a ZFS volume using 12x4TB drives using RAID-Z2, so I expected 40TB of usable space, or ~36.3TiB. However, I only see 32TiB of usable space on the volume. I always wondered why that was so, never figured it out..
- gulikoza 9y agoThere's ton of sites when you google ashift=12, http://louwrentius.com/zfs-performance-and-capacity-impact-of-ashift9-on-4k-sector-drives.html http://louwrentius.com/zfs-performance-and-capacity-impact-o... or https://github.com/zfsonlinux/zfs/issues/548 https://github.com/zfsonlinux/zfs/issues/548 for instance. Basically, ashift=12 increases ZFS block size to 4K. Metadata use full blocks that would be 512b on ashift=9 but are now 4K (due to ashift=12). It wastes at least 3.5Kb more than normal 512 byte blocks for each block that is not filled entirely.
- zzzcpan 9y agoCeph, gluster, object storages, all would do a better job serving SOHO. ZFS is a 90s way of thinking about storage, "a box" way. I don't think it deserves any of that HN hype.
- KaiserPro 9y agoahem gluster? ceph? really? ceph is horrifically slow, and only has a rudimentary posix interface. gluster is just well, terrible. first things first a SOHO office cannot support a clustered filesystem, unless one of the people happens to be a storage specialist. yea I hear lots of noise about how self healing they both are. That is mostly fancy talk for "I don't have backups" Supposedly they both do HA. But yeah, its not something I'd want to support. If you look at gitlab's setup, and their proposed setup (https://about.gitlab.com/2016/12/11/proposed-server-purchase-for-gitlab-com/ https://about.gitlab.com/2016/12/11/proposed-server-purchase...) 32 file servers to serve ~480TBs of disk. Seriously? 4 file servers, GPFS and 4 MD3060e. Thats about a petabyte of usable storage with a streaming throughput of about 40 gigabits a second. "modern" clustered filesystems are mainly just toys. if you want speed, use lustre, if you want sexy software defines awesomeness with next level raid, use GPFS.
- floatboth 9y agolol. If you need the cluster things, I wouldn't consider that SOHO anymore.
- peapicker 9y agoZFS, at least on Solaris, has issue with many multiple readers of the same file, blocking after ~31 simultaneous readers (even when there are NO writers). Ran into this with a third party library which reads a large TTF to produce business PDF documents. The hundreds of reporting processes all slowed to a crawl when accessing the 20Mb Chinese TTF for reporting because ZFS was blocking. I can't change the code since it is third party. The only way I saw to easily fix it was on system startup to copy the fonts under a new subdir in /tmp (so in tmpfs, ie RAM, no ZFS at all there ) and then softlink the dir the product was expecting to the new dir off of /tmp, eliminating the ZFS high-volume multiple-reader bottleneck. Never had this problem with the latest EXT filesystems on my volume groups on my Linux VMs with the same 3rd party library and same volume of throughput.
- nisa 9y agoIf you recreate the bug on Linux and have something like a simple reproduder I guess that the ZoL devs are more than happy to fix or at least understand it: https://github.com/zfsonlinux/zfs https://github.com/zfsonlinux/zfs From reading the pull requests and issues on that repo I've got the impression that the next release 0.7.0 will be quite step forward and there seems to be quite sophisticasted work to tackle performance issues.
- deleted 9y ago[deleted]
- JdeBP 9y agoI suggest getting on to the OpenZFS people and seeing whether this is still a problem in illumos; and fixing it if so. * http://www.open-zfs.org/wiki/Main_Page http://www.open-zfs.org/wiki/Main_Page * https://github.com/openzfs/openzfs https://github.com/openzfs/openzfs * https://wiki.illumos.org/display/illumos/ZFS https://wiki.illumos.org/display/illumos/ZFS
- conductor 9y agoDragonFlyBSD's HAMMER [0] is another viable alternative. Unfortunately the next generation HAMMER2 [1] filesystem's development is moving forward very slowly [2]. Nevertheless, kudos to Matt for his great work. [0] https://www.dragonflybsd.org/hammer/ https://www.dragonflybsd.org/hammer/ [1] https://gitweb.dragonflybsd.org/dragonfly.git/blob_plain/HEAD:/sys/vfs/hammer2/DESIGN https://gitweb.dragonflybsd.org/dragonfly.git/blob_plain/HEA... [2] https://gitweb.dragonflybsd.org/dragonfly.git/history/HEAD:/sys/vfs/hammer2 https://gitweb.dragonflybsd.org/dragonfly.git/history/HEAD:/...
- Veratyr 9y agoI had a look at HAMMER as it seemed it might meet my requirements [0] but I couldn't figure out whether it supports replication or erasure coding. Don't suppose anyone here knows? [0]: https://news.ycombinator.com/item?id=14756787 https://news.ycombinator.com/item?id=14756787
- deleted 9y ago[deleted]
- conductor 9y agoHAMMER does support replication but it doesn't have erasure coding. If you are using a recent Linux Kernel I can suggest you to use dm-integrity [0] (optionally with dm-crypt) with your favorite filesystem. It's not erasure coding but it can help detecting silent data corruption on the disk. [0] https://old.lwn.net/Articles/721738/ https://old.lwn.net/Articles/721738/
- Veratyr 9y agoHmm, it does seem like it should be possible to put mdadm on top of dm-integrity devices, I might try that out.
- jerry40 9y agoDoes anybody use ZFS as replacement for a database backup/restore on a test environment? I'm not sure but it seems that it's possible to use ZFS snapshots in order to quickly restore previous database state. Note: it's just a question, I'm not advising to try that.
- orf 9y agoFile system snapshots of databases are not necessarily consistent, and can not always be restored like that.
- floatboth 9y agoAtomic snapshots like ZFS's are always consistent for Postgres. I guess other databases with a similar write-ahead log can be snapshotted as well?
- anarazel 9y ago> Atomic snapshots like ZFS's are always consistent for Postgres. As long as you make sure to only use one filesystem, i.e. you don't place pg_xlog or some tablespaces on a different filesystem. You can get very weird corruption in such cases :)
- olavgg 9y agoWith ZFS you can also do atomic snapshots of multiple filesystems. https://serverfault.com/questions/608223/is-zfs-snapshot-r-of-several-pools-atomic https://serverfault.com/questions/608223/is-zfs-snapshot-r-o...
- jerry40 9y agoInteresting. I played with ZFS years ago on opensolaris, I don't remember there were atomic snapshots. I definitely need to dive into ZFS again.
- jerry40 9y agoBut if we "stop" this database/schema, make ZFS snapshot (I suggest that the "partition" keeps only this database related files) it may work. It can be useful for testing purposes I guess.
- mixmastamyk 9y agoI've been disappointed in linux filesystems and Intel hardware lately. Little integrity checking in ext4 and btrfs is still having growing pains. Recent search for a svelte laptop with ECC memory yielded nothing. Sheesh, wasn't this stuff invented like 30+ years ago? I understand Intel is segmenting reliability into higher-priced business gear, but as a developer that depends on this stuff for their livelihood the current status quo is not acceptable. Linux should have better options since profit margins are not an impediment.
- edgan 9y agoThe closest you will get that I know of is a Lenovo P50/P51. They have builds with Xeons with ECC. But I remember when ECC memory dropped out of favor. I wish it had never happened.
- solotronics 9y agoall the new AMD chips have ECC by default which is exciting we will see how their mobile chips stack up
- zokier 9y agoHasn't all "big" AMD CPUs supported ECC to some degree as long as the memory controller has been on the CPU? The real problem on that side has been motherboard support which has been afaik pretty much nonexistent.
- Teknoman117 9y agoAll of the FX series CPUs supported ECC memory (albeit the unbuffered kind). As you said, finding a compatible motherboard was a pain. All the ones I found used the massively power hungry 990FX-A chipset. But I did get one, and my home NAS uses an FX-8350. Kind of scary getting an automated email every week or so about a bit flip that would have otherwise been silent corruption on my home desktop.
- throw2016 9y agoThe filesystem as basic infrastructure has to be robust and fuss free. The complex stuff is going to be built on top of that. After years of btrfs I realized while the all the features around snapshotting, send/receive etc are great the cost in performance and other issues is too high. And using plain old ext4 is more often than not the best compromise so you can forgot just about the fs and focus on higher layers.
- StavrosK 9y agoHow do you do snapshots on ext4? Don't large datasets take forever?
- bubblethink 9y agoI don't think ext4 has native snapshots. You can use lvm-thin for snapshots. Note that lvm-thin is different from regular lvm snapshots, which are known to be inefficient. RHEL has decent support for this feature, and will even let you set it up in the GUI during installation.
- StavrosK 9y agoThat snapshots the entire volume, though, right? I don't want to waste time on snapshotting my movies when all I want is my family photos... It seems that the GP's point that the fs should just store files is rather debunked, as snapshots are an extremely useful feature for a filesystem.
- simcop2387 9y agolvm-thin snapshots the entire volume but in a more Copy-on-Write like way. https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Logical_Volume_Manager_Administration/thinprovisioned_volumes.html https://access.redhat.com/documentation/en-US/Red_Hat_Enterp...
- bubblethink 9y agoYes snapshots are for volumes. There's also a yum plugin which will do before and after snapshots. Rhel and suse have this feature which works with lvm-thin as well as btrfs.
- Veratyr 9y agoThis might be somewhat off topic but I'm desperate. I've been looking for a way to store files: - Using parity rather than mirroring. I'm happy to deal with some loss of IOPS in exchange for extra usable storage. - That deals with bitrot. - That I can migrate to without somehow moving all of my files somewhere first (i.e. supports addition/removal of disks). - Is stable (doesn't frequently crash or lose data) - Is free or has transparent pricing (not "Contact Sales"). - Ideally, supports arbitrary stripe width (i.e. 2 blocks data + 1 block parity on a 6 disk array) Unfortunately it doesn't appear that a solution for this exists: - ZFS doesn't support addition of disks unless you're happy to put a RAID0 on top of your RAID5/6 and it doesn't support removal of disks at all when parity is involved. It is possible to migrate by putting giant sparse files on the existing storage, filling the filesystem, removing a sparse file, removing a disk from the original FS and "replacing" the sparse file with the actual disk but this is somewhat risky. - BTRFS has critical bugs and has been unstable even with my RAID1 filesystem. - Ceph mostly works but I always seem to run into bugs that nobody else sees. - I couldn't even figure out how to get GlusterFS to create a volume. - MDADM/hardware RAID don't deal with bitrot. - Minio has hard coded N/2 data N/2 parity erasure coding, which destroys IOPS and drastically reduces capacity in exchange for an obscene level of resiliency I don't need. - FlexRAID either isn't realtime or doesn't deal with bitrot depending which version you choose. - Windows storage spaces are slow as a dog (4 disks = 25MB/s write). - QuoByte, the successor to XtreemFS has erasure coding but has "Contact Us" pricing and trial. - Openstack Swift is complex as hell. - BcacheFS seems extremely promising but it's still in development and EC isn't available yet. I'm currently down to fixing bugs in Ceph, modifying Minio, evaluating Tahoe-LAFS and EMC ScaleIO or building my own solution.
- sfoskett 9y agoThere kinda isn't such a thing at this point. I sure wish there was! And this is the point behind the article: Wouldn't it be awesome if ZFS had continued developing as the super filesystem it looked like last decade? It would do all this and more! Frankly if ZFS supported re-balancing ("re-RAID-ing") and hybrid pools, it would be pretty much everything we all need. I do hope ZFS or Btrfs or something gets there.
- 9y ago
- Perseids 9y ago(Near) zero-cost snapshots and filesystem-based incremental backups are amazing. Just today I was saved by my auto snapshots [1]. Apparently I didn't `git add` a file to my feature branch and without the snapshot I wouldn't have been able to recover it after some extensive resetting and cleaning before I switched back to the feature branch. It's really comforting to have this easy to access [2] safety net available at all times. Now that Ubuntu has ZFS build-in by default, I'm seriously considering switching back, and since I too have been burned by Btrfs, I guess I'll stay with ZFS for quite some time. Still, the criticism of the blog post is fair, e.g. I was only able to get the RAM usage in control after I set hard lower and upper limits of the ARC as kernel boot parameters (`zfs.zfs_arc_max=1073741824 zfs.zfs_arc_min=536870912`). [1] https://github.com/zfsonlinux/zfs-auto-snapshot https://github.com/zfsonlinux/zfs-auto-snapshot [2] The coolest feature is the virtual auto mount where you can access the snapshots via the magical `.zfs` directory at the root of your filesystem.
- rsync 9y ago"(Near) zero-cost snapshots and filesystem-based incremental backups are amazing." This. We[1] offer ZFS filesystems in the cloud[2] and one of the nicest things to explain to customers is that they don't have to think about "incrementals" or "versions" or retention in any way. They can just do a "dumb rsync" to us (mirror) and our ZFS snapshots, on their schedule, will do the rest. In the event of a restore, the customer just browses right into "5 days ago"[3] and sees their entire offsite filesystem as it existed 5 days ago. [1] rsync.net [2] http://www.rsync.net/products/platform.html http://www.rsync.net/products/platform.html [3] rsync.net accounts have a .zfs directory
- cryptonector 9y agoIllumos has a way to expand pools, FYI. IDK if that's in OpenZFS yet. It works thusly: ZFS creates a vdev inside the new larger vdev, then moves all the data from the old vdev to the new vdev, then when all these moves are done the nested vdevs are enlarged. What should originally have happened is this: ZFS should have been closer to a pure CAS FS. I.e., physical block addresses should never have been part of the ZFS Merkle hash tree, thus allowing physical addresses to change without having to rewrite every block from the root down. Now, the question then becomes "how do you get the physical address of a block given just its hash?". And the answer is simple: you store the physical addresses near the logical (CAS) block pointers, and you scribble over those if you move a block. To move a block you'd first write a new copy at the new location, then overwrite the previous "cached" address. This would require some machinery to recover from failures to overwrite cached addresses: a table of in-progress moves, and even a forwarding entry format to write into the moved block's old location. A forwarding entry format would have a checksum, naturally, and would link back into the in-progress-move / move-history table. During a move (e.g., after a crash during a move) one can recover in several ways: you can go use the in-progress-moves table as journal to replay, or you can simply deref block addresses as usual and on checksum mismatch check if you read a forwarding entry or else check the in-progress-moves table. For example, an indirect block should be not an array of zfs_blkptr_t but two arrays, one of logical block pointers (just a checksum and misc metadata), and one of physical locations corresponding to blocks referenced by the first array entries. When computing the checksum of an indirect block, only the array of logical block pointers would be checksummed, thus the Merkle hash tree would never bind physical addresses. The same would apply to znodes, since they contain some block pointers, which would then have three parts: non-blockpointer metadata, an array of logical block pointers, and an array of physical block pointers. The main issue with such a design now is that it's much too hard to retrofit it into ZFS. It would have to be a new filesystem.
- raattgift 9y ago> Illumos has a way to expand pools ... ZFS creates a vdev inside the new larger vdev Huh? > IDK if that's in OpenZFS yet. The openzfs tree (on github) is virtually identical to illumos-gate (on github). > physical block addresses should never have been part of the ZFS Merkle hash tree, thus allowing physical addresses to change without having to rewrite every block from the root down. mahrens deals with this (and block pointer rewriting) here: https://www.youtube.com/watch?v=G2vIdPmsnTI#t=44m53s https://www.youtube.com/watch?v=G2vIdPmsnTI#t=44m53s Even with SSDs IOPS are precious. On rotating media, burning track-to-track seeks in reading and updating a large hash table is a bad plan (cf. the deduplication table).
- Koshkin 9y agoA logical issue that I have with the existence of such filesystems as ZFS and BTRFS is that the problem of "bit rot" should be addressed at a lower abstraction level - hardware or the driver - rather than at the level that should be primarily responsible for user-visible organization of files, directories, etc.
- wtallis 9y agoIf you have a layer that knows there's another copy of the data on a different disk, that layer should be checking for bit-rot even if the individual drives have their own error correction.
- kev009 9y agoYou must not read HW errata or device drivers. It's been posited that the main push for ZFS was perceived bugs in UFS that happened to be LSI firmware bugs once they had the capabilities of ZFS to detect them.
- dragontamer 9y agoHow? Bitrot occurs because the lower-abstraction level hardware fails. When you put a Hard-Drive into storage for say 5 years, the bits may change. Even if a Hard-drive remains in constant use for 5 years... if said files or directories aren't checked and double-checked constantly, the error-correction codes may fail over time. Its a fundamentally different problem from Hard Drives that are being used constantly as say Swap. Hard Drives typically include Hamming codes or ECC bits to address typical corruption issues. ------------- The fundamental principle at hand here is as follows: to ensure integrity of files, you need to regularly check file data. Only the Filesystem would know which files were recently checked.
- Koshkin 9y agoCouldn't the drive's firmware or the driver do the same just as well (except on physical records instead of files)?
- 9y ago
- cmurf 9y agoBtrfs might just become “the ZFS of Linux” but development has faltered lately, with a scary data loss bug derailing RAID 5 and 6 last year and not much heard since. It was not a per se data loss bug. It was Btrfs corrupting parity during scrub when encountering already (non-Btrfs) corrupted data. So a data strip is corrupt somehow, a scrub is started, Btrfs detects the corrupt data and fixes it through reconstruction with good parity, but then sometimes computes a new wrong parity strip and writes it to disk. It's a bad bug, but you're still definitely better off than you were with corrupt data. Also, this bug is fixed in kernel 4.12. https://lkml.org/lkml/2017/5/9/510 https://lkml.org/lkml/2017/5/9/510 Update, minor quibbles: lacking in Btrfs is support for flash Btrfs has such support and optimizations for flash, the gotcha though if you keep up with Btrfs development is there have been changes in FTL behavior and it's an open question whether or not these optimizations are effective for today's flash including NVMe. As for hybrid storage, that's the realm of bcache and dm-cache (managed by LVM) which should work with Btrfs as any other Linux file system. ReFS uses B+ trees (similar to Btrfs) XFS uses B+ trees, Btrfs uses B-trees.
- thibran 9y agoAnother future alternative TFS: https://github.com/redox-os/tfs https://github.com/redox-os/tfs
- fulafel 9y agoHe his talking about "best level of data protection in a small office/home office (SOHO) environment". Trying to do this with FS features is misguided. You need to have backups, and have regular practice in restoring from backups. Some organizations need fancy filesystems in addition to backups, because they want to have high availability that will bridge storage failures. But that has a high cost in complexity, you should only consider it if you have IT/sysadmin staff and the risk management says it's worth the investment in cognitive opportunity cost, IT infrastructure complexity and time spent.
- kev009 9y agoNegative, you wont know you _need_ to use the backups without these FS features, and by the time you finally do you could have rotated through them.
- fulafel 9y agoThe article didn't mention backups at all. If a SOHO environment can afford only either backups or a ZFS storage system, choosing backups leaves much less residual risk on the table. Yes, there is still a risk that corrupted data may end up in backups, but that's true even with ZFS. Ideally you want end-to-end integrity checking and verification, that means application layer and should also be done for backups. But like with all risk management, there are diminishing returns...
- Quequau 9y agoI have to wonder what's going to happen once those storage level random access non-vol memory technologies finally make it out of R&D and into the market. I mean, as it is now it seems like we have a hard enough time dealing with comparatively simple hybrid memory systems.