42 ms·

OpenZFS vs. Btrfs and other file systems (2017)

165 points by stargrave 8y ago

metildaa 8y ago
Is ZFS worth considering at this point? Btrfs seems to be quite featureful and stable, with a plethora of grocers and retailers all over the world using it. Wal-Mart, Kroger (and subsidiaries), all of IBM's old point of sale customers (now Toshiba SuperPOS iirc) all use it on things as small as deli scales all the way to running their point of sale systems and backends.
- noja 8y ago
  btrfs is not stable in some configurations (i.e. raid-5 like configs). How are Walmart using it?
  - etskinner 8y ago
    They just don't use Raid 5/6
    - rleigh 8y ago
      That's by far from the only failure case to worry about. In addition to random dataloss, the most frequent failure cause I dealt with was complete unbalancing making the filesystem read-only. This happened every ~48 hours of continuous use! I've not read that this is fixed yet, but I haven't used it in the recent past due to this making it unsuitable for production use.
- regularfry 8y ago
  Btrfs used to have really nasty behaviour on nearly-full drives. Has that been fixed?
  - unhammer 8y ago
    I've used it for backups for five years now. My USB backup drives have gotten full quite a lot of times because of old snapshots (rsync isn't able to detect moving directories), and I've had zero problems with it – just have to delete some old snapshots and wait a bit and then there's space again. I don't use RAID5, which is where I've heard btrfs is dangerous, just for the snapshots and dedupe (with https://github.com/jbruchon/jdupes/ https://github.com/jbruchon/jdupes/ )
  - jstimpfle 8y ago
    It was used at my old job on a 1TB backup drive, containing lots of hourly snapshots from around 100 other machines that also had btrfs on it (btrfs send). The snapshots were also frequently thinned (i.e. only a few of the older ones kept) to make room for new ones. The filesystem broke at least two times in two years when it was nearly full. None of the fixes we found with web searches worked. We went with formatting and copying the backups from another location, losing only a little bit of data that was not too important. That was around 2015-2017, and the version of Btrfs was about what's in Debian jessie.
- lclarkmichalek 8y ago
  Facebook also use it on millions of machines.
  - mrweasel 8y ago
    Has Facebook ever made it completely clear what they use btrfs for? I believe it was in one of the episodes of bsdnow where they talked about it "only" being used for administrative systems.
    - lclarkmichalek 8y ago
      I think that's a couple of years old. If you want more modern info, I think https://www.youtube.com/watch?v=ME4RhxwPO5A https://www.youtube.com/watch?v=ME4RhxwPO5A (skip to about 8m30) is pretty good, as is https://code.fb.com/open-source/linux/ https://code.fb.com/open-source/linux/. We also talk a lot about the benefits of having cgroup2 work well with BTRFS, which (imo) isn't something that particularly matters with administrative systems, but more for data processing/serving systems. I don't think we've released numbers or percentages, but it's safe to say every request you make to Facebook.com is processed by 1 or more machines with a btrfs filesystem. We've got a ton of value out of it, I hope we talk more about its successes publicly soon :) (I work at fb)
      - exikyut 8y ago
        Here's 8:23 pre-queued: https://youtu.be/ME4RhxwPO5A?t=503 https://youtu.be/ME4RhxwPO5A?t=503 That video's discussing btrfs layering, which is very neat. I am _very_ curious what sort of experience FB has had with btrfs. If FB has decided to go all-in with btrfs, you probably have the most accurate raw data there is to have. Of course, any hyper-scale deployment of a technology is going to produce remarkable/exponential statistics, and these will (sadly/annoyingly) need careful parsing to normalize in a way that is generally accessible and unsurprising. (We live in a very knee-jerk world, and all. Sigh) All this to say, I at least am looking forward to whatever sorts of numbers you end up being able to share. -- After considering the bit about btrfs in the video, and your mention of "1 or more machines", I wonder if you aren't using btrfs on things like load balancer type systems - or, more abstractly, node configurations that can always safely be "thrown away", whether by hardware failure or explicit decommission. But then I realize the chances are you probably use such a model (individual hardware failure must be acceptable; everything important must be n-way redundant) because nothing else makes sense at scale. And then I wonder... exactly what point in that spectrum does btrfs fit in? I wonder if/how you can answer that question.
- ayush--s 8y ago
  zfs used to have a better compression algorithm (lz4) before btrfs landed zstd with kernel version 4.15. Btrfs still doesn't have stable support for RAID 5+6 https://btrfs.wiki.kernel.org/index.php/RAID56 https://btrfs.wiki.kernel.org/index.php/RAID56
  - rleigh 8y ago
    zfs is also getting zstandard: e.g. https://reviews.freebsd.org/D11124?id=30301 https://reviews.freebsd.org/D11124?id=30301 Doesn't appear to be merged yet, but it's coming.
- zielmicha 8y ago
  - fsync on btrfs is extremly slow. The situation has improved in last 3 years, but it is still much slower than ZFS. I just did a simple test and BTFS is 12x slower than ZFS (on my customer SSD, for small writes). - RAID5/6 mode is only experimental in btrfs, while it is really stable in ZFS. - I don't have concrete data for that, but in my experience, BTRFS has high latency (>1 second) even for small file operations when under load. - While that should not be a problem for production systems, I have some crappy hardware where BTRFS oopses or corrupts data, while other filesystems (Ext4, ZFS) work fine.
  - jstimpfle 8y ago
    Yes the 12x slower seems about right. I remember Debian installations taking >1h while ext4 took < 10min. (Installations and package operations are particularly bad since dpkg does many fsync()s when unpacking). I think that was on a spinning drive, though. Anyway at my old job (where we had mainly spinning drives) we used apt and dpkg only with eatmydata. eatmydata is a command which uses LD_PRELOAD hackery to remove fsync() systemcalls.
    - rleigh 8y ago
      This was really painful. Back when I was a Debian developer tracking unstable, I would update every evening and it would sometimes take multiple hours to complete. The fsync performance is a truly awful design flaw, and while eatmydata does make it faster, it's a terrible and dangerious hack! (Which everyone used!) I even had to add eatmydata support to the schroot tool as a command-prefix option to allow every command to be run via eatmydata when using btrfs snapshots. (I've since dropped btrfs support entirely; it was too unreliable with large snapshot turnover rapidly unbalancing the filesystem. Unusable in production.)
    - andoma 8y ago
      I believe the correct solution to this problem would be that the installers would snapshot the system, install all packages without any fsync()ing at all, then finally one sync() and remove the snapshot. Optionally keeping snapshots if user want to roll back from a broken upgrade (for whatever reason). Again, as others have written here, btrfs is great if your software plays along with it, otherwise it might not be so great.
- swinglock 8y ago
  Is Btrfs worth considering? ZFS is one of the most advanced, stable and feature-complete filesystems there is, and is cross-platform. Btrfs is best known for being flakey and in mainline. Had there not been uncertainties about licensing which prevented ZFS from being in mainline there wouldn't be any Btrfs. How can you ever really trust a filesystem developed by adding features first, trying to add stability and usability later anyway?
  - Wowfunhappy 8y ago
    I don't even really understand why ZFS being outside mainline is such an issue. On Debian Stretch, I installed ZFS like any other package, and didn't even realize the kernel was getting recompiled until I read about the licensing situation later on. I did think it was odd that installing the ZFS package took such a long time, but that was a one-time wait.
- lrem 8y ago
  If you're busy and can't be arsed to research past "this has been rock-stable for a decade, so should be safe to use", ZFS is the best choice out there.
- xte 8y ago
  btrfs is designed with a developers POV, zfs with a sysadmin POV. This means that the former is a pain to use, the latter works really well.
- paulie_a 8y ago
  If you care about data not randomly failing, btrfs is not an option. While working in that specific industry it seems like there is a motivation to specifically go out of their way to make things unreliable, extra complicated and utilize bad technologies. The fact that IBM and Toshiba on POS is frightening.
throwaway12iii 8y ago
bcachefs is a really interesting FS in the linux space. Checkout some of the writing about it here: https://www.patreon.com/bcachefs https://www.patreon.com/bcachefs
- ansible 8y ago
  Well, that is interesting. I'm curious that they're talking about implementing encryption at the filesystem layer. This has typically been done elsewhere. And anyway, key management is an issue. Seems like it would be beyond the scope for a filesystem. But I had similar thoughts right before I started learning about the write anywhere layout used by NetApp all those years ago, where mixing layers had enormous benefits (ZFS too). The idea of incorporating a flash translation layer (FTL) is interesting, but there is the hardware support issue. Meaning that yes, you can still buy raw NAND flash memory, but what are you going to connect it to? NAND flash controllers which can present a useful interface to the host processor (like USB) already incorporate a FTL. Similarly, eMMC memory has a FTL layer baked in, and presents itself as a block-addressable device. Raw NAND flash controllers are increasingly rare on modern microcontrollers.
  - cyphar 8y ago
    ZFS encryption works in a similar fashion, and ext4 also has encryption (which is used by default in Android). Personally I'm not a huge fan of this because unlike the clear benefits of giving filesystems control over raw devices, full disk encryption has requirements that can't really be provided partially. You want to ensure no metadata about the filesystem structure or files will be known without knowing the encryption key but this is clearly not possible without having block device layer encryption. ZFS encryption allows for all sorts of useful operations on encrypted filesystems without having the key. This is cool, but also obviously provides a lot of information about the filesystem structure. Also dedup tables aren't encrypted in this setup. ext4 only encrypts filenames and contents, which is worse in some respects.
    - exikyut 8y ago
      > Also dedup tables aren't encrypted in this setup. This sounds like low-hanging fruit for a security researcher to try their hand at: comprehend the dedupe table format, and see what you can derive from it (basically the real-world version of https://xkcd.com/704/ https://xkcd.com/704/).
unhammer 8y ago
On that note, are there any fancy file systems that have acceptable Windows/Mac support? Or are we all doomed to using NTFS or exfat for USB drives that need to work on Linux/Mac/Windows?
- AsyncAwait 8y ago
  Apple has APFS, which might get a decent Linux driver soon, since Apple has recently published a fairly detailed description of how it works. Theoretically, nothing prevents someone writing a Windows driver too, which would allow APFS to become the modern standard for cross-platform requirements. Only I suspect that such Windows driver won't be free or common enough to make this practical.
  - Fnoord 8y ago
    There is a Windows driver for APFS from Paragon available. [1] Proprietary and commercial though. They also provide a free APFS to HFS+ conversion tool. HFS+ can be read from Linux though AFAIK without journaling support (essentially HFS). [1] https://www.paragon-software.com/home/apfs-windows/ https://www.paragon-software.com/home/apfs-windows/
    - AsyncAwait 8y ago
      I suspected somebody like Paragon would have a driver for it, but it being commercial means the adoption rate is probably minuscule and would remain so.
  - dcosson 8y ago
    I don’t get how there isn’t more demand for this, to me it’s such an obvious pain-point for personal computing these days. I guess the number of people dual booting OSes or even using external drives at all these days is so small? I feel like thumb drives are still very common though and getting large enough that it’s worth having a decent file system on them.
    - josteink 8y ago
lrem 8y ago
I'm not sure I agree with the idea that having a single open source project to rule them all is an inherently bad thing. In proprietary software, if Banana corp says that the next version of their Cavendish OS drops x86 support, you might find yourself doomed. But if Debian decides to drop x86 tomorrow, you go "Cool, good luck, I'll just fork." The project itself is never a single point of failure.
- madez 8y ago
  A hegemonial free software project isn't just good because it opposes radical changes but it is also bad because it opposes radical changes.
  - rleigh 8y ago
    For filesystems, it's a tough call. They are one of the most, if not the most, important parts of the system with respect to requiring absolute reliability. Being stable isn't necessarily a bad thing. In the case of OpenZFS, where it's implemented by multiple operating systems with differing requirements, maybe the different priorities and goals of the different teams will result in cross-pollination of ideas and implementation details rather than it stagnating top-down. I.e. there's scope for some differences in features and details, while still adhering to the common design.
- tinus_hn 8y ago
  Systemd
  - CogitoCogito 8y ago
    > Systemd I'm a bit confused by this. Irem wrote: > But if Debian decides to drop x86 tomorrow, you go "Cool, good luck, I'll just fork." When Systemd made its way into debian, debian _was_ forked by devuan (and possibly by others) to keep it out. So...is your point exactly? edit: Maybe I misunderstood entirely and you were using Systemd as an example _supporting_ Irem's post...
    - 8y ago
em-bee 8y ago
ZFS is not suitable for desktop computers. as for example that one anecdote in the comments of the article shows, ZFS doesn't work well on a single disk. the author of the comment is confused by the behavior, and doesn't even consider that this might be by design. another point that shows that ZFS is unsuitable for the desktop is it's reliance on EEC RAM: https://forums.freenas.org/index.php?threads/ecc-vs-non-ecc-ram-and-zfs.15449/ https://forums.freenas.org/index.php?threads/ecc-vs-non-ecc-... i considered getting EEC RAM for my machine just because of ZFS, but i could not find any 16GB strips to fill my board. that's two strikes against desktop use, and enough to let me prefer btrfs. EDIT: i am very happy to see so many replies proving me wrong. when i last researched this topic i was not able to find any counter arguments to needing EEC. greetings, eMBee.
- binaryapparatus 8y ago
  I have been using ZFS for over two years, on any desktop that I control, as both root file system for FreeBSD setups and backup file system in many configurations. In almost all cases I don't have EEC ram and I don't need it really. Your comment is either outdated or wrong, ZFS is THE go-to filesystem on desktop for me. Edit: oh and it works beautifully with single disk setups too.
  - amelius 8y ago
    > In almost all cases I don't have EEC ram and I don't need it really. That's like saying you don't need health insurance because you haven't got ill.
    - binaryapparatus 8y ago
      ECC is nice to have but absolutely not a show stopper if you don't have it. Plenty of articles and discussions online, example: http://jrs-s.net/2015/02/03/will-zfs-and-non-ecc-ram-kill-your-data/ http://jrs-s.net/2015/02/03/will-zfs-and-non-ecc-ram-kill-yo... Claim that you can only use ZFS with ECC memory to feel safe is false.
      - 8y ago
blinkingled 8y ago
HAMMER2/DragonFly is already looking great with cow/snaphots, compression, live dedup etc and one of its design goals is to work great on low memory systems. Once it is feature complete with clustering support it'll be a great choice to have. Although looks like it'll be DragonFly only.
- xte 8y ago
  This. Also the author do not mention that LLNL zfs Linux port is not as production ready than FreeBSD one. Personally I use nilfs2 on top of lvm since is Linux mainline from many many years, it a logfs like Hammer volumes, so it protect effectively against accidental overwrite/deletion and can be live resized both in grow and shrink so coupled with LVM can be a poor man's pooled storage... But Hammer on Linux will be a success like OpenSSH!
  - blinkingled 8y ago
    I was running DragonFly/HAMMER1 on ESXi with physical disks attached. This is my NAS Backup with Samba. Got it migrated over to HAMMER 2 recently and it's pretty sweet.
- pmoriarty 8y ago
  Is there any reason preventing it from being ported to other BSDs, or Linux even?
  - blinkingled 8y ago
    From my reading in the past primarily because cross OS portability was not a design goal for hammer. This lets them take advantage of tighter integration with DragonFly's VM and VFS subsystems.
    - takeda 8y ago
      Hmm, I saw Matt Dillon's post on FreeBSD mailing list offering Hammer to be ported to FBSD, but looked like there was not much enthusiasm about from the FBSD team. It's possible it was Hammer1 and because of that he integrated Hammer2 more tightly with DFBSD.
sirmike_ 8y ago
Opinions are good, bad and ugly. Sometimes all at once. Thanks for letting us know one nerd's take on openzfs. The only thing standing in zfs way (to 1st tier Linux support) is a company springing up and heading the charge. 75 percent sure this will happen eventually with ZoL at some point in a decade's time. However, another point of measure is money. Synology is huge and seems to think btrfs is good enough for business. They compete with ixsystems directly. We could go all day and into 2020 with points and counter points. The licensing issue is weak but will eventually be moot with enough money and time behind the right project. I'm not in any position to judge that last point for sure just offering my IT pro/business take on it. To be fair bsd-ish-ness has a role to play on the backend and is bullet proof and battle tested for the right mission. But the same can be said for Linux too. The last point I'll make is that today, I can run the Linux kernel from my msft windows 10 machine natively. No one here would have saw that in 2000. Does bsd, zfs have a similar anecdotal Hero Epic like this; where the enemy capitulated so thoroughly?
- rleigh 8y ago
  Dont' we already have the beginnings of such support with Canonical's support of ZFS in the Ubuntu Linux kernel? I'm writing this reply on a system running ZFS as the rootfs. % findmnt / TARGET SOURCE FSTYPE OPTIONS / rpool/ROOT/default zfs rw,relatime,xattr,noacl Regarding Linux kernel on Windows 10, if you're referring to The Linux subsystem, there's no Linux kernel there, it's a Windows subsystem implementing the Linux system calls directly. Unless you're referring to full Hyper-V virtualisation, where you can run FreeBSD just as you can run Linux.
  - sirmike_ 8y ago
    I also didn't think the canonical effort went anywhere? Maybe I'm wrong. Re:wsl You are correct thank you for the catch. I was missing a few key words.
    - rleigh 8y ago
      I'm not sure what you mean by "went anywhere". It's perfectly functional as it stands. The main missing piece is proper support in the installer for using it as a root filesystem. Currently this requires manual setup, which is tedious, but even this is functional once done.
hawski 8y ago
I was always hoping for more adoption of NILFS [0] - a log structured FS with continuous snapshots. In theory it should never lose data and that's a reasoning behind it not having fsck. I've seen a blog post that was debating this choice and presenting a few situations when fsck would be necessary, but I can't find it. It would be great for /home for many normal users - they could always go back. That could also be a problem when one wants to make sure that a sensitive file is really removed. It could be also used at least for some kind of archive. Possibly also for development. [0] https://en.wikipedia.org/wiki/NILFS https://en.wikipedia.org/wiki/NILFS
- exikyut 8y ago
  I was curious how "never lose data" works given that HDDs do not provide infinite capacity: the log is circular. This makes NILFS good for NAND flash. The continuous per-fwrite snapshotting functionality sounds awesome - I've had a few (minor but annoying) accidents with rm of late, so I'd definitely appreciate an rm-proof undelete. I was trying to figure out if NILFS had any sort of COW functionality, then realized, d'oh, of course it effectively does, via the log. To address the elephant in the room - checksumming - unfortunately NILFS doesn't (yet?) do this. The WP article links to https://www.spinics.net/lists/linux-nilfs/msg01063.html https://www.spinics.net/lists/linux-nilfs/msg01063.html, which states that the CRC32Cs that are written to disk are used only for the remount/recovery stage, and are apparently not suitable for realtime verification.
- xte 8y ago
  Using it on root and nearly all personal fses I second, especially hoping that it's development speed up a bit, the cleaner it's still a pain and a small wrapper to diffs between cp's can be a very nice addition!
- rodgerd 8y ago
  Unfortunately the mindshare it could have had was killed by Linus announcing btrfs was the future, long before it was close to ready. Almost a decade of wasted opportunity...
nimbius 8y ago
anecdotal report: Im running a medium sized MariaDB database of appointments and invoice backups on a 4 disk BTRFS filesystem with deduplication. snapshots work, its stable, and it performs well. I feel like so much of BTRFS though was death by committee. It was trying to do every single thing ZFS did, regardless of whether or not the specific feature from ZFS was mediocre to begin with. focusing on a release cycle and communicating would have helped it alot instead of vague things like "it is ready enough" and "some features arent ready"
- eeperson 8y ago
  BTRFS does provide a status for all the major features: https://btrfs.wiki.kernel.org/index.php/Status https://btrfs.wiki.kernel.org/index.php/Status
elfchief 8y ago
Is it just me or is it absolutely insane that zfs and btrfs are the only common filesystems out there that do data checksumming? I don't want or need the extra complexity of either of them in a lot of cases, but I'd sure as hell like to know if my data is corrupt...
- ianlevesque 8y ago
  Yeah it should be standard. Equally crazy is that APFS is brand new and didn't add it. I belive the justification was that the hardware does it on flash memory for their devices, but ZFS really exposed how unreliable hardware is.
  - AlphaSite 8y ago
    I mean Apple has their own flash controllers, so they’re really the only company that owns the FS and the flash controller. So really they can depend on it.
    - skrause 8y ago
      What about external hard disks with terrible USB controllers? That's where data checksumming would be even more important and where Apple doesn't produce the hardware.
      - tobias3 8y ago
        Maybe APFS was mainly developed for iOS?
      - derefr 8y ago
        Apple doesn’t recommend using APFS for external media.
        pvg 8y ago
        Really? Can you remember where you've seen this non-recommendation?
jplayer01 8y ago
It's a travesty that so few people care about moving to more reliable filesystems. A decade later and we're still stuck with ext4 and now btrfs, which I do use nowadays but I'm absolutely aware of the high risk of data corruption because the underlying code is terrible. Windows is even worse with NTFS, which is a complete pile of garbage. I've fixed hundreds of computers with NTFS corruption over my lifetime and it's mind-boggling that Microsoft is fine with this shit.
- zaarn 8y ago
  The thing about ext4 is that it's sufficiently mediocre at what it does. It's fairly crash resistant, has well worn and battle-tested codepaths and generally seems to not loose data outside bitrot. So why pick btrfs with it's dataloss stories or ZFS with it's complex management? It's an effort to switch to a new FS as default, especially if either itself or the integration is not as well tested or completed yet. So everyone just sticks to the thing they know how it sucks.
  - snuxoll 8y ago
    Maybe it’s because I’ve already been using it for years, but I don’t really consider ZFS any more difficult to manage than the DM/LVM/*FS stack I work with on most Linux systems. Compared to just running gdisk and using raw partitions - sure, but I’ve been using LVM for over a decade now.
    - zaarn 8y ago
      ZFS is a lot more complicated if you're expecting the average consumer to handle it. It doesn't even handle 1-disk-storage expansion. Even Synology and QNAP now have plug-and-play solutions to expand a RAID one disk at a time. You can even start plugging in higher capacity disks and once you have enough extra parity, the higher capacity becomes used magically. No need for wasteful RAID10/50/60 setups. Just plug in more drives.
      - RantyDave 8y ago
InTheArena 8y ago
The challenge here is that ZFS is dominate by Oracle right now, and if anyone thinks that Oracle is going to act in the best interest of the community, rather then killing the golden goose to peel the meat off it's bones, take a look at what Oracle has done with Java recently. Or what they did with Solaris. Or what they do with Oracle Cloud. in addition, we know that there are patent landmines out there because NetApp and Oracle have a (undisclosed) settlement on the issue. Using ZFS is a probably good way to get a visit from your legendary friendly Oracle sales agent and auditor. Oracle could fix all of this with a quick re-license or patent indemnification, but they haven't and almost certainly won't. My Synology (and all modern Synologies) uses BTRFS, and it seems pretty solid so far. I would have preferred that ZFS had matured and been dominate in FS, but there are some really risky things that come with ZFS.
- snuxoll 8y ago
  ZFS is not dominated by Oracle, Open-ZFS is the source of record for anything other than Solaris and critical features are being added that Oracle cannot use without opening their branch back up under the CDDL. The CDDL also includes a patent agreement, Oracle can’t retroactively waive it.
  - InTheArena 8y ago
    Every single one of your statements was also true of Java. Go look at what the Java community is going through right now.
    - simion314 8y ago
      >Go look at what the Java community is going through right now. You did not understand what happened with the latest Oracle Java news and you are spreading FUD. People have a choice what Java implementation to use, they can pay for latest Oracle implementations, use Open JDK (it is included in Debian and most distributions), you can get support from RedHat and other big vendors. There is no risk of getting in trouble by using Open JDK, MS and Google had trouble because they "forked" Java
paulie_a 8y ago
Is btrfs still a thing? Reading other people's experiences, atleast a couple distros dropping it as a default, and my personal experience of it crashing completely. I'm surprised anyone would even consider it as an option. It simply isn't a reliable file system.
- zlynx 8y ago
  As personal experiences go, btrfs works great for me. I've been using it on a 6 drive NAS since 2012, and on two laptops with SSDs. In my experience "crashes" are symptoms of people who didn't read the documentation and don't understand how to recover. For example, on the laptops I've often run out of disk space. And btrfs can run into a situation where it can't clean up partly used blocks without more disk space. To recover you have to add more space. I usually used the swap partition, but a USB stick also works. Or a RAM drive, but that's risky. Some other issues I've had to help people with is snapshot deletion. Deleting a btrfs snapshot appears instant but none of the used space comes back. That's because it is doing it in the background and it can take a really long time. OTOH, ext4 does exactly the same thing with background delete of very large files.
  - cookiecaper 8y ago
    >In my experience "crashes" are symptoms of people who didn't read the documentation and don't understand how to recover. aka most users. Filesystems must be reasonably resilient to incorrect usage. This is a problem across the board; if you're not careful setting up a ZFS pool, it's easy to accidentally create a scenario where you're unwittingly striping across multiple devices, greatly increasing the likelihood of a catastrophic data loss event without ensuring the user is fully aware.
  - paulie_a 8y ago
    I've lost multiple terabytes with btrfs on multiple different systems. It was essentially a random occurance. These were all cases of simply using it as a regular filesystem. I read the docs I attempted recovery. They didn't work. And quite frankly I shouldn't need to take those steps anyways. As for your example that seems insane someone should have to jump through hoops like that to utilize a hard drive. Btrfs is junk and I am glad it has been removed as the default on a couple distros.
zmix 8y ago
Hot title. Boring article.
zmix 8y ago
Is ZFS still incapable of enlarging/extending a filesystem by adding new disks to it without rebuilding the whole FS infrastructure?
- snuxoll 8y ago
  You’re confusing the expansion of a ZFS pool with the expansion of a RAID-Z vdev. Throwing new disks into a pool has always been possible, adding them to an existing RAID-Z vdev isn’t (yet, block pointer rewrite and RAID-Z expansion are coming). I run exclusively with mirrored vdev’s and have never run into issues growing my pool.
  - Wowfunhappy 8y ago
    Is block pointer rewrite actually coming? It seems like it has been stalled for years and years...
    - snuxoll 8y ago
      There was just a talk on it this year, it’s coming but between the time it takes to go stable in OpenZFS and the time it takes to hit downstream (FreeBSD, ZoL) it might not be for a while yet.
- aidenn0 8y ago
  It's been able to do online enlarge and extend forever, and it will even automatically enlarge if you replace disks in a vdev one at a time allowing for resilvering. Perhaps you're thinking of shrinking, which it still cannot do without rebuilding the whole FS infrastructure.
- cookiecaper 8y ago
  While I definitely sympathize with the desire here, mirrored disks are almost always the best way to go. It has saved my butt just in my home setup a couple of times, most recently when a power connector got loose and started shutting a disk on and off basically at random. Mirrored disk mean quick recovery from such scenarios and relatively-limited impact during degradation. My opinion after trying both parity-checked redundancy and true mirrored redundancy is that any serious data storage should be strictly on mirrored volumes (preferably with disks of substantially different ages so you don't risk them dying within a couple weeks of each other, possibly before a replacement gets installed). I once read "50% storage efficiency is plenty, don't be greedy". Although painful for the home user, that seems like the right attitude to take for important stuff, especially as volume sizes increase. It can be a very slow and very fraught process to recover several TBs from parity data.
Mister_Snuggles 8y ago
I've been burned by Btrfs enough times to avoid it. I had a VM running OpenSuse Tumbleweed - I know, rolling release, don't expect stability, etc - and it would consistently freeze after being up and unused for a few days. Rebuilt the VM but used XFS instead of Btrfs, it was rock solid. This happened within the last four months. At work, we have a number of machines using Btrfs. At some point they get into a state where the weekly Btrfs scrub process hangs the machine for hours at a time. The solution in every case was to move that filesystem onto a non-Btrfs volume. This is an ongoing issue, but I'm not in the team responsible so I don't know what they're doing with this. Anecdotally, this seems to be a problem when there is a lot of writing going on with the volumes. Machines that use Btrfs root but do most of their work on NFS volumes don't seem to experience this issue. I also have a laptop running OpenSuse Tumbleweed with Btrfs. One day it somehow got itself into a state where it would hang when mounting the filesystem read/write, but it would be fine if it was mounted read-only. I can't recall exactly what happened to cause it, but I'm pretty sure I didn't do anything exceptional with the computer - I think I rebooted it from the GUI while it was doing a scrub in the background, but that's just a guess. I think this happened about a year ago. This is all disappointing because some of the things Btrfs enables are awesome. In particular, OpenSuse's snapper has saved me countless hour of work. I haven't played with the send/receive stuff, but it seems like a great solution for backups. In the meantime, I'm using ZFS on a variety of FreeBSD boxes with a variety of workloads (including one running more VMs than its RAM should support) and it's been rock solid. At this point it will be a long time before I give Btrfs another look. It has the potential to be really good, and I want it to be good, but I don't believe it's there yet.
- cookiecaper 8y ago
  Yes, this is similar to my fairly recent experience with btrfs. While it's competitive on paper and there are several meta-arguments in its favor (being in mainline, for example), the cold hard facts just don't stack up. btrfs fares far worse and requires much more tuning than a generic ZFS installation. A great example is that if you ever try to run a database or a large virtual machine on a btrfs mount, you'll quickly become acquainted with chattr +C for "nocow". Any write-intensive applications grinds the filesystem to a halt until you disable the copy-on-write functionality for that file specifically, making it so snapshots, etc., are ineffective and the FS stops tracking the changes that would make them possible. Whereas on ZFS, like any application that uses fixed block sizes, virtualization and databases _may_ require some alignment tuning to avoid a read-modify-write cycle, and ZFS works without fuss from that moment on. I know that Facebook is investing heavily in btrfs right now, but to be frank, I don't see the point. btrfs doesn't have its youth as an excuse anymore and the core issues that exist seem like they're pretty fundamental. It's a gargantuan waste of effort when people could just take the tack that Ubuntu has taken and decide all that FUD around the CDDL, kicked up by fears that the commercial Unices were plotting to take their revenge on Linux, that maybe all that isn't super relevant anymore, and they can just start shipping ZoL.
madhadron 8y ago
Quick note on btrfs: unless you've been dealing with the latest kernels in the last few months, reserve judgement. Facebook has put a truly ridiculous number of hours on that filesystem in the last year or so and fixed a concomitant number of issues.
- walrus01 8y ago
  From what I'm seeing in this thread, basically two things: People who have been running btrfs in production (reckless, imho) on kernels that are 2.5+ years old or worse. And then they're surprised and disappointed when it loses data. For everyone who's posted here with a btrfs data loss story, it would be helpful to quantify your anecdote with what kernel you were running.
  - deno 8y ago
    Top post in this thread is about btrfs is on Tumbleweed, which is not an old kernel. My experiences have been similar. Btrfs is a great backup stress tool. I have no idea how Facebook is running it in production…
    - rb2k_ 8y ago
      The "how" consists mostly of reasonably up-to-date kernels that track main line as much as possible. As far as why, https://code.fb.com/open-source/linux/ https://code.fb.com/open-source/linux/ has a nice quote: "Btrfs has played a role in increasing efficiency and resource utilization in Facebook’s data centers in a number of different applications. Recently, Btrfs helped eliminate priority inversions caused by the journaling behavior of the previous filesystem, when used for I/O control with cgroup2 (described below). Btrfs is the only filesystem implementation that currently works with resource isolation, and it’s now deployed on millions of servers, driving significant efficiency gains."
      - deno 8y ago
        Is there any reason cgroup2 io can’t be implemented in non-CoW fs?
aidenn0 8y ago
ZoL and Btrfs are the only two filesystems that I've ever had rendered unmountable without any reported disk error. In both cases, ECC ram was in use, and mirrors were used. Btrfs was a known bug, and the kind folks on #btrfs were able to help me recover (though it was a quite involved process). For ZFS, the consensus in #zfs and #zfsonlinux was that with the error message I was getting "I hope you have backups."[1] I see XFS mentioned here, but in the admittedly unusaly use-case of editing a file that when compiled and modprobed may hang the kernel, I found that vim was insufficiently fsync-ing to ensure I had valid data (either old or new copy would have been acceptible). I don't know what my point was with this comment except "All filesystems are buggy, so backup your data" 1: It should be noted that in the case of ZFS, I was on vacation and it was my laptop that had no critical data, but I absolutely needed it working for other things, so I only spent a couple hours trying to recover; The btrfs issue happened at home where I had plenty other systems to use and where I was more motivated to recover the data.
linsomniac 8y ago
I've been super happy with ZFS for over a decade. All the pain I've had with it has been related to deduplication requiring massive amounts of RAM. When you don't have the RAM, it tends to be flaky. But I've never run into data loss with it, despite my best attempts. My current backup server would need ~50GB of RAM just for the in-RAM deduplication tables. I've been thinking of trying btrfs for the backup server, but the deduplication story is not something that's obvious, it all seems to be third-party, which makes me skeptical. But it worth trying. I might just try a test Dragonfly setup, because my simple testing of HAMMER seemed very promising, years ago. I've had pretty bad experiences with btrfs in the distant past. Around a decade ago I had all my company laptops using btrfs and it worked great, as long as we didn't get >80% usage or something. It worked well for around a year. But then the reliability of btrfs dropped, and we had repeated data loss and switched back to ext.
- poizan42 8y ago
  > I've had pretty bad experiences with btrfs in the distant past. Around a decade ago I had all my company laptops using btrfs and it worked great, as long as we didn't get >80% usage or something. It worked well for around a year. But then the reliability of btrfs dropped, and we had repeated data loss and switched back to ext. btrfs wasn't considered stable until 2012. Using an experimental filesystem for all your company laptops seems to be asking for trouble.
- mbrumlow 8y ago
  You really can't blame ZFS for needing ram for dedup. The raw math around looking up checksum just demands it. There are some other systems that can achieve fast dedup, but instead of ram they would require a ton of super fast SSDs to store the tables. There are some other types of dedup that don't guarantee 100% dedup, but then the dedup levels would very depending on data sets -- this may be fine for VMs and user data. They work by being using data locality, predicting which dedup table would needed to dedup the next N bytes. But it is not likely we are going to see this any time soon because the patents on this sort of dedup is fairly recent.
  - LeoPanthera 8y ago
CathyWest 8y ago
I get it during the Sun years, but why the hell does Oracle continue developing Btrfs when they can just release ZFS under a GPL-compatible license and be done with it? It reallys seems like they are working against their own interests here.
- coredog64 8y ago
  They’re still selling Solaris licenses to enterprises. If they keep the FUD churning they can continue to suck what is essentially free money from those customers.
  - CathyWest 8y ago
    Does a GPL-compatible ZFS really detract from their FUD churning Solaris business, though? Those customers can't redistribute Solaris any more than they can redistribute Linux with a CDDL-licensed ZFS. If they really wanted to switch to Linux they could do so today and be in the exact same legal position license-wise. Not to mention they could switch to FreeBSD and be able to redistribute, but again, I doubt this is something very many Solaris users care about. Perhaps the cost of porting their applications away from Solaris is too high, or their middle managers are Oracle fanboys, whatever the reason is, a GPL-compatible ZFS does not threaten that business model. I think a more likely reason why they pour money into the Btrfs death march while having a turnkey solution just sitting around in the next office over waiting to be singed off by the legal department as CDDL+GPL or Apache or whatever is that each project is the pet of a different pointy-haired boss and they refuse to cooperate internally.

165 points by stargrave 8y ago

Is ZFS worth considering at this point? Btrfs seems to be quite featureful and stable, with a plethora of grocers and retailers all over the world using it. Wal-Mart, Kroger (and subsidiaries), all of IBM's old point of sale customers (now Toshiba SuperPOS iirc) all use it on things as small as deli scales all the way to running their point of sale systems and backends.

noja 8y ago
btrfs is not stable in some configurations (i.e. raid-5 like configs). How are Walmart using it?
- etskinner 8y ago
  They just don't use Raid 5/6
  - rleigh 8y ago
    That's by far from the only failure case to worry about. In addition to random dataloss, the most frequent failure cause I dealt with was complete unbalancing making the filesystem read-only. This happened every ~48 hours of continuous use! I've not read that this is fixed yet, but I haven't used it in the recent past due to this making it unsuitable for production use.
regularfry 8y ago
Btrfs used to have really nasty behaviour on nearly-full drives. Has that been fixed?
- unhammer 8y ago
  I've used it for backups for five years now. My USB backup drives have gotten full quite a lot of times because of old snapshots (rsync isn't able to detect moving directories), and I've had zero problems with it – just have to delete some old snapshots and wait a bit and then there's space again. I don't use RAID5, which is where I've heard btrfs is dangerous, just for the snapshots and dedupe (with https://github.com/jbruchon/jdupes/ https://github.com/jbruchon/jdupes/ )
- jstimpfle 8y ago
  It was used at my old job on a 1TB backup drive, containing lots of hourly snapshots from around 100 other machines that also had btrfs on it (btrfs send). The snapshots were also frequently thinned (i.e. only a few of the older ones kept) to make room for new ones. The filesystem broke at least two times in two years when it was nearly full. None of the fixes we found with web searches worked. We went with formatting and copying the backups from another location, losing only a little bit of data that was not too important. That was around 2015-2017, and the version of Btrfs was about what's in Debian jessie.
lclarkmichalek 8y ago
Facebook also use it on millions of machines.
- mrweasel 8y ago
  Has Facebook ever made it completely clear what they use btrfs for? I believe it was in one of the episodes of bsdnow where they talked about it "only" being used for administrative systems.
  - lclarkmichalek 8y ago
    I think that's a couple of years old. If you want more modern info, I think https://www.youtube.com/watch?v=ME4RhxwPO5A https://www.youtube.com/watch?v=ME4RhxwPO5A (skip to about 8m30) is pretty good, as is https://code.fb.com/open-source/linux/ https://code.fb.com/open-source/linux/. We also talk a lot about the benefits of having cgroup2 work well with BTRFS, which (imo) isn't something that particularly matters with administrative systems, but more for data processing/serving systems. I don't think we've released numbers or percentages, but it's safe to say every request you make to Facebook.com is processed by 1 or more machines with a btrfs filesystem. We've got a ton of value out of it, I hope we talk more about its successes publicly soon :) (I work at fb)
    - exikyut 8y ago
      Here's 8:23 pre-queued: https://youtu.be/ME4RhxwPO5A?t=503 https://youtu.be/ME4RhxwPO5A?t=503 That video's discussing btrfs layering, which is very neat. I am _very_ curious what sort of experience FB has had with btrfs. If FB has decided to go all-in with btrfs, you probably have the most accurate raw data there is to have. Of course, any hyper-scale deployment of a technology is going to produce remarkable/exponential statistics, and these will (sadly/annoyingly) need careful parsing to normalize in a way that is generally accessible and unsurprising. (We live in a very knee-jerk world, and all. Sigh) All this to say, I at least am looking forward to whatever sorts of numbers you end up being able to share. -- After considering the bit about btrfs in the video, and your mention of "1 or more machines", I wonder if you aren't using btrfs on things like load balancer type systems - or, more abstractly, node configurations that can always safely be "thrown away", whether by hardware failure or explicit decommission. But then I realize the chances are you probably use such a model (individual hardware failure must be acceptable; everything important must be n-way redundant) because nothing else makes sense at scale. And then I wonder... exactly what point in that spectrum does btrfs fit in? I wonder if/how you can answer that question.
ayush--s 8y ago
zfs used to have a better compression algorithm (lz4) before btrfs landed zstd with kernel version 4.15. Btrfs still doesn't have stable support for RAID 5+6 https://btrfs.wiki.kernel.org/index.php/RAID56 https://btrfs.wiki.kernel.org/index.php/RAID56
- rleigh 8y ago
  zfs is also getting zstandard: e.g. https://reviews.freebsd.org/D11124?id=30301 https://reviews.freebsd.org/D11124?id=30301 Doesn't appear to be merged yet, but it's coming.
zielmicha 8y ago
- fsync on btrfs is extremly slow. The situation has improved in last 3 years, but it is still much slower than ZFS. I just did a simple test and BTFS is 12x slower than ZFS (on my customer SSD, for small writes). - RAID5/6 mode is only experimental in btrfs, while it is really stable in ZFS. - I don't have concrete data for that, but in my experience, BTRFS has high latency (>1 second) even for small file operations when under load. - While that should not be a problem for production systems, I have some crappy hardware where BTRFS oopses or corrupts data, while other filesystems (Ext4, ZFS) work fine.
- jstimpfle 8y ago
  Yes the 12x slower seems about right. I remember Debian installations taking >1h while ext4 took < 10min. (Installations and package operations are particularly bad since dpkg does many fsync()s when unpacking). I think that was on a spinning drive, though. Anyway at my old job (where we had mainly spinning drives) we used apt and dpkg only with eatmydata. eatmydata is a command which uses LD_PRELOAD hackery to remove fsync() systemcalls.
  - rleigh 8y ago
    This was really painful. Back when I was a Debian developer tracking unstable, I would update every evening and it would sometimes take multiple hours to complete. The fsync performance is a truly awful design flaw, and while eatmydata does make it faster, it's a terrible and dangerious hack! (Which everyone used!) I even had to add eatmydata support to the schroot tool as a command-prefix option to allow every command to be run via eatmydata when using btrfs snapshots. (I've since dropped btrfs support entirely; it was too unreliable with large snapshot turnover rapidly unbalancing the filesystem. Unusable in production.)
  - andoma 8y ago
    I believe the correct solution to this problem would be that the installers would snapshot the system, install all packages without any fsync()ing at all, then finally one sync() and remove the snapshot. Optionally keeping snapshots if user want to roll back from a broken upgrade (for whatever reason). Again, as others have written here, btrfs is great if your software plays along with it, otherwise it might not be so great.
swinglock 8y ago
Is Btrfs worth considering? ZFS is one of the most advanced, stable and feature-complete filesystems there is, and is cross-platform. Btrfs is best known for being flakey and in mainline. Had there not been uncertainties about licensing which prevented ZFS from being in mainline there wouldn't be any Btrfs. How can you ever really trust a filesystem developed by adding features first, trying to add stability and usability later anyway?
- Wowfunhappy 8y ago
  I don't even really understand why ZFS being outside mainline is such an issue. On Debian Stretch, I installed ZFS like any other package, and didn't even realize the kernel was getting recompiled until I read about the licensing situation later on. I did think it was odd that installing the ZFS package took such a long time, but that was a one-time wait.
lrem 8y ago
If you're busy and can't be arsed to research past "this has been rock-stable for a decade, so should be safe to use", ZFS is the best choice out there.
xte 8y ago
btrfs is designed with a developers POV, zfs with a sysadmin POV. This means that the former is a pain to use, the latter works really well.
paulie_a 8y ago
If you care about data not randomly failing, btrfs is not an option. While working in that specific industry it seems like there is a motivation to specifically go out of their way to make things unreliable, extra complicated and utilize bad technologies. The fact that IBM and Toshiba on POS is frightening.

throwaway12iii 8y ago

bcachefs is a really interesting FS in the linux space. Checkout some of the writing about it here: https://www.patreon.com/bcachefs https://www.patreon.com/bcachefs

ansible 8y ago
Well, that is interesting. I'm curious that they're talking about implementing encryption at the filesystem layer. This has typically been done elsewhere. And anyway, key management is an issue. Seems like it would be beyond the scope for a filesystem. But I had similar thoughts right before I started learning about the write anywhere layout used by NetApp all those years ago, where mixing layers had enormous benefits (ZFS too). The idea of incorporating a flash translation layer (FTL) is interesting, but there is the hardware support issue. Meaning that yes, you can still buy raw NAND flash memory, but what are you going to connect it to? NAND flash controllers which can present a useful interface to the host processor (like USB) already incorporate a FTL. Similarly, eMMC memory has a FTL layer baked in, and presents itself as a block-addressable device. Raw NAND flash controllers are increasingly rare on modern microcontrollers.
- cyphar 8y ago
  ZFS encryption works in a similar fashion, and ext4 also has encryption (which is used by default in Android). Personally I'm not a huge fan of this because unlike the clear benefits of giving filesystems control over raw devices, full disk encryption has requirements that can't really be provided partially. You want to ensure no metadata about the filesystem structure or files will be known without knowing the encryption key but this is clearly not possible without having block device layer encryption. ZFS encryption allows for all sorts of useful operations on encrypted filesystems without having the key. This is cool, but also obviously provides a lot of information about the filesystem structure. Also dedup tables aren't encrypted in this setup. ext4 only encrypts filenames and contents, which is worse in some respects.
  - exikyut 8y ago
    > Also dedup tables aren't encrypted in this setup. This sounds like low-hanging fruit for a security researcher to try their hand at: comprehend the dedupe table format, and see what you can derive from it (basically the real-world version of https://xkcd.com/704/ https://xkcd.com/704/).

unhammer 8y ago

On that note, are there any fancy file systems that have acceptable Windows/Mac support? Or are we all doomed to using NTFS or exfat for USB drives that need to work on Linux/Mac/Windows?

AsyncAwait 8y ago
Apple has APFS, which might get a decent Linux driver soon, since Apple has recently published a fairly detailed description of how it works. Theoretically, nothing prevents someone writing a Windows driver too, which would allow APFS to become the modern standard for cross-platform requirements. Only I suspect that such Windows driver won't be free or common enough to make this practical.
- Fnoord 8y ago
  There is a Windows driver for APFS from Paragon available. [1] Proprietary and commercial though. They also provide a free APFS to HFS+ conversion tool. HFS+ can be read from Linux though AFAIK without journaling support (essentially HFS). [1] https://www.paragon-software.com/home/apfs-windows/ https://www.paragon-software.com/home/apfs-windows/
  - AsyncAwait 8y ago
    I suspected somebody like Paragon would have a driver for it, but it being commercial means the adoption rate is probably minuscule and would remain so.
- dcosson 8y ago
  I don’t get how there isn’t more demand for this, to me it’s such an obvious pain-point for personal computing these days. I guess the number of people dual booting OSes or even using external drives at all these days is so small? I feel like thumb drives are still very common though and getting large enough that it’s worth having a decent file system on them.
  - josteink 8y ago

lrem 8y ago

I'm not sure I agree with the idea that having a single open source project to rule them all is an inherently bad thing. In proprietary software, if Banana corp says that the next version of their Cavendish OS drops x86 support, you might find yourself doomed. But if Debian decides to drop x86 tomorrow, you go "Cool, good luck, I'll just fork." The project itself is never a single point of failure.

madez 8y ago
A hegemonial free software project isn't just good because it opposes radical changes but it is also bad because it opposes radical changes.
- rleigh 8y ago
  For filesystems, it's a tough call. They are one of the most, if not the most, important parts of the system with respect to requiring absolute reliability. Being stable isn't necessarily a bad thing. In the case of OpenZFS, where it's implemented by multiple operating systems with differing requirements, maybe the different priorities and goals of the different teams will result in cross-pollination of ideas and implementation details rather than it stagnating top-down. I.e. there's scope for some differences in features and details, while still adhering to the common design.
tinus_hn 8y ago
Systemd
- CogitoCogito 8y ago
  > Systemd I'm a bit confused by this. Irem wrote: > But if Debian decides to drop x86 tomorrow, you go "Cool, good luck, I'll just fork." When Systemd made its way into debian, debian _was_ forked by devuan (and possibly by others) to keep it out. So...is your point exactly? edit: Maybe I misunderstood entirely and you were using Systemd as an example _supporting_ Irem's post...
  - 8y ago

em-bee 8y ago

ZFS is not suitable for desktop computers. as for example that one anecdote in the comments of the article shows, ZFS doesn't work well on a single disk. the author of the comment is confused by the behavior, and doesn't even consider that this might be by design. another point that shows that ZFS is unsuitable for the desktop is it's reliance on EEC RAM: https://forums.freenas.org/index.php?threads/ecc-vs-non-ecc-ram-and-zfs.15449/ https://forums.freenas.org/index.php?threads/ecc-vs-non-ecc-... i considered getting EEC RAM for my machine just because of ZFS, but i could not find any 16GB strips to fill my board. that's two strikes against desktop use, and enough to let me prefer btrfs. EDIT: i am very happy to see so many replies proving me wrong. when i last researched this topic i was not able to find any counter arguments to needing EEC. greetings, eMBee.

binaryapparatus 8y ago
I have been using ZFS for over two years, on any desktop that I control, as both root file system for FreeBSD setups and backup file system in many configurations. In almost all cases I don't have EEC ram and I don't need it really. Your comment is either outdated or wrong, ZFS is THE go-to filesystem on desktop for me. Edit: oh and it works beautifully with single disk setups too.
- amelius 8y ago
  > In almost all cases I don't have EEC ram and I don't need it really. That's like saying you don't need health insurance because you haven't got ill.
  - binaryapparatus 8y ago
    ECC is nice to have but absolutely not a show stopper if you don't have it. Plenty of articles and discussions online, example: http://jrs-s.net/2015/02/03/will-zfs-and-non-ecc-ram-kill-your-data/ http://jrs-s.net/2015/02/03/will-zfs-and-non-ecc-ram-kill-yo... Claim that you can only use ZFS with ECC memory to feel safe is false.
    - 8y ago

blinkingled 8y ago

HAMMER2/DragonFly is already looking great with cow/snaphots, compression, live dedup etc and one of its design goals is to work great on low memory systems. Once it is feature complete with clustering support it'll be a great choice to have. Although looks like it'll be DragonFly only.

xte 8y ago
This. Also the author do not mention that LLNL zfs Linux port is not as production ready than FreeBSD one. Personally I use nilfs2 on top of lvm since is Linux mainline from many many years, it a logfs like Hammer volumes, so it protect effectively against accidental overwrite/deletion and can be live resized both in grow and shrink so coupled with LVM can be a poor man's pooled storage... But Hammer on Linux will be a success like OpenSSH!
- blinkingled 8y ago
  I was running DragonFly/HAMMER1 on ESXi with physical disks attached. This is my NAS Backup with Samba. Got it migrated over to HAMMER 2 recently and it's pretty sweet.
pmoriarty 8y ago
Is there any reason preventing it from being ported to other BSDs, or Linux even?
- blinkingled 8y ago
  From my reading in the past primarily because cross OS portability was not a design goal for hammer. This lets them take advantage of tighter integration with DragonFly's VM and VFS subsystems.
  - takeda 8y ago
    Hmm, I saw Matt Dillon's post on FreeBSD mailing list offering Hammer to be ported to FBSD, but looked like there was not much enthusiasm about from the FBSD team. It's possible it was Hammer1 and because of that he integrated Hammer2 more tightly with DFBSD.

sirmike_ 8y ago

Opinions are good, bad and ugly. Sometimes all at once. Thanks for letting us know one nerd's take on openzfs. The only thing standing in zfs way (to 1st tier Linux support) is a company springing up and heading the charge. 75 percent sure this will happen eventually with ZoL at some point in a decade's time. However, another point of measure is money. Synology is huge and seems to think btrfs is good enough for business. They compete with ixsystems directly. We could go all day and into 2020 with points and counter points. The licensing issue is weak but will eventually be moot with enough money and time behind the right project. I'm not in any position to judge that last point for sure just offering my IT pro/business take on it. To be fair bsd-ish-ness has a role to play on the backend and is bullet proof and battle tested for the right mission. But the same can be said for Linux too. The last point I'll make is that today, I can run the Linux kernel from my msft windows 10 machine natively. No one here would have saw that in 2000. Does bsd, zfs have a similar anecdotal Hero Epic like this; where the enemy capitulated so thoroughly?

rleigh 8y ago
Dont' we already have the beginnings of such support with Canonical's support of ZFS in the Ubuntu Linux kernel? I'm writing this reply on a system running ZFS as the rootfs. % findmnt / TARGET SOURCE FSTYPE OPTIONS / rpool/ROOT/default zfs rw,relatime,xattr,noacl Regarding Linux kernel on Windows 10, if you're referring to The Linux subsystem, there's no Linux kernel there, it's a Windows subsystem implementing the Linux system calls directly. Unless you're referring to full Hyper-V virtualisation, where you can run FreeBSD just as you can run Linux.
- sirmike_ 8y ago
  I also didn't think the canonical effort went anywhere? Maybe I'm wrong. Re:wsl You are correct thank you for the catch. I was missing a few key words.
  - rleigh 8y ago
    I'm not sure what you mean by "went anywhere". It's perfectly functional as it stands. The main missing piece is proper support in the installer for using it as a root filesystem. Currently this requires manual setup, which is tedious, but even this is functional once done.

hawski 8y ago

I was always hoping for more adoption of NILFS [0] - a log structured FS with continuous snapshots. In theory it should never lose data and that's a reasoning behind it not having fsck. I've seen a blog post that was debating this choice and presenting a few situations when fsck would be necessary, but I can't find it. It would be great for /home for many normal users - they could always go back. That could also be a problem when one wants to make sure that a sensitive file is really removed. It could be also used at least for some kind of archive. Possibly also for development. [0] https://en.wikipedia.org/wiki/NILFS https://en.wikipedia.org/wiki/NILFS

exikyut 8y ago
I was curious how "never lose data" works given that HDDs do not provide infinite capacity: the log is circular. This makes NILFS good for NAND flash. The continuous per-fwrite snapshotting functionality sounds awesome - I've had a few (minor but annoying) accidents with rm of late, so I'd definitely appreciate an rm-proof undelete. I was trying to figure out if NILFS had any sort of COW functionality, then realized, d'oh, of course it effectively does, via the log. To address the elephant in the room - checksumming - unfortunately NILFS doesn't (yet?) do this. The WP article links to https://www.spinics.net/lists/linux-nilfs/msg01063.html https://www.spinics.net/lists/linux-nilfs/msg01063.html, which states that the CRC32Cs that are written to disk are used only for the remount/recovery stage, and are apparently not suitable for realtime verification.
xte 8y ago
Using it on root and nearly all personal fses I second, especially hoping that it's development speed up a bit, the cleaner it's still a pain and a small wrapper to diffs between cp's can be a very nice addition!
rodgerd 8y ago
Unfortunately the mindshare it could have had was killed by Linus announcing btrfs was the future, long before it was close to ready. Almost a decade of wasted opportunity...

nimbius 8y ago

anecdotal report: Im running a medium sized MariaDB database of appointments and invoice backups on a 4 disk BTRFS filesystem with deduplication. snapshots work, its stable, and it performs well. I feel like so much of BTRFS though was death by committee. It was trying to do every single thing ZFS did, regardless of whether or not the specific feature from ZFS was mediocre to begin with. focusing on a release cycle and communicating would have helped it alot instead of vague things like "it is ready enough" and "some features arent ready"

eeperson 8y ago
BTRFS does provide a status for all the major features: https://btrfs.wiki.kernel.org/index.php/Status https://btrfs.wiki.kernel.org/index.php/Status

elfchief 8y ago

Is it just me or is it absolutely insane that zfs and btrfs are the only common filesystems out there that do data checksumming? I don't want or need the extra complexity of either of them in a lot of cases, but I'd sure as hell like to know if my data is corrupt...

ianlevesque 8y ago
Yeah it should be standard. Equally crazy is that APFS is brand new and didn't add it. I belive the justification was that the hardware does it on flash memory for their devices, but ZFS really exposed how unreliable hardware is.
- AlphaSite 8y ago
  I mean Apple has their own flash controllers, so they’re really the only company that owns the FS and the flash controller. So really they can depend on it.
  - skrause 8y ago
    What about external hard disks with terrible USB controllers? That's where data checksumming would be even more important and where Apple doesn't produce the hardware.
    - tobias3 8y ago
      Maybe APFS was mainly developed for iOS?
    - derefr 8y ago
      Apple doesn’t recommend using APFS for external media.
      - pvg 8y ago
        Really? Can you remember where you've seen this non-recommendation?

jplayer01 8y ago

It's a travesty that so few people care about moving to more reliable filesystems. A decade later and we're still stuck with ext4 and now btrfs, which I do use nowadays but I'm absolutely aware of the high risk of data corruption because the underlying code is terrible. Windows is even worse with NTFS, which is a complete pile of garbage. I've fixed hundreds of computers with NTFS corruption over my lifetime and it's mind-boggling that Microsoft is fine with this shit.

zaarn 8y ago
The thing about ext4 is that it's sufficiently mediocre at what it does. It's fairly crash resistant, has well worn and battle-tested codepaths and generally seems to not loose data outside bitrot. So why pick btrfs with it's dataloss stories or ZFS with it's complex management? It's an effort to switch to a new FS as default, especially if either itself or the integration is not as well tested or completed yet. So everyone just sticks to the thing they know how it sucks.
- snuxoll 8y ago
  Maybe it’s because I’ve already been using it for years, but I don’t really consider ZFS any more difficult to manage than the DM/LVM/*FS stack I work with on most Linux systems. Compared to just running gdisk and using raw partitions - sure, but I’ve been using LVM for over a decade now.
  - zaarn 8y ago
    ZFS is a lot more complicated if you're expecting the average consumer to handle it. It doesn't even handle 1-disk-storage expansion. Even Synology and QNAP now have plug-and-play solutions to expand a RAID one disk at a time. You can even start plugging in higher capacity disks and once you have enough extra parity, the higher capacity becomes used magically. No need for wasteful RAID10/50/60 setups. Just plug in more drives.
    - RantyDave 8y ago

InTheArena 8y ago

The challenge here is that ZFS is dominate by Oracle right now, and if anyone thinks that Oracle is going to act in the best interest of the community, rather then killing the golden goose to peel the meat off it's bones, take a look at what Oracle has done with Java recently. Or what they did with Solaris. Or what they do with Oracle Cloud. in addition, we know that there are patent landmines out there because NetApp and Oracle have a (undisclosed) settlement on the issue. Using ZFS is a probably good way to get a visit from your legendary friendly Oracle sales agent and auditor. Oracle could fix all of this with a quick re-license or patent indemnification, but they haven't and almost certainly won't. My Synology (and all modern Synologies) uses BTRFS, and it seems pretty solid so far. I would have preferred that ZFS had matured and been dominate in FS, but there are some really risky things that come with ZFS.

snuxoll 8y ago
ZFS is not dominated by Oracle, Open-ZFS is the source of record for anything other than Solaris and critical features are being added that Oracle cannot use without opening their branch back up under the CDDL. The CDDL also includes a patent agreement, Oracle can’t retroactively waive it.
- InTheArena 8y ago
  Every single one of your statements was also true of Java. Go look at what the Java community is going through right now.
  - simion314 8y ago
    >Go look at what the Java community is going through right now. You did not understand what happened with the latest Oracle Java news and you are spreading FUD. People have a choice what Java implementation to use, they can pay for latest Oracle implementations, use Open JDK (it is included in Debian and most distributions), you can get support from RedHat and other big vendors. There is no risk of getting in trouble by using Open JDK, MS and Google had trouble because they "forked" Java

paulie_a 8y ago

Is btrfs still a thing? Reading other people's experiences, atleast a couple distros dropping it as a default, and my personal experience of it crashing completely. I'm surprised anyone would even consider it as an option. It simply isn't a reliable file system.

zlynx 8y ago
As personal experiences go, btrfs works great for me. I've been using it on a 6 drive NAS since 2012, and on two laptops with SSDs. In my experience "crashes" are symptoms of people who didn't read the documentation and don't understand how to recover. For example, on the laptops I've often run out of disk space. And btrfs can run into a situation where it can't clean up partly used blocks without more disk space. To recover you have to add more space. I usually used the swap partition, but a USB stick also works. Or a RAM drive, but that's risky. Some other issues I've had to help people with is snapshot deletion. Deleting a btrfs snapshot appears instant but none of the used space comes back. That's because it is doing it in the background and it can take a really long time. OTOH, ext4 does exactly the same thing with background delete of very large files.
- cookiecaper 8y ago
  >In my experience "crashes" are symptoms of people who didn't read the documentation and don't understand how to recover. aka most users. Filesystems must be reasonably resilient to incorrect usage. This is a problem across the board; if you're not careful setting up a ZFS pool, it's easy to accidentally create a scenario where you're unwittingly striping across multiple devices, greatly increasing the likelihood of a catastrophic data loss event without ensuring the user is fully aware.
- paulie_a 8y ago
  I've lost multiple terabytes with btrfs on multiple different systems. It was essentially a random occurance. These were all cases of simply using it as a regular filesystem. I read the docs I attempted recovery. They didn't work. And quite frankly I shouldn't need to take those steps anyways. As for your example that seems insane someone should have to jump through hoops like that to utilize a hard drive. Btrfs is junk and I am glad it has been removed as the default on a couple distros.

zmix 8y ago

Hot title. Boring article.

zmix 8y ago

Is ZFS still incapable of enlarging/extending a filesystem by adding new disks to it without rebuilding the whole FS infrastructure?

snuxoll 8y ago
You’re confusing the expansion of a ZFS pool with the expansion of a RAID-Z vdev. Throwing new disks into a pool has always been possible, adding them to an existing RAID-Z vdev isn’t (yet, block pointer rewrite and RAID-Z expansion are coming). I run exclusively with mirrored vdev’s and have never run into issues growing my pool.
- Wowfunhappy 8y ago
  Is block pointer rewrite actually coming? It seems like it has been stalled for years and years...
  - snuxoll 8y ago
    There was just a talk on it this year, it’s coming but between the time it takes to go stable in OpenZFS and the time it takes to hit downstream (FreeBSD, ZoL) it might not be for a while yet.
aidenn0 8y ago
It's been able to do online enlarge and extend forever, and it will even automatically enlarge if you replace disks in a vdev one at a time allowing for resilvering. Perhaps you're thinking of shrinking, which it still cannot do without rebuilding the whole FS infrastructure.
cookiecaper 8y ago
While I definitely sympathize with the desire here, mirrored disks are almost always the best way to go. It has saved my butt just in my home setup a couple of times, most recently when a power connector got loose and started shutting a disk on and off basically at random. Mirrored disk mean quick recovery from such scenarios and relatively-limited impact during degradation. My opinion after trying both parity-checked redundancy and true mirrored redundancy is that any serious data storage should be strictly on mirrored volumes (preferably with disks of substantially different ages so you don't risk them dying within a couple weeks of each other, possibly before a replacement gets installed). I once read "50% storage efficiency is plenty, don't be greedy". Although painful for the home user, that seems like the right attitude to take for important stuff, especially as volume sizes increase. It can be a very slow and very fraught process to recover several TBs from parity data.

Mister_Snuggles 8y ago

I've been burned by Btrfs enough times to avoid it. I had a VM running OpenSuse Tumbleweed - I know, rolling release, don't expect stability, etc - and it would consistently freeze after being up and unused for a few days. Rebuilt the VM but used XFS instead of Btrfs, it was rock solid. This happened within the last four months. At work, we have a number of machines using Btrfs. At some point they get into a state where the weekly Btrfs scrub process hangs the machine for hours at a time. The solution in every case was to move that filesystem onto a non-Btrfs volume. This is an ongoing issue, but I'm not in the team responsible so I don't know what they're doing with this. Anecdotally, this seems to be a problem when there is a lot of writing going on with the volumes. Machines that use Btrfs root but do most of their work on NFS volumes don't seem to experience this issue. I also have a laptop running OpenSuse Tumbleweed with Btrfs. One day it somehow got itself into a state where it would hang when mounting the filesystem read/write, but it would be fine if it was mounted read-only. I can't recall exactly what happened to cause it, but I'm pretty sure I didn't do anything exceptional with the computer - I think I rebooted it from the GUI while it was doing a scrub in the background, but that's just a guess. I think this happened about a year ago. This is all disappointing because some of the things Btrfs enables are awesome. In particular, OpenSuse's snapper has saved me countless hour of work. I haven't played with the send/receive stuff, but it seems like a great solution for backups. In the meantime, I'm using ZFS on a variety of FreeBSD boxes with a variety of workloads (including one running more VMs than its RAM should support) and it's been rock solid. At this point it will be a long time before I give Btrfs another look. It has the potential to be really good, and I want it to be good, but I don't believe it's there yet.

cookiecaper 8y ago
Yes, this is similar to my fairly recent experience with btrfs. While it's competitive on paper and there are several meta-arguments in its favor (being in mainline, for example), the cold hard facts just don't stack up. btrfs fares far worse and requires much more tuning than a generic ZFS installation. A great example is that if you ever try to run a database or a large virtual machine on a btrfs mount, you'll quickly become acquainted with chattr +C for "nocow". Any write-intensive applications grinds the filesystem to a halt until you disable the copy-on-write functionality for that file specifically, making it so snapshots, etc., are ineffective and the FS stops tracking the changes that would make them possible. Whereas on ZFS, like any application that uses fixed block sizes, virtualization and databases _may_ require some alignment tuning to avoid a read-modify-write cycle, and ZFS works without fuss from that moment on. I know that Facebook is investing heavily in btrfs right now, but to be frank, I don't see the point. btrfs doesn't have its youth as an excuse anymore and the core issues that exist seem like they're pretty fundamental. It's a gargantuan waste of effort when people could just take the tack that Ubuntu has taken and decide all that FUD around the CDDL, kicked up by fears that the commercial Unices were plotting to take their revenge on Linux, that maybe all that isn't super relevant anymore, and they can just start shipping ZoL.

madhadron 8y ago

Quick note on btrfs: unless you've been dealing with the latest kernels in the last few months, reserve judgement. Facebook has put a truly ridiculous number of hours on that filesystem in the last year or so and fixed a concomitant number of issues.

walrus01 8y ago
From what I'm seeing in this thread, basically two things: People who have been running btrfs in production (reckless, imho) on kernels that are 2.5+ years old or worse. And then they're surprised and disappointed when it loses data. For everyone who's posted here with a btrfs data loss story, it would be helpful to quantify your anecdote with what kernel you were running.
- deno 8y ago
  Top post in this thread is about btrfs is on Tumbleweed, which is not an old kernel. My experiences have been similar. Btrfs is a great backup stress tool. I have no idea how Facebook is running it in production…
  - rb2k_ 8y ago
    The "how" consists mostly of reasonably up-to-date kernels that track main line as much as possible. As far as why, https://code.fb.com/open-source/linux/ https://code.fb.com/open-source/linux/ has a nice quote: "Btrfs has played a role in increasing efficiency and resource utilization in Facebook’s data centers in a number of different applications. Recently, Btrfs helped eliminate priority inversions caused by the journaling behavior of the previous filesystem, when used for I/O control with cgroup2 (described below). Btrfs is the only filesystem implementation that currently works with resource isolation, and it’s now deployed on millions of servers, driving significant efficiency gains."
    - deno 8y ago
      Is there any reason cgroup2 io can’t be implemented in non-CoW fs?

aidenn0 8y ago

ZoL and Btrfs are the only two filesystems that I've ever had rendered unmountable without any reported disk error. In both cases, ECC ram was in use, and mirrors were used. Btrfs was a known bug, and the kind folks on #btrfs were able to help me recover (though it was a quite involved process). For ZFS, the consensus in #zfs and #zfsonlinux was that with the error message I was getting "I hope you have backups."[1] I see XFS mentioned here, but in the admittedly unusaly use-case of editing a file that when compiled and modprobed may hang the kernel, I found that vim was insufficiently fsync-ing to ensure I had valid data (either old or new copy would have been acceptible). I don't know what my point was with this comment except "All filesystems are buggy, so backup your data" 1: It should be noted that in the case of ZFS, I was on vacation and it was my laptop that had no critical data, but I absolutely needed it working for other things, so I only spent a couple hours trying to recover; The btrfs issue happened at home where I had plenty other systems to use and where I was more motivated to recover the data.

linsomniac 8y ago

I've been super happy with ZFS for over a decade. All the pain I've had with it has been related to deduplication requiring massive amounts of RAM. When you don't have the RAM, it tends to be flaky. But I've never run into data loss with it, despite my best attempts. My current backup server would need ~50GB of RAM just for the in-RAM deduplication tables. I've been thinking of trying btrfs for the backup server, but the deduplication story is not something that's obvious, it all seems to be third-party, which makes me skeptical. But it worth trying. I might just try a test Dragonfly setup, because my simple testing of HAMMER seemed very promising, years ago. I've had pretty bad experiences with btrfs in the distant past. Around a decade ago I had all my company laptops using btrfs and it worked great, as long as we didn't get >80% usage or something. It worked well for around a year. But then the reliability of btrfs dropped, and we had repeated data loss and switched back to ext.

poizan42 8y ago
> I've had pretty bad experiences with btrfs in the distant past. Around a decade ago I had all my company laptops using btrfs and it worked great, as long as we didn't get >80% usage or something. It worked well for around a year. But then the reliability of btrfs dropped, and we had repeated data loss and switched back to ext. btrfs wasn't considered stable until 2012. Using an experimental filesystem for all your company laptops seems to be asking for trouble.
mbrumlow 8y ago
You really can't blame ZFS for needing ram for dedup. The raw math around looking up checksum just demands it. There are some other systems that can achieve fast dedup, but instead of ram they would require a ton of super fast SSDs to store the tables. There are some other types of dedup that don't guarantee 100% dedup, but then the dedup levels would very depending on data sets -- this may be fine for VMs and user data. They work by being using data locality, predicting which dedup table would needed to dedup the next N bytes. But it is not likely we are going to see this any time soon because the patents on this sort of dedup is fairly recent.
- LeoPanthera 8y ago

CathyWest 8y ago

I get it during the Sun years, but why the hell does Oracle continue developing Btrfs when they can just release ZFS under a GPL-compatible license and be done with it? It reallys seems like they are working against their own interests here.

coredog64 8y ago
They’re still selling Solaris licenses to enterprises. If they keep the FUD churning they can continue to suck what is essentially free money from those customers.
- CathyWest 8y ago
  Does a GPL-compatible ZFS really detract from their FUD churning Solaris business, though? Those customers can't redistribute Solaris any more than they can redistribute Linux with a CDDL-licensed ZFS. If they really wanted to switch to Linux they could do so today and be in the exact same legal position license-wise. Not to mention they could switch to FreeBSD and be able to redistribute, but again, I doubt this is something very many Solaris users care about. Perhaps the cost of porting their applications away from Solaris is too high, or their middle managers are Oracle fanboys, whatever the reason is, a GPL-compatible ZFS does not threaten that business model. I think a more likely reason why they pour money into the Btrfs death march while having a turnkey solution just sitting around in the next office over waiting to be singed off by the legal department as CDDL+GPL or Apache or whatever is that each project is the pet of a different pointy-haired boss and they refuse to cooperate internally.