5 ms·

Can you speak more about why ZFS is better than BTRFS?

nisa 12y ago
I'm not the op but I'm running ZoL on a small cluster with 200 HDDs and 50 machines. I've also tested btrfs. Why is ZFS better (IMHO): + Tooling. The zpool and zfs commands are clear, easy to use and well documented. btrfs e.g. has no way to get a list of files with checksum problems. Also automounting and setting flexible mount points as well as the whole zfs get/set concept felt like a really concise way to configure all aspects easily. + Architecture. ZFS uses merkle hash trees with multiple copies spread over the disk (ditto blocks). This means that disk failures likely won't affect metadata and every possible disk problem is accounted for (except main memory). btrfs only uses crc32c and has 2 copies of the metadata tree on disk (but not spread over the disk afaik) or simple RAID1 in case of multiple disks. So in case of disk problems btrfs tends to corrupt faster. I've actually tested this and it's almost impossible to corrupt ZFS metadata even with hundreds of bad blocks. btrfs tends to switch faster to read-only mode with bad disks. + Compression: LZ4 is slightly better than LZO + L2ARC/ARC - easy to plug in a SSD cache. ARC is better than the pagecache for most workloads. + It's really fun to work with. A lot of good documentation. Dealing with btrfs felt very different... + RAIDZ(2-3) and copies=n for datasets. Both don't exist in btrfs (yet). Reasons against ZoL: - Memory. I've got out of memory errors because ARC memory is not freed fast enough. Also ZFS hogs a lot of memory. At least 750MB to 1GB kernel memory with several disks. btrfs needs far less. There is upcoming work to fix the ARC out of memory issues through.. it's on the roadmap from the developers. - Stability. This also applies to btrfs but it's not as rock-solid as ext4. Especially with broken disks and usage of advanced features you'll run eventually into bugs. But the "normal" operations where 100% stable for me with ZoL. - It's not native Linux. You have some Solaris Porting layer modules and integration into the Linux kernel is not as tight as you may like it to be. E.g. cgroups based io throttling was not possible, last time I looked. This may change in the future but it's work to integrate it and if e.g. the module does not not compile for a new kernel you are screwed. - btrfs will be supported and be a default choice in 1-2 years in likely all major Linux enterprise distributions. SLES already made it the default. For most workloads ZFS is likely not worth it. At least it's tough to sell against a default option. - rootfs on a ZoL volume is quite a lot of work and difficult to get working. But this is largely a distribution problem.
- ryao 12y ago
  Below are a few remarks: To expand on your LZ4 comment, LZ4 is 3 times faster than LZO at decompression: https://code.google.com/p/lz4/ https://code.google.com/p/lz4/ However, LZ4's real innovation is on incompressible data. I do not have a reference at the moment, but there have been benchmarks on the userland lz4 tool on incompressible data that show it processing 10GB/sec. This is because LZ4 uses a hash table that enables it to give up when trying to compress data far sooner than other lempel-ziv implementations do. As for stability, it is hard to for me to tell what you mean by that, but I just wrote a series of blog posts on the topic. If there are any outstanding issues not listed there, please file an issue on github: https://github.com/zfsonlinux/zfs/issues/new https://github.com/zfsonlinux/zfs/issues/new I also encourage you to file an issue requesting support for cgroup bsaed IO throttling. You are the first person to mention it. As for not compiling with new kernels, the 0.6.3 release supports Linux 2.6.26 to Linux 3.16 while HEAD adds support for Linux 3.17 release candidates. I briefly touched on the rootfs on ZFS issue in my blog post. I plan to add ZFS support to syslinux in the near future as per a discussion that I had with Peter Alvin at LinuxCon North America 2014. I expect this to resolve the issue for those willing to use syslinux as their bootloader.
  - nisa 12y ago
    Wow! Thanks for your reply. I've experienced some minor issues with severe broken disks where the SPL layer discards the disk but the zpool is still online. It's not a major issue and I've yet to encounter the issue again I've already filed a related bug: https://github.com/zfsonlinux/zfs/issues/2508 https://github.com/zfsonlinux/zfs/issues/2508 but I'm not an expert. If I can gather more data I'll try to submit as much data and hints I can find. As for cgroups blkio - there is already a bug report: https://github.com/zfsonlinux/zfs/issues/1952 https://github.com/zfsonlinux/zfs/issues/1952 Sorry. It was not my intention to suggest that ZFS does not compile on new kernel versions. I've never had problems with that. I just wanted to point out that it's not in the mainline kernel and for some people that might be important. Thanks for your great work on ZoL! It made my life a lot easier :)
    - ryao 12y ago
jaytaylor 12y ago
BTRFS is still maturing. Kernel bugs involving BTRFS are still popping up pretty regularly.
deleted 12y ago
[deleted]
ryao 12y ago
A btrfs versus ZFS comparison probably deserves a blog post of its own, but I will try to address your question. I wrote the following on this topic last year: https://groups.google.com/d/msg/funtoo-dev/g9OY_vqVpCM/VTKF8Ef9ab4J https://groups.google.com/d/msg/funtoo-dev/g9OY_vqVpCM/VTKF8... However, significant time has passed and it requires some corrections to be current: 1. I have not heard of any recent data corruption issues in btrfs, although I have not looked into them lately. 2. btrfs now has experimental RAID 5/6 support, but is neither production ready nor as refined as ZFS' raidz. 3. I should have said "inline block-based data deduplication". You can (ab)use reflinks to achieve a file-level data deduplication in btrfs, but it is not quite the same. btrfs now has a bedup tool that makes using reflinks somewhat easier now: https://btrfs.wiki.kernel.org/index.php/Deduplication https://btrfs.wiki.kernel.org/index.php/Deduplication 4. btrfs now has some kind of incremental send/recv operation. However, it is not clear to me how it handles consistency issues from having "write-able snapshots": https://btrfs.wiki.kernel.org/index.php/Incremental_Backup https://btrfs.wiki.kernel.org/index.php/Incremental_Backup 5. Illumos' ZFS implementation is now able to store small files in the dnode, which is improves its efficiency when storing small files in a manner similar to btrfs' block suballocation. This feature will likely be in ZoL 0.6.4. Aside from those corrections, what I wrote in that mailing list email should still be relevant today. However, there are a few advantages that ZFS has over btrfs that I recall offhand that I do not see here there or in nisa's reply: 0. ZFS uses 256-bit checksums with algorithms that are still considered to be good today. btrfs uses checksum algorithms that are known to be weak. In specific, btrfs uses CRC32 on 32-bit processors and CRC64 on 64-bit processors. CRC32 is the same algorithm used by TCP/IP. Its deficiencies are well documented: http://noahdavids.org/self_published/CRC_and_checksum.html http://noahdavids.org/self_published/CRC_and_checksum.html I have not examined CRC64, but I am not particularly confident in it. btrfs should have room in its on-disk data structures that would allow it to implement 256-bit checksums in a future disk format extension, but until then, its checksum implementations are vastly inferior. 1. The ztest utility that I described in the blog allows ZFS developers to catch issues that would have otherwise gone into production and debug them from userland. No other filesystem has something like quite like it. 2. ZFSOnLinux is the only kernel filesystem driver that is kernel version-independent, so if you are unable to upgrade your kernel, you can still get fixes. The inability of people to always update their kernels is an issue Linus mentioned at LinuxCon North America 2014. 3. The CDDL gives the ZFSOnLinux a patent grant for the ZFS patent portfolio. This is something that btrfs does not have and will likely never have unless Oracle decides to provide one. Consequently, Oracle is the only company in the world that I know is able to ship products incorporating the btrfs source code without being at risk should btrfs infringe on one of the dozens if not hundreds of patents in the ZFS patent portfolio. A small subset of them can be accessed from the Jeff Bonwick Wikipedia page: https://en.wikipedia.org/wiki/Jeff_Bonwick https://en.wikipedia.org/wiki/Jeff_Bonwick
DiabloD3 12y ago
Working RAID 5 and 6, RAID "7" (Z3[1], triple parity, where Z/5 is single, and Z2/6 is dual), tiered RAID setups (such as JBOD'ed RAID-Z3s), zvols (as in, but not limited to, swap partitions in ZFS), more nuanced multi-controller and failover/spare setups, write-only journals (ZIL), second tier caching (L2ARC), LZ4 transparent compression, better designed snapshot and snapshot cloning support, more mature CLI tools, support on other OS's (with shared code bases for bonus points), configurable checksum algos, and a few other things that I'm forgetting at the moment. [1]: Seen that 90 drive Supermicro drive chassis[2]? Three 29-drive RAID-Z3s with 3 hot spares (which is shared across the three RAIDs) in a single storage pool (round robin-esqued), and you could plug that into a 2U with 16 small SSDs (like those Crucial M550 128GBs I use now) for the ZIL/L2ARC farming, and then have, well, near-infinite IO performance. Good luck trying to assemble that with BTRFS. [2]: A wet dream for CEPH users, too.
- mdellabitta 12y ago
  Re: [1], isn't that far too many drives in one RAIDZ? Referencing this: http://www.solarisinternals.com/wiki/index.php/ZFS_Best_Practices_Guide#RAIDZ_Configuration_Requirements_and_Recommendations http://www.solarisinternals.com/wiki/index.php/ZFS_Best_Prac...
  - DiabloD3 12y ago
    Yes and no. The optimal number for RAIDZ/Z1/Z2, claimed by many benchmarks, is (power of two count of drives) plus (number of drives for parity)[1], with the upper bound being the likelihood of another drive failing before rebuild can complete. There is a tradeoff of, for example, using 3 RAIDZs each with a third of the drives in your pool, or 1 RAIDZ3 using all the drives in your pool. In the first case, a rebuild takes less time, but the chances of all data in that RAIDZ being lost is (I think) 4 times higher (but statistics is not my strong point; it may be higher). In the second case, a rebuild takes longer, the chances of another drive failing before rebuild is complete is higher, however, you require three failures before you have a problem; the array is still fully protected from another drive failure after two drive failures. The numbers in the example I gave, however, are not quite optimum for performance reasons (you'd be better off with 16 or 24), but most people don't like tons of hotspares (even though that case likely eats drives like old Sun thumpers would). All of that said, it depends entirely on how much you want data reliability. And, also, it was just an example. Not necessarily a good one. I was just illustrating that ZFS can handle that many drives with ease, but BTRFS would not be a good fit. [1]: There seems to be no difference between power of two and half way steps in real world performance when the number of drives are big enough, as in, 8, 12, and 16 will all perform similarly.

nisa 12y ago

I'm not the op but I'm running ZoL on a small cluster with 200 HDDs and 50 machines. I've also tested btrfs. Why is ZFS better (IMHO): + Tooling. The zpool and zfs commands are clear, easy to use and well documented. btrfs e.g. has no way to get a list of files with checksum problems. Also automounting and setting flexible mount points as well as the whole zfs get/set concept felt like a really concise way to configure all aspects easily. + Architecture. ZFS uses merkle hash trees with multiple copies spread over the disk (ditto blocks). This means that disk failures likely won't affect metadata and every possible disk problem is accounted for (except main memory). btrfs only uses crc32c and has 2 copies of the metadata tree on disk (but not spread over the disk afaik) or simple RAID1 in case of multiple disks. So in case of disk problems btrfs tends to corrupt faster. I've actually tested this and it's almost impossible to corrupt ZFS metadata even with hundreds of bad blocks. btrfs tends to switch faster to read-only mode with bad disks. + Compression: LZ4 is slightly better than LZO + L2ARC/ARC - easy to plug in a SSD cache. ARC is better than the pagecache for most workloads. + It's really fun to work with. A lot of good documentation. Dealing with btrfs felt very different... + RAIDZ(2-3) and copies=n for datasets. Both don't exist in btrfs (yet). Reasons against ZoL: - Memory. I've got out of memory errors because ARC memory is not freed fast enough. Also ZFS hogs a lot of memory. At least 750MB to 1GB kernel memory with several disks. btrfs needs far less. There is upcoming work to fix the ARC out of memory issues through.. it's on the roadmap from the developers. - Stability. This also applies to btrfs but it's not as rock-solid as ext4. Especially with broken disks and usage of advanced features you'll run eventually into bugs. But the "normal" operations where 100% stable for me with ZoL. - It's not native Linux. You have some Solaris Porting layer modules and integration into the Linux kernel is not as tight as you may like it to be. E.g. cgroups based io throttling was not possible, last time I looked. This may change in the future but it's work to integrate it and if e.g. the module does not not compile for a new kernel you are screwed. - btrfs will be supported and be a default choice in 1-2 years in likely all major Linux enterprise distributions. SLES already made it the default. For most workloads ZFS is likely not worth it. At least it's tough to sell against a default option. - rootfs on a ZoL volume is quite a lot of work and difficult to get working. But this is largely a distribution problem.

ryao 12y ago
Below are a few remarks: To expand on your LZ4 comment, LZ4 is 3 times faster than LZO at decompression: https://code.google.com/p/lz4/ https://code.google.com/p/lz4/ However, LZ4's real innovation is on incompressible data. I do not have a reference at the moment, but there have been benchmarks on the userland lz4 tool on incompressible data that show it processing 10GB/sec. This is because LZ4 uses a hash table that enables it to give up when trying to compress data far sooner than other lempel-ziv implementations do. As for stability, it is hard to for me to tell what you mean by that, but I just wrote a series of blog posts on the topic. If there are any outstanding issues not listed there, please file an issue on github: https://github.com/zfsonlinux/zfs/issues/new https://github.com/zfsonlinux/zfs/issues/new I also encourage you to file an issue requesting support for cgroup bsaed IO throttling. You are the first person to mention it. As for not compiling with new kernels, the 0.6.3 release supports Linux 2.6.26 to Linux 3.16 while HEAD adds support for Linux 3.17 release candidates. I briefly touched on the rootfs on ZFS issue in my blog post. I plan to add ZFS support to syslinux in the near future as per a discussion that I had with Peter Alvin at LinuxCon North America 2014. I expect this to resolve the issue for those willing to use syslinux as their bootloader.
- nisa 12y ago
  Wow! Thanks for your reply. I've experienced some minor issues with severe broken disks where the SPL layer discards the disk but the zpool is still online. It's not a major issue and I've yet to encounter the issue again I've already filed a related bug: https://github.com/zfsonlinux/zfs/issues/2508 https://github.com/zfsonlinux/zfs/issues/2508 but I'm not an expert. If I can gather more data I'll try to submit as much data and hints I can find. As for cgroups blkio - there is already a bug report: https://github.com/zfsonlinux/zfs/issues/1952 https://github.com/zfsonlinux/zfs/issues/1952 Sorry. It was not my intention to suggest that ZFS does not compile on new kernel versions. I've never had problems with that. I just wanted to point out that it's not in the mainline kernel and for some people that might be important. Thanks for your great work on ZoL! It made my life a lot easier :)
  - ryao 12y ago

jaytaylor 12y ago

BTRFS is still maturing. Kernel bugs involving BTRFS are still popping up pretty regularly.

deleted 12y ago

[deleted]

ryao 12y ago

A btrfs versus ZFS comparison probably deserves a blog post of its own, but I will try to address your question. I wrote the following on this topic last year: https://groups.google.com/d/msg/funtoo-dev/g9OY_vqVpCM/VTKF8Ef9ab4J https://groups.google.com/d/msg/funtoo-dev/g9OY_vqVpCM/VTKF8... However, significant time has passed and it requires some corrections to be current: 1. I have not heard of any recent data corruption issues in btrfs, although I have not looked into them lately. 2. btrfs now has experimental RAID 5/6 support, but is neither production ready nor as refined as ZFS' raidz. 3. I should have said "inline block-based data deduplication". You can (ab)use reflinks to achieve a file-level data deduplication in btrfs, but it is not quite the same. btrfs now has a bedup tool that makes using reflinks somewhat easier now: https://btrfs.wiki.kernel.org/index.php/Deduplication https://btrfs.wiki.kernel.org/index.php/Deduplication 4. btrfs now has some kind of incremental send/recv operation. However, it is not clear to me how it handles consistency issues from having "write-able snapshots": https://btrfs.wiki.kernel.org/index.php/Incremental_Backup https://btrfs.wiki.kernel.org/index.php/Incremental_Backup 5. Illumos' ZFS implementation is now able to store small files in the dnode, which is improves its efficiency when storing small files in a manner similar to btrfs' block suballocation. This feature will likely be in ZoL 0.6.4. Aside from those corrections, what I wrote in that mailing list email should still be relevant today. However, there are a few advantages that ZFS has over btrfs that I recall offhand that I do not see here there or in nisa's reply: 0. ZFS uses 256-bit checksums with algorithms that are still considered to be good today. btrfs uses checksum algorithms that are known to be weak. In specific, btrfs uses CRC32 on 32-bit processors and CRC64 on 64-bit processors. CRC32 is the same algorithm used by TCP/IP. Its deficiencies are well documented: http://noahdavids.org/self_published/CRC_and_checksum.html http://noahdavids.org/self_published/CRC_and_checksum.html I have not examined CRC64, but I am not particularly confident in it. btrfs should have room in its on-disk data structures that would allow it to implement 256-bit checksums in a future disk format extension, but until then, its checksum implementations are vastly inferior. 1. The ztest utility that I described in the blog allows ZFS developers to catch issues that would have otherwise gone into production and debug them from userland. No other filesystem has something like quite like it. 2. ZFSOnLinux is the only kernel filesystem driver that is kernel version-independent, so if you are unable to upgrade your kernel, you can still get fixes. The inability of people to always update their kernels is an issue Linus mentioned at LinuxCon North America 2014. 3. The CDDL gives the ZFSOnLinux a patent grant for the ZFS patent portfolio. This is something that btrfs does not have and will likely never have unless Oracle decides to provide one. Consequently, Oracle is the only company in the world that I know is able to ship products incorporating the btrfs source code without being at risk should btrfs infringe on one of the dozens if not hundreds of patents in the ZFS patent portfolio. A small subset of them can be accessed from the Jeff Bonwick Wikipedia page: https://en.wikipedia.org/wiki/Jeff_Bonwick https://en.wikipedia.org/wiki/Jeff_Bonwick

DiabloD3 12y ago

Working RAID 5 and 6, RAID "7" (Z3[1], triple parity, where Z/5 is single, and Z2/6 is dual), tiered RAID setups (such as JBOD'ed RAID-Z3s), zvols (as in, but not limited to, swap partitions in ZFS), more nuanced multi-controller and failover/spare setups, write-only journals (ZIL), second tier caching (L2ARC), LZ4 transparent compression, better designed snapshot and snapshot cloning support, more mature CLI tools, support on other OS's (with shared code bases for bonus points), configurable checksum algos, and a few other things that I'm forgetting at the moment. [1]: Seen that 90 drive Supermicro drive chassis[2]? Three 29-drive RAID-Z3s with 3 hot spares (which is shared across the three RAIDs) in a single storage pool (round robin-esqued), and you could plug that into a 2U with 16 small SSDs (like those Crucial M550 128GBs I use now) for the ZIL/L2ARC farming, and then have, well, near-infinite IO performance. Good luck trying to assemble that with BTRFS. [2]: A wet dream for CEPH users, too.

mdellabitta 12y ago
Re: [1], isn't that far too many drives in one RAIDZ? Referencing this: http://www.solarisinternals.com/wiki/index.php/ZFS_Best_Practices_Guide#RAIDZ_Configuration_Requirements_and_Recommendations http://www.solarisinternals.com/wiki/index.php/ZFS_Best_Prac...
- DiabloD3 12y ago
  Yes and no. The optimal number for RAIDZ/Z1/Z2, claimed by many benchmarks, is (power of two count of drives) plus (number of drives for parity)[1], with the upper bound being the likelihood of another drive failing before rebuild can complete. There is a tradeoff of, for example, using 3 RAIDZs each with a third of the drives in your pool, or 1 RAIDZ3 using all the drives in your pool. In the first case, a rebuild takes less time, but the chances of all data in that RAIDZ being lost is (I think) 4 times higher (but statistics is not my strong point; it may be higher). In the second case, a rebuild takes longer, the chances of another drive failing before rebuild is complete is higher, however, you require three failures before you have a problem; the array is still fully protected from another drive failure after two drive failures. The numbers in the example I gave, however, are not quite optimum for performance reasons (you'd be better off with 16 or 24), but most people don't like tons of hotspares (even though that case likely eats drives like old Sun thumpers would). All of that said, it depends entirely on how much you want data reliability. And, also, it was just an example. Not necessarily a good one. I was just illustrating that ZFS can handle that many drives with ease, but BTRFS would not be a good fit. [1]: There seems to be no difference between power of two and half way steps in real world performance when the number of drives are big enough, as in, 8, 12, and 16 will all perform similarly.