4 ms·

I'm not the op but I'm running ZoL on a small cluster with 200 HDDs and 50 machines. I've also tested btrfs. Why is ZFS better (IMHO): + Tooling. The zpool an

by nisa 12y ago

I'm not the op but I'm running ZoL on a small cluster with 200 HDDs and 50 machines. I've also tested btrfs. Why is ZFS better (IMHO): + Tooling. The zpool and zfs commands are clear, easy to use and well documented. btrfs e.g. has no way to get a list of files with checksum problems. Also automounting and setting flexible mount points as well as the whole zfs get/set concept felt like a really concise way to configure all aspects easily. + Architecture. ZFS uses merkle hash trees with multiple copies spread over the disk (ditto blocks). This means that disk failures likely won't affect metadata and every possible disk problem is accounted for (except main memory). btrfs only uses crc32c and has 2 copies of the metadata tree on disk (but not spread over the disk afaik) or simple RAID1 in case of multiple disks. So in case of disk problems btrfs tends to corrupt faster. I've actually tested this and it's almost impossible to corrupt ZFS metadata even with hundreds of bad blocks. btrfs tends to switch faster to read-only mode with bad disks. + Compression: LZ4 is slightly better than LZO + L2ARC/ARC - easy to plug in a SSD cache. ARC is better than the pagecache for most workloads. + It's really fun to work with. A lot of good documentation. Dealing with btrfs felt very different... + RAIDZ(2-3) and copies=n for datasets. Both don't exist in btrfs (yet). Reasons against ZoL: - Memory. I've got out of memory errors because ARC memory is not freed fast enough. Also ZFS hogs a lot of memory. At least 750MB to 1GB kernel memory with several disks. btrfs needs far less. There is upcoming work to fix the ARC out of memory issues through.. it's on the roadmap from the developers. - Stability. This also applies to btrfs but it's not as rock-solid as ext4. Especially with broken disks and usage of advanced features you'll run eventually into bugs. But the "normal" operations where 100% stable for me with ZoL. - It's not native Linux. You have some Solaris Porting layer modules and integration into the Linux kernel is not as tight as you may like it to be. E.g. cgroups based io throttling was not possible, last time I looked. This may change in the future but it's work to integrate it and if e.g. the module does not not compile for a new kernel you are screwed. - btrfs will be supported and be a default choice in 1-2 years in likely all major Linux enterprise distributions. SLES already made it the default. For most workloads ZFS is likely not worth it. At least it's tough to sell against a default option. - rootfs on a ZoL volume is quite a lot of work and difficult to get working. But this is largely a distribution problem.

ryao 12y ago
Below are a few remarks: To expand on your LZ4 comment, LZ4 is 3 times faster than LZO at decompression: https://code.google.com/p/lz4/ https://code.google.com/p/lz4/ However, LZ4's real innovation is on incompressible data. I do not have a reference at the moment, but there have been benchmarks on the userland lz4 tool on incompressible data that show it processing 10GB/sec. This is because LZ4 uses a hash table that enables it to give up when trying to compress data far sooner than other lempel-ziv implementations do. As for stability, it is hard to for me to tell what you mean by that, but I just wrote a series of blog posts on the topic. If there are any outstanding issues not listed there, please file an issue on github: https://github.com/zfsonlinux/zfs/issues/new https://github.com/zfsonlinux/zfs/issues/new I also encourage you to file an issue requesting support for cgroup bsaed IO throttling. You are the first person to mention it. As for not compiling with new kernels, the 0.6.3 release supports Linux 2.6.26 to Linux 3.16 while HEAD adds support for Linux 3.17 release candidates. I briefly touched on the rootfs on ZFS issue in my blog post. I plan to add ZFS support to syslinux in the near future as per a discussion that I had with Peter Alvin at LinuxCon North America 2014. I expect this to resolve the issue for those willing to use syslinux as their bootloader.
- nisa 12y ago
  Wow! Thanks for your reply. I've experienced some minor issues with severe broken disks where the SPL layer discards the disk but the zpool is still online. It's not a major issue and I've yet to encounter the issue again I've already filed a related bug: https://github.com/zfsonlinux/zfs/issues/2508 https://github.com/zfsonlinux/zfs/issues/2508 but I'm not an expert. If I can gather more data I'll try to submit as much data and hints I can find. As for cgroups blkio - there is already a bug report: https://github.com/zfsonlinux/zfs/issues/1952 https://github.com/zfsonlinux/zfs/issues/1952 Sorry. It was not my intention to suggest that ZFS does not compile on new kernel versions. I've never had problems with that. I just wanted to point out that it's not in the mainline kernel and for some people that might be important. Thanks for your great work on ZoL! It made my life a lot easier :)
  - ryao 12y ago
    I am also a Linux kernel contributor, so running the latest kernels is important for me. This prompted me to switch from Nvidia graphics to Intel graphics a couple of months ago. ZoL's future kernel version support should reflect this. As for the issues you linked, the project will likely not be able to tackle them until next year, but being in the tracker means that they will receive attention.

by nisa 12y ago

ryao 12y ago

Below are a few remarks: To expand on your LZ4 comment, LZ4 is 3 times faster than LZO at decompression: https://code.google.com/p/lz4/ https://code.google.com/p/lz4/ However, LZ4's real innovation is on incompressible data. I do not have a reference at the moment, but there have been benchmarks on the userland lz4 tool on incompressible data that show it processing 10GB/sec. This is because LZ4 uses a hash table that enables it to give up when trying to compress data far sooner than other lempel-ziv implementations do. As for stability, it is hard to for me to tell what you mean by that, but I just wrote a series of blog posts on the topic. If there are any outstanding issues not listed there, please file an issue on github: https://github.com/zfsonlinux/zfs/issues/new https://github.com/zfsonlinux/zfs/issues/new I also encourage you to file an issue requesting support for cgroup bsaed IO throttling. You are the first person to mention it. As for not compiling with new kernels, the 0.6.3 release supports Linux 2.6.26 to Linux 3.16 while HEAD adds support for Linux 3.17 release candidates. I briefly touched on the rootfs on ZFS issue in my blog post. I plan to add ZFS support to syslinux in the near future as per a discussion that I had with Peter Alvin at LinuxCon North America 2014. I expect this to resolve the issue for those willing to use syslinux as their bootloader.

nisa 12y ago
Wow! Thanks for your reply. I've experienced some minor issues with severe broken disks where the SPL layer discards the disk but the zpool is still online. It's not a major issue and I've yet to encounter the issue again I've already filed a related bug: https://github.com/zfsonlinux/zfs/issues/2508 https://github.com/zfsonlinux/zfs/issues/2508 but I'm not an expert. If I can gather more data I'll try to submit as much data and hints I can find. As for cgroups blkio - there is already a bug report: https://github.com/zfsonlinux/zfs/issues/1952 https://github.com/zfsonlinux/zfs/issues/1952 Sorry. It was not my intention to suggest that ZFS does not compile on new kernel versions. I've never had problems with that. I just wanted to point out that it's not in the mainline kernel and for some people that might be important. Thanks for your great work on ZoL! It made my life a lot easier :)
- ryao 12y ago
  I am also a Linux kernel contributor, so running the latest kernels is important for me. This prompted me to switch from Nvidia graphics to Intel graphics a couple of months ago. ZoL's future kernel version support should reflect this. As for the issues you linked, the project will likely not be able to tackle them until next year, but being in the tracker means that they will receive attention.