4 ms·

> Whoever seems to be singing the praises of ZFS on Linux hasn't put it through its paces in modern, multi-tenant container workloads. I ran a Hadoop Cluster w

by nisa 10y ago

> Whoever seems to be singing the praises of ZFS on Linux hasn't put it through its paces in modern, multi-tenant container workloads. I ran a Hadoop Cluster with it? Does that count? Your problem is probably the ARC and memory problems due to slow shrinking or stuff like that? There is some work or at least the intention to use the pagecache infrastructure for the ARC to make things more smooth. However at the moment it's still vmalloc afaik. You can reduce the ARC size and you'll be probably fine with your containers if they need a lot of memory. The SPL isn't so bad it's more or less wrappers. have fun with btrfs! It's a horrid mess! Looks like you never had the pleasure! btrfs is also a non starter on basically everything that goes beyond a single disk - even their RAID-1 implementation is strange, RAID5,6 are considered experimental and I could go on. ZFS for Ubuntu was and is a great idea!

sargun 10y ago
No, specifically multi-tenant, cgroups workloads. It works great for single-tenant workloads, esp. when you can tune the ARC to your workload. I know some folks who run it for large data workloads, and it works awesome for them. Unfortunately, our large container workload hasn't had the same luck. The issues I've hit are the following: https://github.com/zfsonlinux/zfs/issues/5814 https://github.com/zfsonlinux/zfs/issues/5814 https://github.com/zfsonlinux/zfs/issues/5535 https://github.com/zfsonlinux/zfs/issues/5535 Unfortunately, these problems stem from page faults occurring inside of VM that propagate to ZFS. If the fault occurs during cgroup memory pressure, ZFS / SPL may fail to allocate memory. ZFS will never get out of this case unless memory is freed up elsewhere in the hierarchy. The other issue we had was with ZFS integration. There are a few symbols around mounts that are exported GPL only, and they interfere with volume GC, and mounting, causing some issues with standard tools like Systemd, and others. For the most part, our container applications are stateless, or soft state, in this, we don't rely on RAID1, 5, 6, etc... but instead RAID0, or no RAID at all. If we detect a bad disk, we'd rather just evacuate the containers, and restore state later. I'd love to hear about your experience with BtrFS. What issues have you seen with it (single disk, or RAID0 -- I know RAID1, 5, and 6 are hokey at best)? How did the project handle your issues?
- nisa 10y ago
  Yeah. These ZFS issues look like they are deal breakers. In this use case btrfs could really be a good idea - afaik cgroup and kernel integration is better there. > I'd love to hear about your experience with BtrFS. What issues have you seen with it (single disk, or RAID0 -- I know RAID1, 5, and 6 are hokey at best)? How did the project handle your issues? Single disk, no RAID - lockups, broken filesystems (undeletable files), filesystem not mountable - however in recent kernels things are likely better (>4.8). Performance and defrag are issues on btrfs, scrub and balance take all io and there are some warts... might be good enough for your use case, through. RAID-1 is not really double read-throughput, not sure how RAID-0 is implemented. There is a mailinglist and and a bugzilla but the ZFS issue tracker was always more helpful.

by nisa 10y ago

sargun 10y ago

No, specifically multi-tenant, cgroups workloads. It works great for single-tenant workloads, esp. when you can tune the ARC to your workload. I know some folks who run it for large data workloads, and it works awesome for them. Unfortunately, our large container workload hasn't had the same luck. The issues I've hit are the following: https://github.com/zfsonlinux/zfs/issues/5814 https://github.com/zfsonlinux/zfs/issues/5814 https://github.com/zfsonlinux/zfs/issues/5535 https://github.com/zfsonlinux/zfs/issues/5535 Unfortunately, these problems stem from page faults occurring inside of VM that propagate to ZFS. If the fault occurs during cgroup memory pressure, ZFS / SPL may fail to allocate memory. ZFS will never get out of this case unless memory is freed up elsewhere in the hierarchy. The other issue we had was with ZFS integration. There are a few symbols around mounts that are exported GPL only, and they interfere with volume GC, and mounting, causing some issues with standard tools like Systemd, and others. For the most part, our container applications are stateless, or soft state, in this, we don't rely on RAID1, 5, 6, etc... but instead RAID0, or no RAID at all. If we detect a bad disk, we'd rather just evacuate the containers, and restore state later. I'd love to hear about your experience with BtrFS. What issues have you seen with it (single disk, or RAID0 -- I know RAID1, 5, and 6 are hokey at best)? How did the project handle your issues?

nisa 10y ago
Yeah. These ZFS issues look like they are deal breakers. In this use case btrfs could really be a good idea - afaik cgroup and kernel integration is better there. > I'd love to hear about your experience with BtrFS. What issues have you seen with it (single disk, or RAID0 -- I know RAID1, 5, and 6 are hokey at best)? How did the project handle your issues? Single disk, no RAID - lockups, broken filesystems (undeletable files), filesystem not mountable - however in recent kernels things are likely better (>4.8). Performance and defrag are issues on btrfs, scrub and balance take all io and there are some warts... might be good enough for your use case, through. RAID-1 is not really double read-throughput, not sure how RAID-0 is implemented. There is a mailinglist and and a bugzilla but the ZFS issue tracker was always more helpful.