5 ms·

Flavour: Ubuntu Server Headline: Dump ZFS on Linux Affiliation: Containers @ Netflix Description: ZFS On Linux is poorly integrated mess through the SPL. Mem

by sargun 10y ago

Flavour: Ubuntu Server Headline: Dump ZFS on Linux Affiliation: Containers @ Netflix Description: ZFS On Linux is poorly integrated mess through the SPL. Memory management is an active detractor from performance, stability, and operations. It's unlikely that it's ever going to be "native" on Linux. Even more unlikely is its integration into upstream. Unfortunately, ZoL, unlike other out of tree additions Canonical has added, such as proprietary drivers and codecs, ZoL has real alternatives, like BtrFS, and BCacheFS. I think it would make more sense to throw your weight behind these projects where there will be long term benefit to the community as opposed to short term benefit to y'all. Whoever seems to be singing the praises of ZFS on Linux hasn't put it through its paces in modern, multi-tenant container workloads. It requires active awareness of its existence unlike ZFS, and EXT4. To me, this is a fundamental regression. Do not fall privy to the sunk cost fallacy, instead continue to actively weigh your choices, and as soon as the opportunity cost for !ZoL or !SPL becomes reasonable, jump.

nisa 10y ago
> Whoever seems to be singing the praises of ZFS on Linux hasn't put it through its paces in modern, multi-tenant container workloads. I ran a Hadoop Cluster with it? Does that count? Your problem is probably the ARC and memory problems due to slow shrinking or stuff like that? There is some work or at least the intention to use the pagecache infrastructure for the ARC to make things more smooth. However at the moment it's still vmalloc afaik. You can reduce the ARC size and you'll be probably fine with your containers if they need a lot of memory. The SPL isn't so bad it's more or less wrappers. have fun with btrfs! It's a horrid mess! Looks like you never had the pleasure! btrfs is also a non starter on basically everything that goes beyond a single disk - even their RAID-1 implementation is strange, RAID5,6 are considered experimental and I could go on. ZFS for Ubuntu was and is a great idea!
- sargun 10y ago
  No, specifically multi-tenant, cgroups workloads. It works great for single-tenant workloads, esp. when you can tune the ARC to your workload. I know some folks who run it for large data workloads, and it works awesome for them. Unfortunately, our large container workload hasn't had the same luck. The issues I've hit are the following: https://github.com/zfsonlinux/zfs/issues/5814 https://github.com/zfsonlinux/zfs/issues/5814 https://github.com/zfsonlinux/zfs/issues/5535 https://github.com/zfsonlinux/zfs/issues/5535 Unfortunately, these problems stem from page faults occurring inside of VM that propagate to ZFS. If the fault occurs during cgroup memory pressure, ZFS / SPL may fail to allocate memory. ZFS will never get out of this case unless memory is freed up elsewhere in the hierarchy. The other issue we had was with ZFS integration. There are a few symbols around mounts that are exported GPL only, and they interfere with volume GC, and mounting, causing some issues with standard tools like Systemd, and others. For the most part, our container applications are stateless, or soft state, in this, we don't rely on RAID1, 5, 6, etc... but instead RAID0, or no RAID at all. If we detect a bad disk, we'd rather just evacuate the containers, and restore state later. I'd love to hear about your experience with BtrFS. What issues have you seen with it (single disk, or RAID0 -- I know RAID1, 5, and 6 are hokey at best)? How did the project handle your issues?
  - nisa 10y ago
    Yeah. These ZFS issues look like they are deal breakers. In this use case btrfs could really be a good idea - afaik cgroup and kernel integration is better there. > I'd love to hear about your experience with BtrFS. What issues have you seen with it (single disk, or RAID0 -- I know RAID1, 5, and 6 are hokey at best)? How did the project handle your issues? Single disk, no RAID - lockups, broken filesystems (undeletable files), filesystem not mountable - however in recent kernels things are likely better (>4.8). Performance and defrag are issues on btrfs, scrub and balance take all io and there are some warts... might be good enough for your use case, through. RAID-1 is not really double read-throughput, not sure how RAID-0 is implemented. There is a mailinglist and and a bugzilla but the ZFS issue tracker was always more helpful.
dustinkirkland 10y ago
Hmm, so I happen to really like ZFS under my LXD containers on Ubuntu. It has always worked very well, very fast, with instant startup times and snapshots. However, ZFS is in no way required to run containers on Ubuntu. You can use any filesystem you like under your Docker and LXD containers. So I'm not sure I get your request "Dump ZFS on Linux"... There are plenty of people who love ZFS on Linux, and appreciate Ubuntu for it. The Ubuntu platform is about choice, and you have the choice of ZFS or many other filesystems under your containers at Netflix...
sargun 10y ago
PS: None of these views represent the views of my employer. These are my personal views on the long-term view of the ecosystem.

by sargun 10y ago

nisa 10y ago

> Whoever seems to be singing the praises of ZFS on Linux hasn't put it through its paces in modern, multi-tenant container workloads. I ran a Hadoop Cluster with it? Does that count? Your problem is probably the ARC and memory problems due to slow shrinking or stuff like that? There is some work or at least the intention to use the pagecache infrastructure for the ARC to make things more smooth. However at the moment it's still vmalloc afaik. You can reduce the ARC size and you'll be probably fine with your containers if they need a lot of memory. The SPL isn't so bad it's more or less wrappers. have fun with btrfs! It's a horrid mess! Looks like you never had the pleasure! btrfs is also a non starter on basically everything that goes beyond a single disk - even their RAID-1 implementation is strange, RAID5,6 are considered experimental and I could go on. ZFS for Ubuntu was and is a great idea!

sargun 10y ago
No, specifically multi-tenant, cgroups workloads. It works great for single-tenant workloads, esp. when you can tune the ARC to your workload. I know some folks who run it for large data workloads, and it works awesome for them. Unfortunately, our large container workload hasn't had the same luck. The issues I've hit are the following: https://github.com/zfsonlinux/zfs/issues/5814 https://github.com/zfsonlinux/zfs/issues/5814 https://github.com/zfsonlinux/zfs/issues/5535 https://github.com/zfsonlinux/zfs/issues/5535 Unfortunately, these problems stem from page faults occurring inside of VM that propagate to ZFS. If the fault occurs during cgroup memory pressure, ZFS / SPL may fail to allocate memory. ZFS will never get out of this case unless memory is freed up elsewhere in the hierarchy. The other issue we had was with ZFS integration. There are a few symbols around mounts that are exported GPL only, and they interfere with volume GC, and mounting, causing some issues with standard tools like Systemd, and others. For the most part, our container applications are stateless, or soft state, in this, we don't rely on RAID1, 5, 6, etc... but instead RAID0, or no RAID at all. If we detect a bad disk, we'd rather just evacuate the containers, and restore state later. I'd love to hear about your experience with BtrFS. What issues have you seen with it (single disk, or RAID0 -- I know RAID1, 5, and 6 are hokey at best)? How did the project handle your issues?
- nisa 10y ago
  Yeah. These ZFS issues look like they are deal breakers. In this use case btrfs could really be a good idea - afaik cgroup and kernel integration is better there. > I'd love to hear about your experience with BtrFS. What issues have you seen with it (single disk, or RAID0 -- I know RAID1, 5, and 6 are hokey at best)? How did the project handle your issues? Single disk, no RAID - lockups, broken filesystems (undeletable files), filesystem not mountable - however in recent kernels things are likely better (>4.8). Performance and defrag are issues on btrfs, scrub and balance take all io and there are some warts... might be good enough for your use case, through. RAID-1 is not really double read-throughput, not sure how RAID-0 is implemented. There is a mailinglist and and a bugzilla but the ZFS issue tracker was always more helpful.

dustinkirkland 10y ago

Hmm, so I happen to really like ZFS under my LXD containers on Ubuntu. It has always worked very well, very fast, with instant startup times and snapshots. However, ZFS is in no way required to run containers on Ubuntu. You can use any filesystem you like under your Docker and LXD containers. So I'm not sure I get your request "Dump ZFS on Linux"... There are plenty of people who love ZFS on Linux, and appreciate Ubuntu for it. The Ubuntu platform is about choice, and you have the choice of ZFS or many other filesystems under your containers at Netflix...

sargun 10y ago

PS: None of these views represent the views of my employer. These are my personal views on the long-term view of the ecosystem.