4 ms·

I have messed with ZFS on Linux on Ubuntu and I have to say that I would not yet trust it in production. It's not as bullet proof as it needs to be and still un

by devoply 10y ago

I have messed with ZFS on Linux on Ubuntu and I have to say that I would not yet trust it in production. It's not as bullet proof as it needs to be and still under heavy dev. Not even at version 1.0 yet.

petre 10y ago
It's bullet proof on Solaris and FreeBSD.
- user5994461 10y ago
  Which doesn't say anything about its state on Linux.
  - FooBarWidget 10y ago
    You didn't get the hint. He's saying you should be using Solaris or FreeBSD instead of Linux.
  - ianai 10y ago
    The implementation might be lacking but the underlying FS should be more reliable. I'd still argue that ZFS should be deployed on FreeBSD or Solaris. There are plenty of ways to fire up a Linux environment from there.
iso-8859-1 10y ago
Do you trust btrfs? Suse has been having it as the default since 2014...
- deleted 10y ago
  [deleted]
- jlgaddis 10y ago
  > The parity RAID code has multiple serious data-loss bugs in it. It should not be used for anything other than testing purposes. [0] [0]: https://btrfs.wiki.kernel.org/index.php/RAID56 https://btrfs.wiki.kernel.org/index.php/RAID56
  - penagwin 10y ago
    Important to note that is only referring to Raid 5 and 6
phil21 10y ago
Depends on what you're worried about. Operationally speaking I agree, it's not plug and play. But it's at a point where it safely stores your data correctly. Perhaps some init scripts fail on boot to import your pool/etc. but the data is there. We do run it production, but we also have in-house tooling built around it.
dmm 10y ago
Do you have any specific reasons not to trust ZoL? ZFS-on-Linux devs say it's ready for production[1]. Lawrence Livermore laboratory stores petabytes of data using ZoL[2]. If we're sharing anecdotes, ZoL has served me fantastically for several years. [1] https://clusterhq.com/2014/09/11/state-zfs-on-linux/ https://clusterhq.com/2014/09/11/state-zfs-on-linux/ [2] http://computation.llnl.gov/newsroom/livermores-zfs-linux-port-hit-it-industry http://computation.llnl.gov/newsroom/livermores-zfs-linux-po...
- user5994461 10y ago
  I thought "ZoL" was a pun with ZFS and LOL to tell how not ready it is for production ^^
  - ryao 10y ago
    ZoL is an acronym for ZFSonLinux.
- devoply 10y ago
  My reasons are as follows: 1) Seen users complaining about data loss on issues on github. 2) Had the init script fail on upgrade and had to fix it by hand when upgrading Ubuntu. Probably a one time issue. Need a bit more reliability from a file system.
- otterley 10y ago
  We have encountered a reproducible panic and deadlocks when a containerized process gets terminated by the kernel for exceeding its memory limit: https://github.com/zfsonlinux/zfs/issues/5535 https://github.com/zfsonlinux/zfs/issues/5535 We're strongly considering using something else until this gets addressed. The problem is, we don't know what, because every other CoW implementation also has issues. * dm-thinp: Slow, wastes disk space * OverlayFS: No SELinux support * aufs: Not in mainline or CentOS kernel; rename(2) not implemented correctly; slow writes
  - brendangregg 10y ago
    The issue you link to was opened a day ago. If that were me, I'd see how quickly it was fixed before strongly considering something else.
lscotte 10y ago
It's maturing rapidly and has proven to be very stable so far. We're not using it by default everywhere, at least not yet, and building out an AMI that uses ZFS for the rootfs is still a bit of a research project - but we have been using it to do RAID0 striping of ephemeral drives for a year or two on a number of workloads.
jlgaddis 10y ago
My newly built (Ubuntu 16.04 LTS) workstation is using ZFS exclusively. I'm keeping my fingers crossed.
- shiftoutbox 10y ago
  My workstation uses FreeBSD 11 with zfs on root and it just works . I don't have to worry about the filesystem.
deleted 10y ago
[deleted]
tnorgaard 10y ago
We have been running ZFS on Linux in production since April 2015 on over 1500 instances in AWS EC2 with Ubuntu 14.04 and 16.04. Only one kernel panic observed so far, on a Jenkins/CI instance, but that was due to Jenkins doing magic on ZFS mounts, believing it was a Solaris ZFS mount. In our opinion, when we made the switch, it was much more important to trust the integrity of the data, than any possible kernel panic.
brendangregg 10y ago
We've actually been running it in production at Netflix for a few microservices for over a year (as Scott said, for a few workloads, but a long way from everywhere). I don't think we've made any kind of announcement, but "ZFS" has shown up in a number of Netflix presentations and slide decks: eg, for Titus (container management). ZFS has worked well on Linux for us. I keep meaning to blog about it, but there's been so many things to share (BPF has kept me more busy). Glad Scott found the time to share the root ZFS stuff.
- voncopec 10y ago
  If I had to choose between a filesystem with silent and/or visible data corruption up to pretty much eating itself and having to restore an entire server, versus a filesystem for which you can trust but could have a kernel deadlock/panic..I would choose the latter, and in-fact did. I have seen a few servers with ext4/mdraid over the last five years have serious corruption but have had to reset a ZoL server maybe twice.
  - ianhowson 10y ago
    Story time. I transitioned an md RAID1 from spinning disks to SSDs last week. After I removed the last spinning disk, one of the SSDs started returning garbage. 1/3 reads are returning garbage and ext4 freaks out, of course. It's too late and the array is shot. I restore from backup. This would have been a non-event with ZFS. I've got a few production ZoL arrays running and the only problems I've had have been around memory consumption and responsiveness under load. Data integrity has been perfect.
    - voncopec 10y ago
      I've seen the same type of thing with respect to memory and load.
sigstoat 10y ago
i've been using zfs on ubuntu since ~2010 for a small set of machines, reading/writing 24/7 with different loads. it's worked great through quite a few drive replacements, and various other hardware failures. i'm perfectly willing to believe there may be some rare situations where zfs on linux will cause you a problem. but i bet they're rare enough it'll have saved you a few times before it bites you.
jeduardo 10y ago
Well, we (and by this I mean myself and my fantastic team) have been running it since 2015 as the main filesystem for a double-digit number of KVM hosts running a triple-digit number of virtual machines executing an interesting mix of workloads, ranging from lightweight (file servers for light sharing, web application hosts) to heavy I/O bound ones (databases, build farms) with fantastic results so far. All this on Debian Stable. The setup process was a bit painful given some interesting delays when using some HW storage controllers that caused udev to not make some HDD devices available under /dev before the ZFS scripts kicked in and we have been bitten a couple times by changes (or bugs) in the boot scripts, however the gains provided by ZFS in terms of data integrity, backup, and virtual machine provisioning workflow were definitely worth it.

by devoply 10y ago

petre 10y ago

It's bullet proof on Solaris and FreeBSD.

user5994461 10y ago
Which doesn't say anything about its state on Linux.
- FooBarWidget 10y ago
  You didn't get the hint. He's saying you should be using Solaris or FreeBSD instead of Linux.
- ianai 10y ago
  The implementation might be lacking but the underlying FS should be more reliable. I'd still argue that ZFS should be deployed on FreeBSD or Solaris. There are plenty of ways to fire up a Linux environment from there.

iso-8859-1 10y ago

Do you trust btrfs? Suse has been having it as the default since 2014...

deleted 10y ago
[deleted]
jlgaddis 10y ago
> The parity RAID code has multiple serious data-loss bugs in it. It should not be used for anything other than testing purposes. [0] [0]: https://btrfs.wiki.kernel.org/index.php/RAID56 https://btrfs.wiki.kernel.org/index.php/RAID56
- penagwin 10y ago
  Important to note that is only referring to Raid 5 and 6

phil21 10y ago

Depends on what you're worried about. Operationally speaking I agree, it's not plug and play. But it's at a point where it safely stores your data correctly. Perhaps some init scripts fail on boot to import your pool/etc. but the data is there. We do run it production, but we also have in-house tooling built around it.

dmm 10y ago

Do you have any specific reasons not to trust ZoL? ZFS-on-Linux devs say it's ready for production[1]. Lawrence Livermore laboratory stores petabytes of data using ZoL[2]. If we're sharing anecdotes, ZoL has served me fantastically for several years. [1] https://clusterhq.com/2014/09/11/state-zfs-on-linux/ https://clusterhq.com/2014/09/11/state-zfs-on-linux/ [2] http://computation.llnl.gov/newsroom/livermores-zfs-linux-port-hit-it-industry http://computation.llnl.gov/newsroom/livermores-zfs-linux-po...

user5994461 10y ago
I thought "ZoL" was a pun with ZFS and LOL to tell how not ready it is for production ^^
- ryao 10y ago
  ZoL is an acronym for ZFSonLinux.
devoply 10y ago
My reasons are as follows: 1) Seen users complaining about data loss on issues on github. 2) Had the init script fail on upgrade and had to fix it by hand when upgrading Ubuntu. Probably a one time issue. Need a bit more reliability from a file system.
otterley 10y ago
We have encountered a reproducible panic and deadlocks when a containerized process gets terminated by the kernel for exceeding its memory limit: https://github.com/zfsonlinux/zfs/issues/5535 https://github.com/zfsonlinux/zfs/issues/5535 We're strongly considering using something else until this gets addressed. The problem is, we don't know what, because every other CoW implementation also has issues. * dm-thinp: Slow, wastes disk space * OverlayFS: No SELinux support * aufs: Not in mainline or CentOS kernel; rename(2) not implemented correctly; slow writes
- brendangregg 10y ago
  The issue you link to was opened a day ago. If that were me, I'd see how quickly it was fixed before strongly considering something else.

lscotte 10y ago

It's maturing rapidly and has proven to be very stable so far. We're not using it by default everywhere, at least not yet, and building out an AMI that uses ZFS for the rootfs is still a bit of a research project - but we have been using it to do RAID0 striping of ephemeral drives for a year or two on a number of workloads.

jlgaddis 10y ago

My newly built (Ubuntu 16.04 LTS) workstation is using ZFS exclusively. I'm keeping my fingers crossed.

shiftoutbox 10y ago
My workstation uses FreeBSD 11 with zfs on root and it just works . I don't have to worry about the filesystem.

deleted 10y ago

[deleted]

tnorgaard 10y ago

We have been running ZFS on Linux in production since April 2015 on over 1500 instances in AWS EC2 with Ubuntu 14.04 and 16.04. Only one kernel panic observed so far, on a Jenkins/CI instance, but that was due to Jenkins doing magic on ZFS mounts, believing it was a Solaris ZFS mount. In our opinion, when we made the switch, it was much more important to trust the integrity of the data, than any possible kernel panic.

brendangregg 10y ago

We've actually been running it in production at Netflix for a few microservices for over a year (as Scott said, for a few workloads, but a long way from everywhere). I don't think we've made any kind of announcement, but "ZFS" has shown up in a number of Netflix presentations and slide decks: eg, for Titus (container management). ZFS has worked well on Linux for us. I keep meaning to blog about it, but there's been so many things to share (BPF has kept me more busy). Glad Scott found the time to share the root ZFS stuff.

voncopec 10y ago
If I had to choose between a filesystem with silent and/or visible data corruption up to pretty much eating itself and having to restore an entire server, versus a filesystem for which you can trust but could have a kernel deadlock/panic..I would choose the latter, and in-fact did. I have seen a few servers with ext4/mdraid over the last five years have serious corruption but have had to reset a ZoL server maybe twice.
- ianhowson 10y ago
  Story time. I transitioned an md RAID1 from spinning disks to SSDs last week. After I removed the last spinning disk, one of the SSDs started returning garbage. 1/3 reads are returning garbage and ext4 freaks out, of course. It's too late and the array is shot. I restore from backup. This would have been a non-event with ZFS. I've got a few production ZoL arrays running and the only problems I've had have been around memory consumption and responsiveness under load. Data integrity has been perfect.
  - voncopec 10y ago
    I've seen the same type of thing with respect to memory and load.

sigstoat 10y ago

i've been using zfs on ubuntu since ~2010 for a small set of machines, reading/writing 24/7 with different loads. it's worked great through quite a few drive replacements, and various other hardware failures. i'm perfectly willing to believe there may be some rare situations where zfs on linux will cause you a problem. but i bet they're rare enough it'll have saved you a few times before it bites you.

jeduardo 10y ago

Well, we (and by this I mean myself and my fantastic team) have been running it since 2015 as the main filesystem for a double-digit number of KVM hosts running a triple-digit number of virtual machines executing an interesting mix of workloads, ranging from lightweight (file servers for light sharing, web application hosts) to heavy I/O bound ones (databases, build farms) with fantastic results so far. All this on Debian Stable. The setup process was a bit painful given some interesting delays when using some HW storage controllers that caused udev to not make some HDD devices available under /dev before the ZFS scripts kicked in and we have been bitten a couple times by changes (or bugs) in the boot scripts, however the gains provided by ZFS in terms of data integrity, backup, and virtual machine provisioning workflow were definitely worth it.