9 ms·

Anyone here using ZFS in production these days? If so what OS and implementation? What have been your experiences or gotchas you experienced?

noaheverett 3y ago
Running in production for about 3 years with Ubuntu 20.04 / zfs 0.8.3. ZFS is being used as the datastore for a cluster of LXD/LXC instances over multiple physical hosts. I have the OS setup on its own dedicated drive and ZFS striped/cloned over 4 NVMe drives. No gotchas / issues, works well, easy to setup. I am looking forward to the Direct IO speed improvements for NVMe drives with https://github.com/openzfs/zfs/pull/10018 https://github.com/openzfs/zfs/pull/10018 edit: one thing I forgot to mention is, when creating your pool make sure to import your drives by ID (zpool import -d /dev/disk/by-id/ <poolname>) instead of name in case name assignments change somehow [1] [1] https://superuser.com/questions/1732532/zfs-disk-drive-letter-changed-how-to-reimport-by-id https://superuser.com/questions/1732532/zfs-disk-drive-lette...
- throw0101c 3y ago
  > I am looking forward to the Direct IO speed improvements for NVMe drives with https://github.com/openzfs/zfs/pull/10018 https://github.com/openzfs/zfs/pull/10018 See also "Scaling ZFS for NVMe" by Allan Jude at EuroBSDcon 2022: * https://www.youtube.com/watch?v=v8sl8gj9UnA https://www.youtube.com/watch?v=v8sl8gj9UnA
  - noaheverett 3y ago
    Sweet, I appreciate the link!
- doctor_eval 3y ago
  Sorry if this is a newbie question - but what are you using to share the ZFS file system between physical hosts? I’ve been out of this particular game for a long time.
  - noaheverett 3y ago
    In our case each node's ZFS store operates independently, it's not shared. We spin up "VMs" (LXC based) on each node and they talk to each via our private network. Hope that helps!
drewg123 3y ago
We use ZFS in production for the non-content filesystems on a large, ever increasing, percentage of our Netflix Open Connect CDN nodes, replacing geom's gmirror. We had one gotcha (caught on a very limited set of canaries) where a buggy bootloader crashed part-way through boot, leaving the ZFS "bootonce" stuff in a funky state requiring manual recovery (the nodes with gmirror were fine, and fell back to the old image without fuss). This has since been fixed. Note that we do not use ZFS for content, since it is incompatible with efficient use of sendfile (both because there is no async handler for ZFS, so no async sendfile, and because the ARC is not integrated with the page cache, so content would require an extra memory copy to be served).
- bakul 3y ago
  Is integrating ARC with the page cache a lost cause? If not, may be Netflix can fund it!
  - postmodest 3y ago
    I would expect page cache to be one of those platform-dependent things that prevent OpenZFS from doing that. Especially on Linux, and especially because AFAIK the Linux version has the most eyes on it.
    - rincebrain 3y ago
      My understanding, not having dug into it, is that it's possible but just work nobody has done yet, though I'm not sure what the relevant interfaces are in the Linux kernel. One thing that makes the interfaces in Linux much messier than the FreeBSD ones is that a lot of the core functionality you might like to leverage in Linux (workqueues, basically anything more complicated than just calling kmalloc, and of course any SIMD save/restore state, to name three examples I've stumbled over recently) are marked EXPORT_SYMBOL_GPL or just entirely not exported in newer releases, so you get to reimplement the wheel for those, whereas on FreeBSD it's trivial to just use their implementations of such things and shim them to the Solaris-ish interfaces the non-platform-specific code expects. So that makes the Linux-specific code a lot heavier, because upstream is actively hostile.
nightfly 3y ago
Yes, Ubuntu 20.04 and 22.04. But we've been running ZFS in some form or other for 10+ years. ACL support not as good/easy to use as Solaris/FreeBSD. Not having weird pathological performance issues with kernel memory allocation like we had with FreeBSD though. Sometimes we have issues with automatic pool import on boot, so that's something to be careful with. The tooling is great though, and we've never had catastrophic failure that was due to ZFS, only due to failing hardware.
trws 3y ago
I’m not a filesystem admin, but we at LLNL use OpenZFS as the storage layer for all of our Lustre file systems in production, including using raid-z for resilience in each pool (order 100 disks each), and have for most of a decade. That combined with improvements in Lustre have taken the rate of data loss or need to clear large scale shared file systems down to nearly zero. There’s a reason we spend as many engineer hours as we do maintaining it, it’s worth it. LLNL openzfs project: https://computing.llnl.gov/projects/openzfs https://computing.llnl.gov/projects/openzfs Old presentation from intel with info on what was one of our bigger deployments in 2016 (~50pb): https://www.intel.com/content/dam/www/public/us/en/documents/presentation/benefits-zfs-for-lustre-deployments.pdf https://www.intel.com/content/dam/www/public/us/en/documents...
- muxator 3y ago
  If I'm not mistaken the linux port of ZFS that later became OpenZFS started at LLNL and was a port from FreeBSD (it may have been release ~9). I believe it was called ZFS On Linux or something like that. Nice how things have evolved: from FreeBSD to linux and back. In my mind this has always been a very inspiring example of a public institution working for the public good.
  - rincebrain 3y ago
    FreeBSD had its own ZFS port. ZoL, if my ancient memory serves, was at LLNL, not based on the FreeBSD port (if you go _very_ far back in the commit history you can see Brian rebasing against OpenSolaris revisions), but like 2 or 3 different orgs originally announced Linux ports at the same time and then all pooled together, since originally only one of the three was going to have a POSIX layer (the other two didn't need a working POSIX filesystem layer). (I'm not actually sure how much came of this collaboration, I just remember being very amused when within the span of a week or two, three different orgs announced ports, looked at each other, and went "...wait.") Then for a while people developed on either the FreeBSD port, the illumos fork called OpenZFS, or the Linux port, but because (among other reasons) a bunch of development kept happening on the Linux port, it became the defacto upstream and got renamed "OpenZFS", and then FreeBSD more or less got a fresh port from the OpenZFS codebase that is now what it's based on. The macOS port got a fresh sync against that codebase recently and is slowly trying to merge in, and then from there, ???
SkyMarshal 3y ago
Not in production, but using ZoL on my personal workstations. https://zfsonlinux.org/ https://zfsonlinux.org/ Some discussion: https://www.reddit.com/r/NixOS/comments/ops0n0/big_shoutout_to_the_nixos_team_for_the_excellent/ https://www.reddit.com/r/NixOS/comments/ops0n0/big_shoutout_...
unixhero 3y ago
Yes. Using latest ZFS ZFS On Linux distrib on Debian. Using Proxmox. Never had and problems ever, ever.
shrubble 3y ago
The gotcha on Proxmox is that you can't do swapfiles on ZFS, so if your swap isn't made big enough when installing and you format everything as ZFS you have to live with it or do odd workarounds.
- tlamponi 3y ago
  That's not a gotcha on Proxmox projects but a gotcha everywhere, SWAP on ZFS needs a lot of special care to avoid that it needs to allocate memory during swapping memory out (e.g., if low on memory), causing hard-freezes. It can be made somewhat[0] stable with using no (or a simple compression like ZLE), avoiding log devices and caches it can work, but it's way simpler and guaranteed stable to just use a separate partition. [0] even setting the myriad of options there still exist reports about hangs, like https://github.com/openzfs/zfs/issues/7734#issuecomment-416706195 https://github.com/openzfs/zfs/issues/7734#issuecomment-4167...
- gorkish 3y ago
  1. You shouldn't swap on ZFS, but really 2. You shouldn't swap on a filesystem, but really 3. You shouldn't swap at all. On a VM host you have a bit of leeway for memory overcommit, but ideally if you have a box that frequently has to rely on swap, something is bad wrong. I don't think it's really fair to criticize a worst-choice configuration for being poorly supported here. There are simply too many opportunities for catch-22/deadlock scenarios in the interaction of the fs and allocator during OOM...
gigatexal 3y ago
Any of the enterprise customers of klara Systems are likely ZFS production folks. https://klarasystems.com/?amp https://klarasystems.com/?amp
crest 3y ago
I use it as my default file system on FreeBSD. It was rough in FreeBSD 7.x (around 2009), but starting with FreeBSD 8.x it has been rock solid to this day. The only gotcha (which the documentation warns about) has been that automatic block level deduplication is only useful in a few special applications and has a large main memory overhead unless you can accept terrible performance for normal operations (e.g. a bandwidth limited offsite backup).
yjftsjthsd-h 3y ago
Sure; we get good mileage out of compression and snapshots (well, mostly send-recv for moving data around rather than snapshots in-place). I think the only problems have been very specific to our install process (non-standard kernel in the live environment; if we used the normal distro install process it would be fine).
szundi 3y ago
Yes and it is awesome, no issues.
Drybones 3y ago
We use ZFS on every server we deploy We typically use Proxmox. It’s a convenient node host setup and usually has a very up to date zfs and it’s stable I just wouldn’t use the Proxmox web ui for zfs configuration. It doesn’t have up to date options. Always configure zfs on the cli
- tlamponi 3y ago
  > I just wouldn’t use the Proxmox web ui for zfs configuration. It doesn’t have up to date options. What options are missing for you? Would be great if you could open an enhancement request over at https://bugzilla.proxmox.com/ https://bugzilla.proxmox.com/ for tracking this (no promises on (immediate) implementation though).
justinclift 3y ago
TrueNAS (www.truenas.com) uses ZFS for the storage layer across it's product range (storage software). They have both FreeBSD and Linux based stuff, targeting different use cases.
- explorer83 3y ago
  I use TrueNAS (FreeBSD) version. ZFS makes keeping data snapshots and external backups a fairly easy process. It really cut down on server data management time. The only issue I found with it is there was a now long patched SMB bug a few years ago in the TrueNAS software that caused my storage pool to become corrupted after a large transfer. And there was no way for me to recover the data. It was all lost despite the disks still spinning and presumably still having most and the 1 and 0s recoverable.
  - justinclift 3y ago
    Hopefully you had backups?
DvdGiessen 3y ago
In production on SmartOS (illumos) servers running applications and VM's, on TrueNAS and plain FreeBSD for various storage and backups, and on a few Linux-based workstations. Using mirrors and raidz2 depending on the needs of the machines. We've successfully survived numerous disk failures (a broken batch of HDD's giving all kinds of small read errors, an SSD that completely failed and disappeared, etc), and were in most cases able to replace them without a second of downtime (would have been all cases if not for disks placed in hard-to-reach places, now only a few minutes downtime to physically swap the disk). Snapshots work perfectly as well. Systems are set up to automatically make snapshots using [1], on boot, on a timer, and right before potentially dangerous operations such as package manager commands as well. I've rolled back after botched OS updates without problems; after a reboot the machine was back in it's old state. Also rolled back a live system a few times after a broken package update, restoring the filesystem state without any issues. Easily accessing old versions of a file is an added bonus which has been helpful a few times. Send/receive is ideal for backups. We are able to send snapshots between machines, even across different OSes, without issues. We've also moved entire pools from one OS to another without problems. Knowing we have automatic snapshots and external backups configured also allows me to be very liberal with giving root access to inexperienced people to various (non-critical) machines, knowing that if anything breaks it will always be easy to roll back, and encouraging them to learn by experimenting a bit, to the point where we can even diff between snapshots to inspect what changed and learn from that. Biggest gotchas so far have been on my personal Arch Linux setup, where the out-of-tree nature of ZFS has caused some issues like a incompatible kernel being installed, the ZFS module failing to compile, and my workstation subsequently being unable to boot. But even that was solved by my entire system running on ZFS: a single rollback from my bootloader [2] and all was back the way it was before. Having good tooling set up definitely helped a lot. My monkey brain has the tendency to think "surely I got it right this time, so no need to make a snapshot before trying out X!", especially when experimenting on my own workstation. Automating snapshots using a systemd timer and hooks added to my package manager saved me a number of times. [1]: https://github.com/psy0rz/zfs_autobackup https://github.com/psy0rz/zfs_autobackup [2]: https://zfsbootmenu.org/ https://zfsbootmenu.org/
- csdvrx 3y ago
  > Systems are set up to automatically make snapshots I do that with sqlite to keep a selection of snapshots from the last hours, days etc. https://github.com/csdvrx/zfs-autosnapshot https://github.com/csdvrx/zfs-autosnapshot
enneff 3y ago
I use ZFS on Debian for my home file server. The setup is just a tiny NUC with a couple of large USB hard drives, mirrored with ZFS. I’ve had drives fail and painlessly replaced and resilvered them. This is easily the most hassle free file storage setup I’ve owned; been going strong over 10 years now with little to no maintenance.
- deleted 3y ago
  [deleted]
- rsaxvc 3y ago
  Pretty much same setup here except I used a Lacie atom NAS, added a USB stick to the motherboard for booting.
quags 3y ago
I have been using zfs for years from ubuntu 18. Easy snapshots, monitoring, choices for raid levels, and ability to very easily copy a dataset remotely with resume or incremental support is awesome. I mainly use it for kvm systems each with their own dataset. Coming from mdadm + lvm from my previous set up is night and day for doing snapshots and backups. I do not use zfs on root for ubuntu instead I do a raid1 software set up for the os and then a zfs set up on other disks - zfs on root was the only gotcha. For FreeBSD zfs on root works fine.
mattjaynes 3y ago
ZFS on Linux has improved a lot in the last few years. We (prematurely) moved to using it in production for our MySQL data about 5 years ago and initially it was a nightmare due to unexplained stalling which would hang MySQL for 15-30 minutes at random times. I'm sure it shortened my life a few years trying to figure out what was wrong when everything was on fire. Fortunately, they have resolved those issues in the subsequent releases and it's been much more pleasant after that.
benlivengood 3y ago
"production" at home on Debian 11, previously on FreeBSD 10-13. The weirdest gotcha has been related to sending encrypted raw snapshots to remote machines[0],[1]. These have been the first instabilities I had with ZFS in roughly 15 years around the filesystem since switching to native encryption this year. Native encryption seems to be barely stable for production use; no actual data corruption but automatic synchronization (I use znapzend) was breaking frequently. Recent kernel updates fixed my problem although some of the bug reports are still open. I only moved on from FreeBSD because of more familiarity with Linux. A slightly annoying property of snapshots and clones is the inability to fully re-root a tree of snapshots, e.g. permanently split a clone from its original source and allow first-class send/receive from that clone. The snapshot which originated the clone needs to stick around forever[2]. This prevents a typical virtual machine imagine process of keeping a base image up to date over time that VMs can be cloned from when desired and eventually removing the storage used by the original base image after e.g. several OS upgrades. I don't have any big performance requirements and most file storage is throughput based on spinning disks which can easily saturate the gigabit network. I also use ZFS on my laptop's SSD under Ubuntu with about 1GB/s performance and no shortage of IOPS and the ability to send snapshots off to the backup system which is pretty nice. Ubuntu is going backwards on support for ZFS and native encryption uses a hacky intermediate key under LUKS, but it works. [0] https://github.com/openzfs/zfs/issues/12014 https://github.com/openzfs/zfs/issues/12014 [1] https://github.com/openzfs/zfs/issues/12594 https://github.com/openzfs/zfs/issues/12594 [2]https://serverfault.com/questions/265779/split-a-zfs-clone https://serverfault.com/questions/265779/split-a-zfs-clone
crabbone 3y ago
We don't use ZFS as a filesystem, we use it to aggregate multiple EBS into a single pool, and present it to... stock Ubuntu (one found in AWS marketplace) as a single big block device. We also use ZFS for testing of our own block device. My biggest problems with it are related to this activity. Hot-plugging or removing block devices from the pool often leads to unrepeatable pool, and the tests have to be scrambled because the whole system needs to be rebooted.
- gorkish 3y ago
  >We don't use ZFS as a filesystem, we use it to aggregate multiple EBS into a single pool, and present it to... stock Ubuntu (one found in AWS marketplace) as a single big block device. I don't understand what you mean by this comment. Are you creating a zvol on your zpool and loopback mounting it with a different filesystem? It sounds like you are very much using ZFS as a filesystem, even if you are not booting from it or using it for any kind of high availability purpose. I do agree with you on the matter that there should be a way to clear out an (UNAVAIL vdev / SUSPENDED pool) without a reboot. This has been a thorn in my side for as long as ZFS has existed. The deadlock-by-design line is really quite nonsense.
  - crabbone 3y ago
    No, we don't use it as a filesystem at all. Our product is similar to EBS. So, nothing is mounted anywhere. Users of our product will get a slice or a whole volume thus created and usually would put some filesystem on it, but they will see that volume as a LUN in iSCSI portal we give them an address of. So, in principle, they could also use it to create a ZFS filesystem, but I doubt anyone uses it like that. Most typical usages are either filesystem (something like Ext4 or XFS) or an object store.
cyrnel 3y ago
I use it in Kubernetes via https://github.com/openebs/zfs-localpv https://github.com/openebs/zfs-localpv The PersistentVolume API is a nice way to divvy up a shared resource across different teams, and using ZFS for that gives us the snapshotting, deduplication, and compression for free. For our workloads, it benchmarked faster than XFS so it was a no-brainer.
Datagenerator 3y ago
In production since the first Release of FreeBSD in our Research company. We layered our storage to disconnect hardware Lifecycles from OS. The spinning disks are pooled together using ScaleIO and provided to FreeBSD on ESX as physical devices which live forever through each EOL cycle of the hardware every five up to eight years. We have been serving ten PetaBytes for 15 years without hiccups. We expose SMB3/NFS4 and S3 and have nightly DR to mirror sites using ZFS binary difference Send and Receive cronjobs which are fully automated and monitored. Basically the ScaleIO SDS servers can be replaced during daytime while the upper layer keeps going on for HPC and thousands of other jobs. Since hardware provides more capacity after each Lifecycle we can increase the ZFS pool on the fly and have the most profound 24/7 storage environment without any major interruptions for 15 YEARS and counting. Our partners are having multiple outages due Lustre and other problems, my 2c
mgoetzke 3y ago
We use it with SmartOS on hypervisors running customer Bhyve machines. ZFS (and smartos) works really really well. . The fact how I can replicate live data of arbitrary size (many TB sized filesystems) in small chunks to other hosts every minute greatly increased my deep sleep quality. Of course databases with multi-host write etc are nice, but in our use case, all customers are rather small with just lots and lots of files (medical and otherwise), the database itself is rather small and doesn't need replication. Best thing, on the receiver side of the backup ZFS ensures due to its architecture that the diff is directly applied on top of the existing filesystem, while in normal differential backups one might find out months or years later that one diff snapshot was damaged in transfer or is not accessible. zfs scrubbing with S.M.A.R.T monitoring also helps a lot to ensure drive quality over time. # Gotchas ZFS: - There is no undo etc, this is unix, so beware of wrong commands. - ZFS Scrubbing can be stopped (it does sometimes affect io speed), but ZFS resilvering cannot. This can lead to performance issues. - There must be enough RAM for the caching to work well and synchronous workloads do well with good write cache drives (ZIL) - Data usage patterns should fit well with the Append Log schema of ZFS. E.g databases such as LevelDB worked really well. Others are not slow, but need a good ZIL more then when the pattern fits. SmartOS: Some minor gotchas with how vmadm deletes zfs filesystems or in general with SmartOS, e.g when having too many snapshots, but everything quite predictable.
mnw21cam 3y ago
Yeah, a few years ago I steered our university research group towards using ZFS for a couple of new servers with 1.5PB storage each (2.0PB before raidz2), on RHEL (would have preferred Debian). Absolutely no problems with the system. The IT guys were a bit doubtful whether it would work well, for instance they insisted that XFS would be better at managing parallel requests, but when the system happily saturates a 10Gb ethernet with plenty of bandwidth to spare I don't think there's a problem. We have had ZFS flag up one hard drive so far that was returning false data, which was then hot-swapped out. Other filesystems would probably have just suffered data corruption without us knowing it.
opk 3y ago
My workplace has been using ZFS for years. I've yet to see an alternative come close to being compelling. We use OpenZFS on all our RHEL servers and even workstations with XFS for root partitions and smaller clients. Our primary file server is Solaris 11 but is coming to the end of its long life. The plan is to use FreeBSD on the replacement. ZFS has become so valuable to us that it alone can be a major factor in directing our other choices. Is nice to see it advancing in useful ways. Hopefully this leads to offline dedup without the runtime memory costs.

noaheverett 3y ago

Running in production for about 3 years with Ubuntu 20.04 / zfs 0.8.3. ZFS is being used as the datastore for a cluster of LXD/LXC instances over multiple physical hosts. I have the OS setup on its own dedicated drive and ZFS striped/cloned over 4 NVMe drives. No gotchas / issues, works well, easy to setup. I am looking forward to the Direct IO speed improvements for NVMe drives with https://github.com/openzfs/zfs/pull/10018 https://github.com/openzfs/zfs/pull/10018 edit: one thing I forgot to mention is, when creating your pool make sure to import your drives by ID (zpool import -d /dev/disk/by-id/ <poolname>) instead of name in case name assignments change somehow [1] [1] https://superuser.com/questions/1732532/zfs-disk-drive-letter-changed-how-to-reimport-by-id https://superuser.com/questions/1732532/zfs-disk-drive-lette...

throw0101c 3y ago
> I am looking forward to the Direct IO speed improvements for NVMe drives with https://github.com/openzfs/zfs/pull/10018 https://github.com/openzfs/zfs/pull/10018 See also "Scaling ZFS for NVMe" by Allan Jude at EuroBSDcon 2022: * https://www.youtube.com/watch?v=v8sl8gj9UnA https://www.youtube.com/watch?v=v8sl8gj9UnA
- noaheverett 3y ago
  Sweet, I appreciate the link!
doctor_eval 3y ago
Sorry if this is a newbie question - but what are you using to share the ZFS file system between physical hosts? I’ve been out of this particular game for a long time.
- noaheverett 3y ago
  In our case each node's ZFS store operates independently, it's not shared. We spin up "VMs" (LXC based) on each node and they talk to each via our private network. Hope that helps!

drewg123 3y ago

We use ZFS in production for the non-content filesystems on a large, ever increasing, percentage of our Netflix Open Connect CDN nodes, replacing geom's gmirror. We had one gotcha (caught on a very limited set of canaries) where a buggy bootloader crashed part-way through boot, leaving the ZFS "bootonce" stuff in a funky state requiring manual recovery (the nodes with gmirror were fine, and fell back to the old image without fuss). This has since been fixed. Note that we do not use ZFS for content, since it is incompatible with efficient use of sendfile (both because there is no async handler for ZFS, so no async sendfile, and because the ARC is not integrated with the page cache, so content would require an extra memory copy to be served).

bakul 3y ago
Is integrating ARC with the page cache a lost cause? If not, may be Netflix can fund it!
- postmodest 3y ago
  I would expect page cache to be one of those platform-dependent things that prevent OpenZFS from doing that. Especially on Linux, and especially because AFAIK the Linux version has the most eyes on it.
  - rincebrain 3y ago
    My understanding, not having dug into it, is that it's possible but just work nobody has done yet, though I'm not sure what the relevant interfaces are in the Linux kernel. One thing that makes the interfaces in Linux much messier than the FreeBSD ones is that a lot of the core functionality you might like to leverage in Linux (workqueues, basically anything more complicated than just calling kmalloc, and of course any SIMD save/restore state, to name three examples I've stumbled over recently) are marked EXPORT_SYMBOL_GPL or just entirely not exported in newer releases, so you get to reimplement the wheel for those, whereas on FreeBSD it's trivial to just use their implementations of such things and shim them to the Solaris-ish interfaces the non-platform-specific code expects. So that makes the Linux-specific code a lot heavier, because upstream is actively hostile.

nightfly 3y ago

Yes, Ubuntu 20.04 and 22.04. But we've been running ZFS in some form or other for 10+ years. ACL support not as good/easy to use as Solaris/FreeBSD. Not having weird pathological performance issues with kernel memory allocation like we had with FreeBSD though. Sometimes we have issues with automatic pool import on boot, so that's something to be careful with. The tooling is great though, and we've never had catastrophic failure that was due to ZFS, only due to failing hardware.

trws 3y ago

I’m not a filesystem admin, but we at LLNL use OpenZFS as the storage layer for all of our Lustre file systems in production, including using raid-z for resilience in each pool (order 100 disks each), and have for most of a decade. That combined with improvements in Lustre have taken the rate of data loss or need to clear large scale shared file systems down to nearly zero. There’s a reason we spend as many engineer hours as we do maintaining it, it’s worth it. LLNL openzfs project: https://computing.llnl.gov/projects/openzfs https://computing.llnl.gov/projects/openzfs Old presentation from intel with info on what was one of our bigger deployments in 2016 (~50pb): https://www.intel.com/content/dam/www/public/us/en/documents/presentation/benefits-zfs-for-lustre-deployments.pdf https://www.intel.com/content/dam/www/public/us/en/documents...

muxator 3y ago
If I'm not mistaken the linux port of ZFS that later became OpenZFS started at LLNL and was a port from FreeBSD (it may have been release ~9). I believe it was called ZFS On Linux or something like that. Nice how things have evolved: from FreeBSD to linux and back. In my mind this has always been a very inspiring example of a public institution working for the public good.
- rincebrain 3y ago
  FreeBSD had its own ZFS port. ZoL, if my ancient memory serves, was at LLNL, not based on the FreeBSD port (if you go _very_ far back in the commit history you can see Brian rebasing against OpenSolaris revisions), but like 2 or 3 different orgs originally announced Linux ports at the same time and then all pooled together, since originally only one of the three was going to have a POSIX layer (the other two didn't need a working POSIX filesystem layer). (I'm not actually sure how much came of this collaboration, I just remember being very amused when within the span of a week or two, three different orgs announced ports, looked at each other, and went "...wait.") Then for a while people developed on either the FreeBSD port, the illumos fork called OpenZFS, or the Linux port, but because (among other reasons) a bunch of development kept happening on the Linux port, it became the defacto upstream and got renamed "OpenZFS", and then FreeBSD more or less got a fresh port from the OpenZFS codebase that is now what it's based on. The macOS port got a fresh sync against that codebase recently and is slowly trying to merge in, and then from there, ???

SkyMarshal 3y ago

Not in production, but using ZoL on my personal workstations. https://zfsonlinux.org/ https://zfsonlinux.org/ Some discussion: https://www.reddit.com/r/NixOS/comments/ops0n0/big_shoutout_to_the_nixos_team_for_the_excellent/ https://www.reddit.com/r/NixOS/comments/ops0n0/big_shoutout_...

unixhero 3y ago

Yes. Using latest ZFS ZFS On Linux distrib on Debian. Using Proxmox. Never had and problems ever, ever.

shrubble 3y ago

The gotcha on Proxmox is that you can't do swapfiles on ZFS, so if your swap isn't made big enough when installing and you format everything as ZFS you have to live with it or do odd workarounds.

tlamponi 3y ago
That's not a gotcha on Proxmox projects but a gotcha everywhere, SWAP on ZFS needs a lot of special care to avoid that it needs to allocate memory during swapping memory out (e.g., if low on memory), causing hard-freezes. It can be made somewhat[0] stable with using no (or a simple compression like ZLE), avoiding log devices and caches it can work, but it's way simpler and guaranteed stable to just use a separate partition. [0] even setting the myriad of options there still exist reports about hangs, like https://github.com/openzfs/zfs/issues/7734#issuecomment-416706195 https://github.com/openzfs/zfs/issues/7734#issuecomment-4167...
gorkish 3y ago
1. You shouldn't swap on ZFS, but really 2. You shouldn't swap on a filesystem, but really 3. You shouldn't swap at all. On a VM host you have a bit of leeway for memory overcommit, but ideally if you have a box that frequently has to rely on swap, something is bad wrong. I don't think it's really fair to criticize a worst-choice configuration for being poorly supported here. There are simply too many opportunities for catch-22/deadlock scenarios in the interaction of the fs and allocator during OOM...

gigatexal 3y ago

Any of the enterprise customers of klara Systems are likely ZFS production folks. https://klarasystems.com/?amp https://klarasystems.com/?amp

crest 3y ago

I use it as my default file system on FreeBSD. It was rough in FreeBSD 7.x (around 2009), but starting with FreeBSD 8.x it has been rock solid to this day. The only gotcha (which the documentation warns about) has been that automatic block level deduplication is only useful in a few special applications and has a large main memory overhead unless you can accept terrible performance for normal operations (e.g. a bandwidth limited offsite backup).

yjftsjthsd-h 3y ago

Sure; we get good mileage out of compression and snapshots (well, mostly send-recv for moving data around rather than snapshots in-place). I think the only problems have been very specific to our install process (non-standard kernel in the live environment; if we used the normal distro install process it would be fine).

szundi 3y ago

Yes and it is awesome, no issues.

Drybones 3y ago

We use ZFS on every server we deploy We typically use Proxmox. It’s a convenient node host setup and usually has a very up to date zfs and it’s stable I just wouldn’t use the Proxmox web ui for zfs configuration. It doesn’t have up to date options. Always configure zfs on the cli

tlamponi 3y ago
> I just wouldn’t use the Proxmox web ui for zfs configuration. It doesn’t have up to date options. What options are missing for you? Would be great if you could open an enhancement request over at https://bugzilla.proxmox.com/ https://bugzilla.proxmox.com/ for tracking this (no promises on (immediate) implementation though).

justinclift 3y ago

TrueNAS (www.truenas.com) uses ZFS for the storage layer across it's product range (storage software). They have both FreeBSD and Linux based stuff, targeting different use cases.

explorer83 3y ago
I use TrueNAS (FreeBSD) version. ZFS makes keeping data snapshots and external backups a fairly easy process. It really cut down on server data management time. The only issue I found with it is there was a now long patched SMB bug a few years ago in the TrueNAS software that caused my storage pool to become corrupted after a large transfer. And there was no way for me to recover the data. It was all lost despite the disks still spinning and presumably still having most and the 1 and 0s recoverable.
- justinclift 3y ago
  Hopefully you had backups?

DvdGiessen 3y ago

In production on SmartOS (illumos) servers running applications and VM's, on TrueNAS and plain FreeBSD for various storage and backups, and on a few Linux-based workstations. Using mirrors and raidz2 depending on the needs of the machines. We've successfully survived numerous disk failures (a broken batch of HDD's giving all kinds of small read errors, an SSD that completely failed and disappeared, etc), and were in most cases able to replace them without a second of downtime (would have been all cases if not for disks placed in hard-to-reach places, now only a few minutes downtime to physically swap the disk). Snapshots work perfectly as well. Systems are set up to automatically make snapshots using [1], on boot, on a timer, and right before potentially dangerous operations such as package manager commands as well. I've rolled back after botched OS updates without problems; after a reboot the machine was back in it's old state. Also rolled back a live system a few times after a broken package update, restoring the filesystem state without any issues. Easily accessing old versions of a file is an added bonus which has been helpful a few times. Send/receive is ideal for backups. We are able to send snapshots between machines, even across different OSes, without issues. We've also moved entire pools from one OS to another without problems. Knowing we have automatic snapshots and external backups configured also allows me to be very liberal with giving root access to inexperienced people to various (non-critical) machines, knowing that if anything breaks it will always be easy to roll back, and encouraging them to learn by experimenting a bit, to the point where we can even diff between snapshots to inspect what changed and learn from that. Biggest gotchas so far have been on my personal Arch Linux setup, where the out-of-tree nature of ZFS has caused some issues like a incompatible kernel being installed, the ZFS module failing to compile, and my workstation subsequently being unable to boot. But even that was solved by my entire system running on ZFS: a single rollback from my bootloader [2] and all was back the way it was before. Having good tooling set up definitely helped a lot. My monkey brain has the tendency to think "surely I got it right this time, so no need to make a snapshot before trying out X!", especially when experimenting on my own workstation. Automating snapshots using a systemd timer and hooks added to my package manager saved me a number of times. [1]: https://github.com/psy0rz/zfs_autobackup https://github.com/psy0rz/zfs_autobackup [2]: https://zfsbootmenu.org/ https://zfsbootmenu.org/

csdvrx 3y ago
> Systems are set up to automatically make snapshots I do that with sqlite to keep a selection of snapshots from the last hours, days etc. https://github.com/csdvrx/zfs-autosnapshot https://github.com/csdvrx/zfs-autosnapshot

enneff 3y ago

I use ZFS on Debian for my home file server. The setup is just a tiny NUC with a couple of large USB hard drives, mirrored with ZFS. I’ve had drives fail and painlessly replaced and resilvered them. This is easily the most hassle free file storage setup I’ve owned; been going strong over 10 years now with little to no maintenance.

deleted 3y ago
[deleted]
rsaxvc 3y ago
Pretty much same setup here except I used a Lacie atom NAS, added a USB stick to the motherboard for booting.

quags 3y ago

I have been using zfs for years from ubuntu 18. Easy snapshots, monitoring, choices for raid levels, and ability to very easily copy a dataset remotely with resume or incremental support is awesome. I mainly use it for kvm systems each with their own dataset. Coming from mdadm + lvm from my previous set up is night and day for doing snapshots and backups. I do not use zfs on root for ubuntu instead I do a raid1 software set up for the os and then a zfs set up on other disks - zfs on root was the only gotcha. For FreeBSD zfs on root works fine.

mattjaynes 3y ago

ZFS on Linux has improved a lot in the last few years. We (prematurely) moved to using it in production for our MySQL data about 5 years ago and initially it was a nightmare due to unexplained stalling which would hang MySQL for 15-30 minutes at random times. I'm sure it shortened my life a few years trying to figure out what was wrong when everything was on fire. Fortunately, they have resolved those issues in the subsequent releases and it's been much more pleasant after that.

benlivengood 3y ago

"production" at home on Debian 11, previously on FreeBSD 10-13. The weirdest gotcha has been related to sending encrypted raw snapshots to remote machines[0],[1]. These have been the first instabilities I had with ZFS in roughly 15 years around the filesystem since switching to native encryption this year. Native encryption seems to be barely stable for production use; no actual data corruption but automatic synchronization (I use znapzend) was breaking frequently. Recent kernel updates fixed my problem although some of the bug reports are still open. I only moved on from FreeBSD because of more familiarity with Linux. A slightly annoying property of snapshots and clones is the inability to fully re-root a tree of snapshots, e.g. permanently split a clone from its original source and allow first-class send/receive from that clone. The snapshot which originated the clone needs to stick around forever[2]. This prevents a typical virtual machine imagine process of keeping a base image up to date over time that VMs can be cloned from when desired and eventually removing the storage used by the original base image after e.g. several OS upgrades. I don't have any big performance requirements and most file storage is throughput based on spinning disks which can easily saturate the gigabit network. I also use ZFS on my laptop's SSD under Ubuntu with about 1GB/s performance and no shortage of IOPS and the ability to send snapshots off to the backup system which is pretty nice. Ubuntu is going backwards on support for ZFS and native encryption uses a hacky intermediate key under LUKS, but it works. [0] https://github.com/openzfs/zfs/issues/12014 https://github.com/openzfs/zfs/issues/12014 [1] https://github.com/openzfs/zfs/issues/12594 https://github.com/openzfs/zfs/issues/12594 [2]https://serverfault.com/questions/265779/split-a-zfs-clone https://serverfault.com/questions/265779/split-a-zfs-clone

crabbone 3y ago

We don't use ZFS as a filesystem, we use it to aggregate multiple EBS into a single pool, and present it to... stock Ubuntu (one found in AWS marketplace) as a single big block device. We also use ZFS for testing of our own block device. My biggest problems with it are related to this activity. Hot-plugging or removing block devices from the pool often leads to unrepeatable pool, and the tests have to be scrambled because the whole system needs to be rebooted.

gorkish 3y ago
>We don't use ZFS as a filesystem, we use it to aggregate multiple EBS into a single pool, and present it to... stock Ubuntu (one found in AWS marketplace) as a single big block device. I don't understand what you mean by this comment. Are you creating a zvol on your zpool and loopback mounting it with a different filesystem? It sounds like you are very much using ZFS as a filesystem, even if you are not booting from it or using it for any kind of high availability purpose. I do agree with you on the matter that there should be a way to clear out an (UNAVAIL vdev / SUSPENDED pool) without a reboot. This has been a thorn in my side for as long as ZFS has existed. The deadlock-by-design line is really quite nonsense.
- crabbone 3y ago
  No, we don't use it as a filesystem at all. Our product is similar to EBS. So, nothing is mounted anywhere. Users of our product will get a slice or a whole volume thus created and usually would put some filesystem on it, but they will see that volume as a LUN in iSCSI portal we give them an address of. So, in principle, they could also use it to create a ZFS filesystem, but I doubt anyone uses it like that. Most typical usages are either filesystem (something like Ext4 or XFS) or an object store.

cyrnel 3y ago

I use it in Kubernetes via https://github.com/openebs/zfs-localpv https://github.com/openebs/zfs-localpv The PersistentVolume API is a nice way to divvy up a shared resource across different teams, and using ZFS for that gives us the snapshotting, deduplication, and compression for free. For our workloads, it benchmarked faster than XFS so it was a no-brainer.

Datagenerator 3y ago

In production since the first Release of FreeBSD in our Research company. We layered our storage to disconnect hardware Lifecycles from OS. The spinning disks are pooled together using ScaleIO and provided to FreeBSD on ESX as physical devices which live forever through each EOL cycle of the hardware every five up to eight years. We have been serving ten PetaBytes for 15 years without hiccups. We expose SMB3/NFS4 and S3 and have nightly DR to mirror sites using ZFS binary difference Send and Receive cronjobs which are fully automated and monitored. Basically the ScaleIO SDS servers can be replaced during daytime while the upper layer keeps going on for HPC and thousands of other jobs. Since hardware provides more capacity after each Lifecycle we can increase the ZFS pool on the fly and have the most profound 24/7 storage environment without any major interruptions for 15 YEARS and counting. Our partners are having multiple outages due Lustre and other problems, my 2c

mgoetzke 3y ago

We use it with SmartOS on hypervisors running customer Bhyve machines. ZFS (and smartos) works really really well. . The fact how I can replicate live data of arbitrary size (many TB sized filesystems) in small chunks to other hosts every minute greatly increased my deep sleep quality. Of course databases with multi-host write etc are nice, but in our use case, all customers are rather small with just lots and lots of files (medical and otherwise), the database itself is rather small and doesn't need replication. Best thing, on the receiver side of the backup ZFS ensures due to its architecture that the diff is directly applied on top of the existing filesystem, while in normal differential backups one might find out months or years later that one diff snapshot was damaged in transfer or is not accessible. zfs scrubbing with S.M.A.R.T monitoring also helps a lot to ensure drive quality over time. # Gotchas ZFS: - There is no undo etc, this is unix, so beware of wrong commands. - ZFS Scrubbing can be stopped (it does sometimes affect io speed), but ZFS resilvering cannot. This can lead to performance issues. - There must be enough RAM for the caching to work well and synchronous workloads do well with good write cache drives (ZIL) - Data usage patterns should fit well with the Append Log schema of ZFS. E.g databases such as LevelDB worked really well. Others are not slow, but need a good ZIL more then when the pattern fits. SmartOS: Some minor gotchas with how vmadm deletes zfs filesystems or in general with SmartOS, e.g when having too many snapshots, but everything quite predictable.

mnw21cam 3y ago

Yeah, a few years ago I steered our university research group towards using ZFS for a couple of new servers with 1.5PB storage each (2.0PB before raidz2), on RHEL (would have preferred Debian). Absolutely no problems with the system. The IT guys were a bit doubtful whether it would work well, for instance they insisted that XFS would be better at managing parallel requests, but when the system happily saturates a 10Gb ethernet with plenty of bandwidth to spare I don't think there's a problem. We have had ZFS flag up one hard drive so far that was returning false data, which was then hot-swapped out. Other filesystems would probably have just suffered data corruption without us knowing it.

opk 3y ago

My workplace has been using ZFS for years. I've yet to see an alternative come close to being compelling. We use OpenZFS on all our RHEL servers and even workstations with XFS for root partitions and smaller clients. Our primary file server is Solaris 11 but is coming to the end of its long life. The plan is to use FreeBSD on the replacement. ZFS has become so valuable to us that it alone can be a major factor in directing our other choices. Is nice to see it advancing in useful ways. Hopefully this leads to offline dedup without the runtime memory costs.