6 ms·

We use ZFS in production for the non-content filesystems on a large, ever increasing, percentage of our Netflix Open Connect CDN nodes, replacing geom's gmirror

We use ZFS in production for the non-content filesystems on a large, ever increasing, percentage of our Netflix Open Connect CDN nodes, replacing geom's gmirror. We had one gotcha (caught on a very limited set of canaries) where a buggy bootloader crashed part-way through boot, leaving the ZFS "bootonce" stuff in a funky state requiring manual recovery (the nodes with gmirror were fine, and fell back to the old image without fuss). This has since been fixed. Note that we do not use ZFS for content, since it is incompatible with efficient use of sendfile (both because there is no async handler for ZFS, so no async sendfile, and because the ARC is not integrated with the page cache, so content would require an extra memory copy to be served).

bakul 3y ago
Is integrating ARC with the page cache a lost cause? If not, may be Netflix can fund it!
- postmodest 3y ago
  I would expect page cache to be one of those platform-dependent things that prevent OpenZFS from doing that. Especially on Linux, and especially because AFAIK the Linux version has the most eyes on it.
  - rincebrain 3y ago
    My understanding, not having dug into it, is that it's possible but just work nobody has done yet, though I'm not sure what the relevant interfaces are in the Linux kernel. One thing that makes the interfaces in Linux much messier than the FreeBSD ones is that a lot of the core functionality you might like to leverage in Linux (workqueues, basically anything more complicated than just calling kmalloc, and of course any SIMD save/restore state, to name three examples I've stumbled over recently) are marked EXPORT_SYMBOL_GPL or just entirely not exported in newer releases, so you get to reimplement the wheel for those, whereas on FreeBSD it's trivial to just use their implementations of such things and shim them to the Solaris-ish interfaces the non-platform-specific code expects. So that makes the Linux-specific code a lot heavier, because upstream is actively hostile.
    - Dylan16807 3y ago
      > SIMD save/restore state I wish someone would come in and convince the kernel devs that "hey, if you want EXPORT_SYMBOL_GPL to have legal weight in a copyleft sense then you can't just slap it onto interfaces for political reasons"
      - rincebrain 3y ago
        I don't think they care about it having legal weight, that ship sailed long ago when they started advocating for just slapping SYMBOL_GPL on things out of spite; I think they care about excluding people from using their software. IMO Linus should stop being half-and-half about it and either mark everything SYMBOL_GPL and see how well that goes or stop this nonsense.
        Filligree 3y ago
        I just don't understand why they're so anti-ZFS. I want my data to survive, please...
        rincebrain 3y ago
        My impression is that some of the Linux kernel devs are anti-anything that's not GPL-compatible, of any sort, regardless of the particulars. Linus himself also made remarks about ZFS at one point that were pretty...hostile. [1] [2] > The fact is, the whole point of the GPL is that you're being "paid" in terms of tit-for-tat: we give source code to you for free, but we want source code improvements back. If you don't do that but instead say "I think this is _legal_, but I'm not going to help you" you certainly don't get any help from us. > So things that are outside the kernel tree simply do not matter to us. They get absolutely zero attention. We simply don't care. It's that simple. > And things that don't do that "give back" have no business talking about us being assholes when we don't care about them. > See? Note that there's at least one unfixed Linux kernel bug that was found by OpenZFS users, reproducible without using OpenZFS in any way, reported with a patch, and ignored. [3] So "not giving back" is a dubious claim. [1] - https://arstechnica.com/gadgets/2020/01/linus-torvalds-zfs-statements-arent-right-heres-the-straight-dope/ https://arstechnica.com/gadgets/2020/01/linus-torvalds-zfs-s... [2] - https://www.realworldtech.com/forum/?threadid=189711&curpostid=189959 https://www.realworldtech.com/forum/?threadid=189711&curpost... [3] - https://bugzilla.kernel.org/show_bug.cgi?id=212295 https://bugzilla.kernel.org/show_bug.cgi?id=212295
  - the8472 3y ago
    Could the linux integration use dax[0] to bypass the page cache and go straight to ARC? [0] https://www.kernel.org/doc/Documentation/filesystems/dax.txt https://www.kernel.org/doc/Documentation/filesystems/dax.txt
lifty 3y ago
what do you use for the content filesystem?
- drewg123 3y ago
  FreeBSD's UFS
gigatexal 3y ago
This is amazing. A detail of Netflix that, I a plebe, wouldn’t know if not for this site.
- ComputerGuru 3y ago
  Actually, Drew's presentations about Netflix, FreeBSD, ZFS, saturating high-bandwidth network adapters, etc. are legendary and have been posted far and wide. But having him available to answer questions on HN just takes it to a whole 'nother level.
  - drewg123 3y ago
    You're making me blush.. But, to set the record straight: I actually know very little about ZFS, beyond basic user/admin knowledge (from having run it for ~15 years). I've never spoken about it, and other members of the team I work for at Netflix are far more knowledgeable about ZFS, and are the ones who have managed the conversion of our fleet to ZFS for non-content partitions.
    - gigatexal 3y ago
      Have they ever blogged or spoke at conferences about it? I soak up all that content -- least I try to.
    - ComputerGuru 3y ago
      I’ve devoured your FreeBSD networking presentations but I guess I must have confused a post about tracking down a ZFS bug in production written by someone else with all the other content you’ve produced. Back to the topic at hand, it’s actually scary how few software expose control over whether or not sendfile is used, assuming support is only a matter of OS and kernel version but not taking into account filesystem limitations. I ran into a terrible Samba on FreeBSD bug (shares remotely disconnected and connections reset with moderate levels of concurrent ro access from even a single client) that I ultimately tracked down to sendfile being enabled in the (default?) config - so it wasn’t just the expected “performance requirements not being met” with sendfile on ZFS but even other reliability issues (almost certainly exposing a different underlying bug, tbh). Imagine if Samba didn’t have a tubeable to set/override sendfile support, though.
throw0101c 3y ago
Any use of boot environments for easy(er?) rollbacks of OS updates?
- drewg123 3y ago
  Yes. That's the bootonce thing I was talking about. When we update the OS, we set the "bootonce" flag via bectl activate -t to ensure we fall back to the previous BE if the current BE is borked and not bootable. This is the same functionality we had by keeping a primary and secondary root partition in geom a and toggling the bootable partition via the bootonce flag in gpart.
xmodem 3y ago
If you can share, what type of non-content data do the nodes store? Is this just OS+application+logs?

bakul 3y ago

Is integrating ARC with the page cache a lost cause? If not, may be Netflix can fund it!

postmodest 3y ago
I would expect page cache to be one of those platform-dependent things that prevent OpenZFS from doing that. Especially on Linux, and especially because AFAIK the Linux version has the most eyes on it.
- rincebrain 3y ago
  My understanding, not having dug into it, is that it's possible but just work nobody has done yet, though I'm not sure what the relevant interfaces are in the Linux kernel. One thing that makes the interfaces in Linux much messier than the FreeBSD ones is that a lot of the core functionality you might like to leverage in Linux (workqueues, basically anything more complicated than just calling kmalloc, and of course any SIMD save/restore state, to name three examples I've stumbled over recently) are marked EXPORT_SYMBOL_GPL or just entirely not exported in newer releases, so you get to reimplement the wheel for those, whereas on FreeBSD it's trivial to just use their implementations of such things and shim them to the Solaris-ish interfaces the non-platform-specific code expects. So that makes the Linux-specific code a lot heavier, because upstream is actively hostile.
  - Dylan16807 3y ago
    > SIMD save/restore state I wish someone would come in and convince the kernel devs that "hey, if you want EXPORT_SYMBOL_GPL to have legal weight in a copyleft sense then you can't just slap it onto interfaces for political reasons"
    - rincebrain 3y ago
      I don't think they care about it having legal weight, that ship sailed long ago when they started advocating for just slapping SYMBOL_GPL on things out of spite; I think they care about excluding people from using their software. IMO Linus should stop being half-and-half about it and either mark everything SYMBOL_GPL and see how well that goes or stop this nonsense.
      - Filligree 3y ago
        I just don't understand why they're so anti-ZFS. I want my data to survive, please...
        rincebrain 3y ago
        My impression is that some of the Linux kernel devs are anti-anything that's not GPL-compatible, of any sort, regardless of the particulars. Linus himself also made remarks about ZFS at one point that were pretty...hostile. [1] [2] > The fact is, the whole point of the GPL is that you're being "paid" in terms of tit-for-tat: we give source code to you for free, but we want source code improvements back. If you don't do that but instead say "I think this is _legal_, but I'm not going to help you" you certainly don't get any help from us. > So things that are outside the kernel tree simply do not matter to us. They get absolutely zero attention. We simply don't care. It's that simple. > And things that don't do that "give back" have no business talking about us being assholes when we don't care about them. > See? Note that there's at least one unfixed Linux kernel bug that was found by OpenZFS users, reproducible without using OpenZFS in any way, reported with a patch, and ignored. [3] So "not giving back" is a dubious claim. [1] - https://arstechnica.com/gadgets/2020/01/linus-torvalds-zfs-statements-arent-right-heres-the-straight-dope/ https://arstechnica.com/gadgets/2020/01/linus-torvalds-zfs-s... [2] - https://www.realworldtech.com/forum/?threadid=189711&curpostid=189959 https://www.realworldtech.com/forum/?threadid=189711&curpost... [3] - https://bugzilla.kernel.org/show_bug.cgi?id=212295 https://bugzilla.kernel.org/show_bug.cgi?id=212295
- the8472 3y ago
  Could the linux integration use dax[0] to bypass the page cache and go straight to ARC? [0] https://www.kernel.org/doc/Documentation/filesystems/dax.txt https://www.kernel.org/doc/Documentation/filesystems/dax.txt

lifty 3y ago

what do you use for the content filesystem?

drewg123 3y ago
FreeBSD's UFS

gigatexal 3y ago

This is amazing. A detail of Netflix that, I a plebe, wouldn’t know if not for this site.

ComputerGuru 3y ago
Actually, Drew's presentations about Netflix, FreeBSD, ZFS, saturating high-bandwidth network adapters, etc. are legendary and have been posted far and wide. But having him available to answer questions on HN just takes it to a whole 'nother level.
- drewg123 3y ago
  You're making me blush.. But, to set the record straight: I actually know very little about ZFS, beyond basic user/admin knowledge (from having run it for ~15 years). I've never spoken about it, and other members of the team I work for at Netflix are far more knowledgeable about ZFS, and are the ones who have managed the conversion of our fleet to ZFS for non-content partitions.
  - gigatexal 3y ago
    Have they ever blogged or spoke at conferences about it? I soak up all that content -- least I try to.
  - ComputerGuru 3y ago
    I’ve devoured your FreeBSD networking presentations but I guess I must have confused a post about tracking down a ZFS bug in production written by someone else with all the other content you’ve produced. Back to the topic at hand, it’s actually scary how few software expose control over whether or not sendfile is used, assuming support is only a matter of OS and kernel version but not taking into account filesystem limitations. I ran into a terrible Samba on FreeBSD bug (shares remotely disconnected and connections reset with moderate levels of concurrent ro access from even a single client) that I ultimately tracked down to sendfile being enabled in the (default?) config - so it wasn’t just the expected “performance requirements not being met” with sendfile on ZFS but even other reliability issues (almost certainly exposing a different underlying bug, tbh). Imagine if Samba didn’t have a tubeable to set/override sendfile support, though.

throw0101c 3y ago

Any use of boot environments for easy(er?) rollbacks of OS updates?

drewg123 3y ago
Yes. That's the bootonce thing I was talking about. When we update the OS, we set the "bootonce" flag via bectl activate -t to ensure we fall back to the previous BE if the current BE is borked and not bootable. This is the same functionality we had by keeping a primary and secondary root partition in geom a and toggling the bootable partition via the bootonce flag in gpart.

xmodem 3y ago

If you can share, what type of non-content data do the nodes store? Is this just OS+application+logs?