4 ms·

ZFS Profiling on Arch Linux

183 points by wb14123 3y ago

InvaderFizz 3y ago
This is a great read. It explains in detail the process of enabling debug symbols and using perf to root cause a performance issue. I've never done this type of work, but probably will in the future now that I have an example to follow.
tjoff 3y ago
It is refreshing to see the author explain how and why all the pieces were gathered. Most guides/instructions/bug-reports only lists the end result, omitting information that helps the reader to get a better understanding and be better equipped to solve related issues.
sneak 3y ago
I heard a rumor that ZFS native encryption is somewhat abandoned by its maintainer following it having landed in the release, that it was one person and they moved on/lost interest. I don’t know if this is true or not. I’m still using ZFS on LUKS2 for now, given the issues like this. (You also can’t specify multiple keys in ZFS native encryption.)
- xoa 3y ago
  Sadly there have seemed to be a number of rough edges that have stuck around with ZFS native encryption, which is unfortunate given that the cross platform and remote replication advantages are fundamentally very compelling. Honestly it's kind of curious that polishing that experience hasn't been more of a priority, for commercial interests like iXsystems if no one else. Speaking of which, for people running their ZFS off a NAS the idea of just being able to offload all the encryption and FS computation to that is a compelling part of the value equation beyond sheer storage in principle too. Though that in turn (if one does it with iSCSI) runs into another seemingly neglected area of ZFS (zvols). Nothing is perfect.
  - zrav 3y ago
    What makes you say zvols are neglected? In fact the people that did most of the work porting ZFS to Linux, i.e. LLNL, use zvols for their HPC cluster and they still employ a number developers and the primary maintainer of ZoL. I also remember a quote from them stating that zvols were in fact more mature than vfs.
    - Xorlev 3y ago
      https://github.com/openzfs/zfs/issues/7631 https://github.com/openzfs/zfs/issues/7631 This is a long-standing issue with zvols which affects overall system stability, and has no real solution as of yet.
- justinclift 3y ago
  ZFS on top of LUKS seems to have it's own issues though. :( https://github.com/openzfs/zfs/issues/15533 https://github.com/openzfs/zfs/issues/15533
  - PaulCarrack 3y ago
    Since LUKS is exposed as block device like a hard drive, it's really no different than saying that they are just having issues with certain kind of hard drives. There is a PR out for this: https://github.com/openzfs/zfs/pull/15588 https://github.com/openzfs/zfs/pull/15588
- zrav 3y ago
  See this discussion about the state of native encryption and the linked list of open bugs: https://discourse.practicalzfs.com/t/is-native-encryption-ready-for-production-use/532 https://discourse.practicalzfs.com/t/is-native-encryption-re...
- wb14123 3y ago
  More details on this discussion: https://github.com/openzfs/zfs/issues/6824#issuecomment-1817372085 https://github.com/openzfs/zfs/issues/6824#issuecomment-1817... . Basically he is too busy to continue developing the encryption features but is able to review the related works. There is also discussion about using ZFS on LUKS in the same thread: https://github.com/openzfs/zfs/issues/6824#issuecomment-1819474767 https://github.com/openzfs/zfs/issues/6824#issuecomment-1819...
predmijat 3y ago
On Arch, zfs-linux package has an exact version of linux as a dependency. linux package gets updated fairly often, and by the time zfs-linux package catches up, linux is already on a newer version and you are unable to upgrade. Sometimes it takes quite a while before those two are in sync and you can upgrade your system.
- hnlmorg 3y ago
  Is there lts kernel support for zfs-linux? I found I had to switch to the lts kernel on an Arch gaming rig because of issues with Nvidia drivers.
  - aperum 3y ago
    There is zfs-linux-lts for that.
- mzi 3y ago
  If you reorder your repositories so the zfs repository comes before the default one in the configuration you won't have that problem.
  - commandersaki 3y ago
    Huh, interesting, will have to try that out, thanks.
- tobias2014 3y ago
  In addition to the other comments: you could use the zfs-dkms package which builds it from source on every update
- 79a6ed87 3y ago
  Targetting the linux-lts package as a dependency instead of linux should reduce the impact of this issue
- enormousness 3y ago
  You can install the specific older version of the Linux package that the zfs package needs. You can also pin it so it won't get updated by regular upgrades.
b33j0r 3y ago
Channelling Erik Dubious: “it’s ok, you see the error message, and you navigate to etsy. This config we find is the answer. I lean into neovim this time, today, because it’s all a lessons. And then we type update, and it’s all okay!” The thing he humbly leaves out is “why aren’t you an arcolinux fanboy yet!?”
- b33j0r 3y ago
  I am like disturbed that two people thought complimenting Erik’s work was worth a downvote. Who the heck hates nice people so much to spend that. I’m an arch contributor, your’re barking up the wrong tree.
  - dfc 3y ago
    I think the downvotes had more to do with people not knowing what/who you are referring to. Without the context that you have your comment seems like word salad.
  - Modified3019 3y ago
    Perhaps it's a non-native english speaker thing, but it's very hard to understand your prior post. It looks like a bot post, from the time before AI generation actually became good. The post appears like random disconnected concepts are just thrown together with no clear context or relevancy to the topic of debugging ZFS.
    - b33j0r 3y ago
      You are correct. I really wasn’t aware that talking about famous linux people was such an inside joke in this forum. My bad (sincerely.)
      - dfc 3y ago
        I had never heard of him before. I'm curious what he is he famous for?
nisa 3y ago
Another fix for that is to switch the ZFS license to GPL before building: Just put this line: sed -i "s/CDDL/GPL/g" META in the prepare() section of the PKGBUILD. This will use the in-kernel FPU functions for cryptography and other things like raidz calculations that got EXPORT_SYMBOL_GPL'ed later if the original plumbing is still in the codebase. It's probably not offically recommend and I'm not sure how much these codepaths are tested anymore but I assume that quite a few people that don't need to redistribute anything but need the performance use a hack like this or something similiar. It works fine for me. Personally I lost a ton of respect for the Linux kernel developers after introducing that kernel fpu EXPORT_SYMBOL_GPL thing and backporting it to all kernels and Linus badmouthing ZFS and somehow assuming Oracle is still involved in OpenZFS which is mostly FreeBSD and some sponsors spread out over the world but not related to Oracle.
- nine_k 3y ago
  This evokes in me a distinct feeling of applying a crack to run a pirated game.
  - nisa 3y ago
    I guess it violates the CDDL licence - at the time some distributions like NixOS just patched the EXPORT_SYMBOL_GPL out of the Linux kernel (https://github.com/NixOS/nixpkgs/pull/61076/commits/7b77c27caa8617c82df5c5af6b9ce6ae010d7f9a https://github.com/NixOS/nixpkgs/pull/61076/commits/7b77c27c...) which could even be totally legal for redistribution as the GPL probably allows this?
  - nicman23 3y ago
    it is probably fine as it is just the instructions. i have done it before in a raspberry for some reason i do not remember
- Sodel 3y ago
  I mostly agree with your points except to say that, even if Oracle is not active in the project, the risk is real as long as Oracle owns any of the copyright for any ZFS code. The USL vs BSDi litigation over BSD ultimately came down to a handful of source files out of 18000. ( https://en.m.wikipedia.org/wiki/History_of_the_Berkeley_Software_Distribution https://en.m.wikipedia.org/wiki/History_of_the_Berkeley_Soft... ) I wouldn't talk badly against ZFS, but I would not roll any dice with Oracle as a copyright holder.

183 points by wb14123 3y ago

This is a great read. It explains in detail the process of enabling debug symbols and using perf to root cause a performance issue. I've never done this type of work, but probably will in the future now that I have an example to follow.

tjoff 3y ago

It is refreshing to see the author explain how and why all the pieces were gathered. Most guides/instructions/bug-reports only lists the end result, omitting information that helps the reader to get a better understanding and be better equipped to solve related issues.

sneak 3y ago

I heard a rumor that ZFS native encryption is somewhat abandoned by its maintainer following it having landed in the release, that it was one person and they moved on/lost interest. I don’t know if this is true or not. I’m still using ZFS on LUKS2 for now, given the issues like this. (You also can’t specify multiple keys in ZFS native encryption.)

xoa 3y ago
Sadly there have seemed to be a number of rough edges that have stuck around with ZFS native encryption, which is unfortunate given that the cross platform and remote replication advantages are fundamentally very compelling. Honestly it's kind of curious that polishing that experience hasn't been more of a priority, for commercial interests like iXsystems if no one else. Speaking of which, for people running their ZFS off a NAS the idea of just being able to offload all the encryption and FS computation to that is a compelling part of the value equation beyond sheer storage in principle too. Though that in turn (if one does it with iSCSI) runs into another seemingly neglected area of ZFS (zvols). Nothing is perfect.
- zrav 3y ago
  What makes you say zvols are neglected? In fact the people that did most of the work porting ZFS to Linux, i.e. LLNL, use zvols for their HPC cluster and they still employ a number developers and the primary maintainer of ZoL. I also remember a quote from them stating that zvols were in fact more mature than vfs.
  - Xorlev 3y ago
    https://github.com/openzfs/zfs/issues/7631 https://github.com/openzfs/zfs/issues/7631 This is a long-standing issue with zvols which affects overall system stability, and has no real solution as of yet.
justinclift 3y ago
ZFS on top of LUKS seems to have it's own issues though. :( https://github.com/openzfs/zfs/issues/15533 https://github.com/openzfs/zfs/issues/15533
- PaulCarrack 3y ago
  Since LUKS is exposed as block device like a hard drive, it's really no different than saying that they are just having issues with certain kind of hard drives. There is a PR out for this: https://github.com/openzfs/zfs/pull/15588 https://github.com/openzfs/zfs/pull/15588
zrav 3y ago
See this discussion about the state of native encryption and the linked list of open bugs: https://discourse.practicalzfs.com/t/is-native-encryption-ready-for-production-use/532 https://discourse.practicalzfs.com/t/is-native-encryption-re...
wb14123 3y ago
More details on this discussion: https://github.com/openzfs/zfs/issues/6824#issuecomment-1817372085 https://github.com/openzfs/zfs/issues/6824#issuecomment-1817... . Basically he is too busy to continue developing the encryption features but is able to review the related works. There is also discussion about using ZFS on LUKS in the same thread: https://github.com/openzfs/zfs/issues/6824#issuecomment-1819474767 https://github.com/openzfs/zfs/issues/6824#issuecomment-1819...

predmijat 3y ago

On Arch, zfs-linux package has an exact version of linux as a dependency. linux package gets updated fairly often, and by the time zfs-linux package catches up, linux is already on a newer version and you are unable to upgrade. Sometimes it takes quite a while before those two are in sync and you can upgrade your system.

hnlmorg 3y ago
Is there lts kernel support for zfs-linux? I found I had to switch to the lts kernel on an Arch gaming rig because of issues with Nvidia drivers.
- aperum 3y ago
  There is zfs-linux-lts for that.
mzi 3y ago
If you reorder your repositories so the zfs repository comes before the default one in the configuration you won't have that problem.
- commandersaki 3y ago
  Huh, interesting, will have to try that out, thanks.
tobias2014 3y ago
In addition to the other comments: you could use the zfs-dkms package which builds it from source on every update
79a6ed87 3y ago
Targetting the linux-lts package as a dependency instead of linux should reduce the impact of this issue
enormousness 3y ago
You can install the specific older version of the Linux package that the zfs package needs. You can also pin it so it won't get updated by regular upgrades.

b33j0r 3y ago

Channelling Erik Dubious: “it’s ok, you see the error message, and you navigate to etsy. This config we find is the answer. I lean into neovim this time, today, because it’s all a lessons. And then we type update, and it’s all okay!” The thing he humbly leaves out is “why aren’t you an arcolinux fanboy yet!?”

b33j0r 3y ago
I am like disturbed that two people thought complimenting Erik’s work was worth a downvote. Who the heck hates nice people so much to spend that. I’m an arch contributor, your’re barking up the wrong tree.
- dfc 3y ago
  I think the downvotes had more to do with people not knowing what/who you are referring to. Without the context that you have your comment seems like word salad.
- Modified3019 3y ago
  Perhaps it's a non-native english speaker thing, but it's very hard to understand your prior post. It looks like a bot post, from the time before AI generation actually became good. The post appears like random disconnected concepts are just thrown together with no clear context or relevancy to the topic of debugging ZFS.
  - b33j0r 3y ago
    You are correct. I really wasn’t aware that talking about famous linux people was such an inside joke in this forum. My bad (sincerely.)
    - dfc 3y ago
      I had never heard of him before. I'm curious what he is he famous for?

nisa 3y ago

Another fix for that is to switch the ZFS license to GPL before building: Just put this line: sed -i "s/CDDL/GPL/g" META in the prepare() section of the PKGBUILD. This will use the in-kernel FPU functions for cryptography and other things like raidz calculations that got EXPORT_SYMBOL_GPL'ed later if the original plumbing is still in the codebase. It's probably not offically recommend and I'm not sure how much these codepaths are tested anymore but I assume that quite a few people that don't need to redistribute anything but need the performance use a hack like this or something similiar. It works fine for me. Personally I lost a ton of respect for the Linux kernel developers after introducing that kernel fpu EXPORT_SYMBOL_GPL thing and backporting it to all kernels and Linus badmouthing ZFS and somehow assuming Oracle is still involved in OpenZFS which is mostly FreeBSD and some sponsors spread out over the world but not related to Oracle.

nine_k 3y ago
This evokes in me a distinct feeling of applying a crack to run a pirated game.
- nisa 3y ago
  I guess it violates the CDDL licence - at the time some distributions like NixOS just patched the EXPORT_SYMBOL_GPL out of the Linux kernel (https://github.com/NixOS/nixpkgs/pull/61076/commits/7b77c27caa8617c82df5c5af6b9ce6ae010d7f9a https://github.com/NixOS/nixpkgs/pull/61076/commits/7b77c27c...) which could even be totally legal for redistribution as the GPL probably allows this?
- nicman23 3y ago
  it is probably fine as it is just the instructions. i have done it before in a raspberry for some reason i do not remember
Sodel 3y ago
I mostly agree with your points except to say that, even if Oracle is not active in the project, the risk is real as long as Oracle owns any of the copyright for any ZFS code. The USL vs BSDi litigation over BSD ultimately came down to a handful of source files out of 18000. ( https://en.m.wikipedia.org/wiki/History_of_the_Berkeley_Software_Distribution https://en.m.wikipedia.org/wiki/History_of_the_Berkeley_Soft... ) I wouldn't talk badly against ZFS, but I would not roll any dice with Oracle as a copyright holder.