7 ms·

Understanding ZFS Channel Programs

88 points by rodrigo975 5y ago

rbanffy 5y ago
Not to be confused with https://en.wikipedia.org/wiki/Channel_I/O#Channel_program https://en.wikipedia.org/wiki/Channel_I/O#Channel_program
- monocasa 5y ago
  As the article says, it's a direct reference to mainframe style channel I/O.
  - rbanffy 5y ago
    I know, but it felt very different. For starters, it doesn’t run on the channel subsystem, but uses the CPU for it. Also, channel programs (the IBM ones) can do a lot of stuff - IBM’s ISAM uses self-modifying channel programs. These channels can’t be that flexible because they run within the kernel and being too flexible would be a security risk.
    - monocasa 5y ago
      > For starters, it doesn’t run on the channel subsystem, but uses the CPU for it. Low/mid range mainframes don't really make sense anymore, but channel programs did run on the main CPU on quite a few low/mid range mainframe systems. > Also, channel programs (the IBM ones) can do a lot of stuff - IBM’s ISAM uses self-modifying channel programs. These channels can’t be that flexible because they run within the kernel and being too flexible would be a security risk. There's no reason this couldn't do that either as long as it's run in a preemptable context (which I'm pretty sure is the case from looking at the example program). Schemes like eBPF mainly control code flow in pursuit of the goal of running in non preemptable contexts like interrupts.
Rygian 5y ago
I find it surprising that a filesystem feature is so tightly coupled to a specific programming language. Why is it not an API that lets you begin/commit a "channel program" from any language, then execute zfs commands within session? Surely speed of execution is not so big of a constraint.
- magicalhippo 5y ago
  Given that the whole point is to perform script-like operations as an atomic unit in the kernel, it seems to me you'd be reinventing a lot of script-like stuff by exposing a "bare" API for generating a "program" to be run. For example, consider what the API must support to make a channel program that lists all the child datasets of a given dataset, and takes a snapshot of each child dataset that has a certain ZFS property. The looping cannot be done by the user program, as that would introduce edge cases when datasets are created or destroyed between listing and generating the snapshot commands. As such, picking Lua as the API isn't a bad choice I think.
- Hackbraten 5y ago
  I’m not familiar with the feature at all but I wonder whether your suggested alternative API would be powerful enough to atomically do things like: 1. Find the latest three snapshots, and then 2. delete those three snapshots. With a generic API, that’d take two calls, with a possible TOCTOU in between so it wouldn’t be atomic. But if you send that same program to the ZFS subsystem, and have it evaluate and execute the script, the subsystem would be able to guarantee atomicity, right?
  - throwaway2048 5y ago
    it seems pretty trivial to just have some kind of "begin/end atomic section" flag or function rather than jamming in a whole special purpose scripting enviroment.
    - infogulch 5y ago
      begin() loop {}
IshKebab 5y ago
Does this mean there's a Lua interpreter in ZFS? That's pretty wild!
- ducktective 5y ago
  Try pretty scary
  - deleted 5y ago
    [deleted]
  - wahern 5y ago
    > Channel Programs may only be run with root privileges Compare to eBPF, which not only uses a JIT compiler in the kernel, but on popular Linux distributions like Fedora and Ubuntu permits unprivileged users to run eBPF programs. I would never assume a lack of exploitable bugs in the fork of the Lua interpreter used in FreeBSD, but the fact is that eBPF has seen a steady, if infrequent, stream of actual exploits over the years and in particular unprivileged user exploits.
    - throwaway2048 5y ago
      lua is significantly MORE powerful than eBPF not less, and correspondingly will be easier to build exploits in.
      - mnd999 5y ago
        If you have to have root already this doesn’t buy you much though. Perhaps a hard-to-detect way for disgruntled bofhs to do bad things, that’s about it.
- 0x000000001 5y ago
  Appears so https://svnweb.freebsd.org/base/head/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/lua/?pathrev=324163 https://svnweb.freebsd.org/base/head/sys/cddl/contrib/openso...
neonate 5y ago
http://web.archive.org/web/20210908231536/https://klarasystems.com/articles/understanding-zfs-channel-programs/ http://web.archive.org/web/20210908231536/https://klarasyste...
gbrown_ 5y ago
> kernel operations are faster than userland operations Call me a pedant but this grinds my gears. The speed my CPU will carry out instructions is not determined by if it's in a kernel or user context. Avoiding making switches between those contexts certainly means less work though.
- deleted 5y ago
  [deleted]
- remram 5y ago
  How do you feel when people say C is faster than bash? The speed your CPU carries out instructions is the same. A bash statement is usually more instructions than a C statement though.
openasocket 5y ago
I have a special place in my heart for these sort of "weird" features, which seem really exotic compared to the standard POSIX APIs we're all used to. I just enjoy thinking about alternatives and new ways of doing and architecting things. Even if I can't think of a practical use for such a feature, I always appreciate the novelty. It makes me wonder about the future. Are the typical POSIX-style APIs still going to be dominant 50 or 100 years from now, or will new APIs and ideas start to replace them?
- tenebrisalietum 5y ago
  It seems like CPU peripherals are gaining more and more intelligence. I was reading about PCIe's potential successor CXL and the amount of features that allow peripherals to do their own thing with a minimum of CPU intervention seems to be increasing. So at some point I bet you'll be basically uploading BPF like code to peripherals directly in order to accomplish really fast I/O. Don't know how that's going to look at all from a post-POSIX API perspective.

88 points by rodrigo975 5y ago

Not to be confused with https://en.wikipedia.org/wiki/Channel_I/O#Channel_program https://en.wikipedia.org/wiki/Channel_I/O#Channel_program

monocasa 5y ago
As the article says, it's a direct reference to mainframe style channel I/O.
- rbanffy 5y ago
  I know, but it felt very different. For starters, it doesn’t run on the channel subsystem, but uses the CPU for it. Also, channel programs (the IBM ones) can do a lot of stuff - IBM’s ISAM uses self-modifying channel programs. These channels can’t be that flexible because they run within the kernel and being too flexible would be a security risk.
  - monocasa 5y ago
    > For starters, it doesn’t run on the channel subsystem, but uses the CPU for it. Low/mid range mainframes don't really make sense anymore, but channel programs did run on the main CPU on quite a few low/mid range mainframe systems. > Also, channel programs (the IBM ones) can do a lot of stuff - IBM’s ISAM uses self-modifying channel programs. These channels can’t be that flexible because they run within the kernel and being too flexible would be a security risk. There's no reason this couldn't do that either as long as it's run in a preemptable context (which I'm pretty sure is the case from looking at the example program). Schemes like eBPF mainly control code flow in pursuit of the goal of running in non preemptable contexts like interrupts.

Rygian 5y ago

I find it surprising that a filesystem feature is so tightly coupled to a specific programming language. Why is it not an API that lets you begin/commit a "channel program" from any language, then execute zfs commands within session? Surely speed of execution is not so big of a constraint.

magicalhippo 5y ago
Given that the whole point is to perform script-like operations as an atomic unit in the kernel, it seems to me you'd be reinventing a lot of script-like stuff by exposing a "bare" API for generating a "program" to be run. For example, consider what the API must support to make a channel program that lists all the child datasets of a given dataset, and takes a snapshot of each child dataset that has a certain ZFS property. The looping cannot be done by the user program, as that would introduce edge cases when datasets are created or destroyed between listing and generating the snapshot commands. As such, picking Lua as the API isn't a bad choice I think.
Hackbraten 5y ago
I’m not familiar with the feature at all but I wonder whether your suggested alternative API would be powerful enough to atomically do things like: 1. Find the latest three snapshots, and then 2. delete those three snapshots. With a generic API, that’d take two calls, with a possible TOCTOU in between so it wouldn’t be atomic. But if you send that same program to the ZFS subsystem, and have it evaluate and execute the script, the subsystem would be able to guarantee atomicity, right?
- throwaway2048 5y ago
  it seems pretty trivial to just have some kind of "begin/end atomic section" flag or function rather than jamming in a whole special purpose scripting enviroment.
  - infogulch 5y ago
    begin() loop {}

IshKebab 5y ago

Does this mean there's a Lua interpreter in ZFS? That's pretty wild!

ducktective 5y ago
Try pretty scary
- deleted 5y ago
  [deleted]
- wahern 5y ago
  > Channel Programs may only be run with root privileges Compare to eBPF, which not only uses a JIT compiler in the kernel, but on popular Linux distributions like Fedora and Ubuntu permits unprivileged users to run eBPF programs. I would never assume a lack of exploitable bugs in the fork of the Lua interpreter used in FreeBSD, but the fact is that eBPF has seen a steady, if infrequent, stream of actual exploits over the years and in particular unprivileged user exploits.
  - throwaway2048 5y ago
    lua is significantly MORE powerful than eBPF not less, and correspondingly will be easier to build exploits in.
    - mnd999 5y ago
      If you have to have root already this doesn’t buy you much though. Perhaps a hard-to-detect way for disgruntled bofhs to do bad things, that’s about it.
0x000000001 5y ago
Appears so https://svnweb.freebsd.org/base/head/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/lua/?pathrev=324163 https://svnweb.freebsd.org/base/head/sys/cddl/contrib/openso...

neonate 5y ago

http://web.archive.org/web/20210908231536/https://klarasystems.com/articles/understanding-zfs-channel-programs/ http://web.archive.org/web/20210908231536/https://klarasyste...

gbrown_ 5y ago

> kernel operations are faster than userland operations Call me a pedant but this grinds my gears. The speed my CPU will carry out instructions is not determined by if it's in a kernel or user context. Avoiding making switches between those contexts certainly means less work though.

deleted 5y ago
[deleted]
remram 5y ago
How do you feel when people say C is faster than bash? The speed your CPU carries out instructions is the same. A bash statement is usually more instructions than a C statement though.

openasocket 5y ago

I have a special place in my heart for these sort of "weird" features, which seem really exotic compared to the standard POSIX APIs we're all used to. I just enjoy thinking about alternatives and new ways of doing and architecting things. Even if I can't think of a practical use for such a feature, I always appreciate the novelty. It makes me wonder about the future. Are the typical POSIX-style APIs still going to be dominant 50 or 100 years from now, or will new APIs and ideas start to replace them?

tenebrisalietum 5y ago
It seems like CPU peripherals are gaining more and more intelligence. I was reading about PCIe's potential successor CXL and the amount of features that allow peripherals to do their own thing with a minimum of CPU intervention seems to be increasing. So at some point I bet you'll be basically uploading BPF like code to peripherals directly in order to accomplish really fast I/O. Don't know how that's going to look at all from a post-POSIX API perspective.