5 ms·

I find it surprising that a filesystem feature is so tightly coupled to a specific programming language. Why is it not an API that lets you begin/commit a "chan

by Rygian 5y ago

I find it surprising that a filesystem feature is so tightly coupled to a specific programming language. Why is it not an API that lets you begin/commit a "channel program" from any language, then execute zfs commands within session? Surely speed of execution is not so big of a constraint.

magicalhippo 5y ago
Given that the whole point is to perform script-like operations as an atomic unit in the kernel, it seems to me you'd be reinventing a lot of script-like stuff by exposing a "bare" API for generating a "program" to be run. For example, consider what the API must support to make a channel program that lists all the child datasets of a given dataset, and takes a snapshot of each child dataset that has a certain ZFS property. The looping cannot be done by the user program, as that would introduce edge cases when datasets are created or destroyed between listing and generating the snapshot commands. As such, picking Lua as the API isn't a bad choice I think.
Hackbraten 5y ago
I’m not familiar with the feature at all but I wonder whether your suggested alternative API would be powerful enough to atomically do things like: 1. Find the latest three snapshots, and then 2. delete those three snapshots. With a generic API, that’d take two calls, with a possible TOCTOU in between so it wouldn’t be atomic. But if you send that same program to the ZFS subsystem, and have it evaluate and execute the script, the subsystem would be able to guarantee atomicity, right?
- throwaway2048 5y ago
  it seems pretty trivial to just have some kind of "begin/end atomic section" flag or function rather than jamming in a whole special purpose scripting enviroment.
  - infogulch 5y ago
    begin() loop {}
    - Hackbraten 5y ago
      > If the channel program exhausts an instruction or memory limit, a fatal error will be generated and the program will be stopped
  - Hackbraten 5y ago
    But what is the API supposed to do when it comes across a `begin` flag? Stop all filesystem activity in all programs until yours sends the `end` flag? What if that flag never arrives, e.g. due to a bug in your client code?
    - throwaway2048 5y ago
      delay all commits to disk until a fixed timeout is reached for command submission, if the timeout is reached the call returns failed, the same problem exists with this special purpose scripting environment just in other forms.
hossbeast 5y ago
Or, why isn't it leveraging ebpf? Seems like that would be a better fit here than lua.
- 1over137 5y ago
  Isn't ebpf linux only? ZFS is not.
- IshKebab 5y ago
  That's Linux-specific.
- monocasa 5y ago
  eBPF has significantly greater restrictions on the shape of code that can be run, since it needs to be able to run in interrupt context.
derefr 5y ago
Think of it like a GPU compute shader. Like a shader, you have to submit "the whole thing" to the kernel at once, so that it has the program available all at once in its own context, and can guarantee it won't get blocked waiting for your app to submit "the rest of it." To enable the client program to submit "the whole thing", you really need to represent that sequence of operations as some kind of instruction stream loaded into a buffer — either a text-based one (source code in some language), or bytecode in some ISA. (Why not capture a sequence of API operations that do nothing when called, but instead build such an instruction stream; where "committing" them actually runs the instruction stream? Y'know, like a Redis pipeline? Because a "pipeline" like that can't branch based on the result of executing part of it, unless branching/looping/etc are also implemented as operations in this hypothetical fluent-API-DSL. At which point, you're just programming with the semantics of another language, with extra steps.) And if a static analysis pass is going to occur to check the code for runtime complexity, then you're already transforming the input into a different verified output with an intermediate AST-like structure; so at that point you may as well accept a text-based format and expose a "compile" system call. Just as GPU drivers do for compute shaders. I'm guessing that there's probably an internal ZFS API allowing you to instead pass in some kind of bytecode (LuaJIT bytecode?) and just have it directly static-analyzed and plopped into a kernel handle, rather than parsed, then static-analyzed, then codegen'ed. They'd definitely need something like that for testing.
- josephg 5y ago
  This is a tricky problem, but it’s something all good transactional databases solve. For example, the way this is handled in foundationdb is that you create a transaction, then within that transaction issue a series of read and write calls. The read calls are executed immediately, and they see the database as if other concurrent changes weren’t happening. The write calls get batched up and at the end of the transaction block they are submitted together - with information about which keys were read. On submission, the transaction block can fail with E_RETRY - which means a conflict happened, the transaction was entirely aborted and the userland program should run the transaction again from the start. I’d absolutely adore an API like this for filesystem operations. There are all sorts of half baked, dirty workarounds to get around the lack of atomicity in posix filesystems. Eg, that whole “write a file then rename it” thing - which almost nobody implements correctly because it’s error prone and most correct solutions on top of posix have terrible performance.
ggm 5y ago
lua close to kernel is part of FreeBSD. they are leveraging an execution environment which exists in boot time, for the OS. the bootloader does lua.

by Rygian 5y ago

magicalhippo 5y ago

Given that the whole point is to perform script-like operations as an atomic unit in the kernel, it seems to me you'd be reinventing a lot of script-like stuff by exposing a "bare" API for generating a "program" to be run. For example, consider what the API must support to make a channel program that lists all the child datasets of a given dataset, and takes a snapshot of each child dataset that has a certain ZFS property. The looping cannot be done by the user program, as that would introduce edge cases when datasets are created or destroyed between listing and generating the snapshot commands. As such, picking Lua as the API isn't a bad choice I think.

Hackbraten 5y ago

I’m not familiar with the feature at all but I wonder whether your suggested alternative API would be powerful enough to atomically do things like: 1. Find the latest three snapshots, and then 2. delete those three snapshots. With a generic API, that’d take two calls, with a possible TOCTOU in between so it wouldn’t be atomic. But if you send that same program to the ZFS subsystem, and have it evaluate and execute the script, the subsystem would be able to guarantee atomicity, right?

throwaway2048 5y ago
it seems pretty trivial to just have some kind of "begin/end atomic section" flag or function rather than jamming in a whole special purpose scripting enviroment.
- infogulch 5y ago
  begin() loop {}
  - Hackbraten 5y ago
    > If the channel program exhausts an instruction or memory limit, a fatal error will be generated and the program will be stopped
- Hackbraten 5y ago
  But what is the API supposed to do when it comes across a `begin` flag? Stop all filesystem activity in all programs until yours sends the `end` flag? What if that flag never arrives, e.g. due to a bug in your client code?
  - throwaway2048 5y ago
    delay all commits to disk until a fixed timeout is reached for command submission, if the timeout is reached the call returns failed, the same problem exists with this special purpose scripting environment just in other forms.

hossbeast 5y ago

Or, why isn't it leveraging ebpf? Seems like that would be a better fit here than lua.

1over137 5y ago
Isn't ebpf linux only? ZFS is not.
IshKebab 5y ago
That's Linux-specific.
monocasa 5y ago
eBPF has significantly greater restrictions on the shape of code that can be run, since it needs to be able to run in interrupt context.

derefr 5y ago

Think of it like a GPU compute shader. Like a shader, you have to submit "the whole thing" to the kernel at once, so that it has the program available all at once in its own context, and can guarantee it won't get blocked waiting for your app to submit "the rest of it." To enable the client program to submit "the whole thing", you really need to represent that sequence of operations as some kind of instruction stream loaded into a buffer — either a text-based one (source code in some language), or bytecode in some ISA. (Why not capture a sequence of API operations that do nothing when called, but instead build such an instruction stream; where "committing" them actually runs the instruction stream? Y'know, like a Redis pipeline? Because a "pipeline" like that can't branch based on the result of executing part of it, unless branching/looping/etc are also implemented as operations in this hypothetical fluent-API-DSL. At which point, you're just programming with the semantics of another language, with extra steps.) And if a static analysis pass is going to occur to check the code for runtime complexity, then you're already transforming the input into a different verified output with an intermediate AST-like structure; so at that point you may as well accept a text-based format and expose a "compile" system call. Just as GPU drivers do for compute shaders. I'm guessing that there's probably an internal ZFS API allowing you to instead pass in some kind of bytecode (LuaJIT bytecode?) and just have it directly static-analyzed and plopped into a kernel handle, rather than parsed, then static-analyzed, then codegen'ed. They'd definitely need something like that for testing.

josephg 5y ago
This is a tricky problem, but it’s something all good transactional databases solve. For example, the way this is handled in foundationdb is that you create a transaction, then within that transaction issue a series of read and write calls. The read calls are executed immediately, and they see the database as if other concurrent changes weren’t happening. The write calls get batched up and at the end of the transaction block they are submitted together - with information about which keys were read. On submission, the transaction block can fail with E_RETRY - which means a conflict happened, the transaction was entirely aborted and the userland program should run the transaction again from the start. I’d absolutely adore an API like this for filesystem operations. There are all sorts of half baked, dirty workarounds to get around the lack of atomicity in posix filesystems. Eg, that whole “write a file then rename it” thing - which almost nobody implements correctly because it’s error prone and most correct solutions on top of posix have terrible performance.

ggm 5y ago

lua close to kernel is part of FreeBSD. they are leveraging an execution environment which exists in boot time, for the OS. the bootloader does lua.