4 ms·

LKML discusses merging Rust access to filesystem APIs

keikobadthebad 3y ago
From Jan 15th
nercury 3y ago
I am Rust developer. My advice: favor less abstractions. Especially when interfacing with C, you can always make almost 1:1 rust interface. Start with that. Then, when common patterns start to emerge, do the data abstractions first, i.e. define data structures shared by different implementations. There is no reason why you can't write simple code in Rust. Well, except that it's tempting to over complicate things. Start using traits and generics everywhere, and you will enter meta-programming in generics hell. Soon after, you will start demanding new compiler features to survive there.
- dragonelite 3y ago
  Thats why im also keeping an eye out for zig and check how they will develop the next 5 years or so. Hope they don't go so meta programming heavy.
  - actionfromafar 3y ago
    Zig has the especially interesting feature that it can compile C code.
    - pjmlp 3y ago
      That is more a property of shipping the whole LLVM/Clang package with the zig compiler than anything, though.
      - actionfromafar 3y ago
        Whatever the mechanism, it has big implications for how Zig will be used in practice.
        pjmlp 3y ago
        A mechanism available to any language that feels like shipping clang. It is also how Swift gets its C and Objective-C interop, and will have its C++ interop as well. Or how the Java Panama project extracts header file information for native interop.
- lifthrasiir 3y ago
  Great advice, but the resulting one-to-one mapping is not necessarily safe (in terms of Rust), which is in part why people jump early on advanced Rust features. So I would rephrase yours as follows: do not reach for abstractions to achieve safety without much consideration and justification.
  - nercury 3y ago
    I admit I don't know enough about kernel development, but from general experience one common example is resource cleanup. Say, you have several resources that must be cleaned up in certain order. In C, you would just call the appropriate free functions. When abstracting over that with rust, you have to make an annoying choice: Do what's in C, don't implement automatic Drop, but mark your functions unsafe. You get leanest zero-cost implementation, it is straightforward to understand, but needs additional maintenance care to prevent bugs. Wrap resources in unsafe structs that have automatic drop (when goes out of scope). IMO this is a terrible choice, because the maintainer suddenly has to know about which structures have this unsafe cleanup going on. Simple scopes suddenly matter, order of items suddenly matter, it's a mess. Use reference-counting wrappers to track usage and drop the items when they are no longer in use. Most libraries do this, but it's no longer the leanest possible API. There might be another choice, to achieve both zero-cost execution and correct cleanup using metaprogramming, be it macros, generics, or both. That's exactly what I fear the most.
rightbyte 3y ago
"There is little in the way of type safety here; a function cannot know that it was actually passed a pointer to the right sort of inode." Is there no type tag in the inode structs? Edit: Looking at its definition I don't see the problem.
- actionfromafar 3y ago
  Maybe it meant know at compile time?
- dezgeg 3y ago
  Let's have a concrete example, link() in struct inode_operations is defined as: int (*link) (struct dentry *,struct inode *,struct dentry *); Let's then look at the ext4 implementation of the link() operation: static int ext4_link(struct dentry *old_dentry, struct inode *dir, struct dentry *dentry) { ... if ((ext4_test_inode_flag(dir, EXT4_INODE_PROJINHERIT)) && (!projid_eq(EXT4_I(dir)->i_projid, EXT4_I(old_dentry->d_inode)->i_projid))) return -EXDEV; .... In particular that calls EXT4_I(): static inline struct ext4_inode_info *EXT4_I(struct inode *inode) { return container_of(inode, struct ext4_inode_info, vfs_inode); } Which is basically just direct unchecked pointer type conversion. So, calling ext4_link() on any non-ext4 created struct inode will blow up badly.
arnsholt 3y ago
I reread John Regehr's "The Problem with Friendly C" [0] a while back, and a comment towards the end made me think of the Rust in Linux effort. He speculates that "an influential group such as the Android team could create a friendly C dialect" which in turn can drive adoption and consensus on what a better C language should look like. Of course, we know that hasn't happened since 2015 when that article was written, but what has happened is the Android team's rewrite of the Binder in Rust. So I think the main thrust of the argument is correct: interested parties are converging on an answer to the challenges of writing secure C. It's just that rather than evolving C itself, the consensus that seems to be developing is that Rust is a better answer. 0: https://blog.regehr.org/archives/1287 https://blog.regehr.org/archives/1287
- actionfromafar 3y ago
  I think Rust is an improvement over C, but I also think it's unfortunate that there's nothing which is similar to C, but safer. C is incredibly easy to grok. (Even if the "idealized machine model" has grown a bit more involved with newer standards and optimizations. I'm probably not using the correct terminology but I'm referring to what happens with undefined behaviour.) Remind me again why we can't use Pascal? (Except that it's corny, looks funny and I scoffed at it as a youngun.) I wonder what "C with x" would look like, where x is: - real composable types - checked arrays - maybe some pointer checks? But no: - inheritance - objects And also very compatible with plain C, not sure if 100% language compatible would be a good idea. Keeping compatibility with C was one contributing reason C++ became so complicated. But keeping linker and header compatibility would be a godsend. Maybe realistically what I want is Zig, I realise it's the closest existing fit, even though it has metaprogramming and the syntax is a bit different from C. I should really learn it to see if I would be happy with it. Currently my go-to languages are C++ if I have the time, C# if I don't and Python if I just want to explore thoughts.
  - bananapub 3y ago
    > C is incredibly easy to grok. what does this mean, though? writing good, secure C is so hard that literally no one can do it at scale. not Linus, not Theo, not even djb. why do you find "learning enough to write bad code isn't very hard" to be valuable? I'm not even really convinced that C is that useful as an approximation of "how computers work" anymore - caches and NUMA and speculative execution and mmap and large numbers of varying ability cpu cores and asynchronous code make C more of an approximation of "how the PDP worked in the 1970s" (with an abstraction layer that is an endless rolling nightmare of critical security issues), which I guess is a bit useful, but not as the thing we base the world on. and a C with array bounds checking and length-prefix strings would still have a shitty type system and no meta programming and not even an iteration protocol (or ability to add a nice one). > Remind me again why we can't use Pascal? I would guess it's because Unix-alikes (and for a while Windows, though I guess it had less C and more C++ earlier) ate the world and using the language (almost all of) the OS was written in had huge network effects.
chris_wot 3y ago
The problem with the rust developers creating an API based on a file system that is a “toy” (i.e. a limited subset of features) is that the API abstraction does not cater for all file systems. The advise provided is good - develop a file system module in rust for ext2 and then propose a new API.
- mustache_kimono 3y ago
  > The advice provided is good - develop a file system module in rust for ext2 and then propose a new API. Although I think the point re: the API is not really worth quibbling too much about, I think "Go rewrite ext2 in Rust" is pretty goofy advice, when the purpose is something like "so we will know what the API should look like." The minute someone implements a filesystem in Rust used by real people is when the rubber will meet the road. Rust filesystems may be different. Everyone is on notice -- APIs aren't stable in the kernel. They especially aren't stable when no one is using them. Something should be there to avoid 8 different implementations, but this endless -- go reimplement an existing facility, as an exercise, teaches us very little, and is mostly a waste of resources (which I'm beginning to believe is the point).

From Jan 15th

I am Rust developer. My advice: favor less abstractions. Especially when interfacing with C, you can always make almost 1:1 rust interface. Start with that. Then, when common patterns start to emerge, do the data abstractions first, i.e. define data structures shared by different implementations. There is no reason why you can't write simple code in Rust. Well, except that it's tempting to over complicate things. Start using traits and generics everywhere, and you will enter meta-programming in generics hell. Soon after, you will start demanding new compiler features to survive there.

dragonelite 3y ago
Thats why im also keeping an eye out for zig and check how they will develop the next 5 years or so. Hope they don't go so meta programming heavy.
- actionfromafar 3y ago
  Zig has the especially interesting feature that it can compile C code.
  - pjmlp 3y ago
    That is more a property of shipping the whole LLVM/Clang package with the zig compiler than anything, though.
    - actionfromafar 3y ago
      Whatever the mechanism, it has big implications for how Zig will be used in practice.
      - pjmlp 3y ago
        A mechanism available to any language that feels like shipping clang. It is also how Swift gets its C and Objective-C interop, and will have its C++ interop as well. Or how the Java Panama project extracts header file information for native interop.
lifthrasiir 3y ago
Great advice, but the resulting one-to-one mapping is not necessarily safe (in terms of Rust), which is in part why people jump early on advanced Rust features. So I would rephrase yours as follows: do not reach for abstractions to achieve safety without much consideration and justification.
- nercury 3y ago
  I admit I don't know enough about kernel development, but from general experience one common example is resource cleanup. Say, you have several resources that must be cleaned up in certain order. In C, you would just call the appropriate free functions. When abstracting over that with rust, you have to make an annoying choice: Do what's in C, don't implement automatic Drop, but mark your functions unsafe. You get leanest zero-cost implementation, it is straightforward to understand, but needs additional maintenance care to prevent bugs. Wrap resources in unsafe structs that have automatic drop (when goes out of scope). IMO this is a terrible choice, because the maintainer suddenly has to know about which structures have this unsafe cleanup going on. Simple scopes suddenly matter, order of items suddenly matter, it's a mess. Use reference-counting wrappers to track usage and drop the items when they are no longer in use. Most libraries do this, but it's no longer the leanest possible API. There might be another choice, to achieve both zero-cost execution and correct cleanup using metaprogramming, be it macros, generics, or both. That's exactly what I fear the most.

rightbyte 3y ago

"There is little in the way of type safety here; a function cannot know that it was actually passed a pointer to the right sort of inode." Is there no type tag in the inode structs? Edit: Looking at its definition I don't see the problem.

actionfromafar 3y ago
Maybe it meant know at compile time?
dezgeg 3y ago
Let's have a concrete example, link() in struct inode_operations is defined as: int (*link) (struct dentry *,struct inode *,struct dentry *); Let's then look at the ext4 implementation of the link() operation: static int ext4_link(struct dentry *old_dentry, struct inode *dir, struct dentry *dentry) { ... if ((ext4_test_inode_flag(dir, EXT4_INODE_PROJINHERIT)) && (!projid_eq(EXT4_I(dir)->i_projid, EXT4_I(old_dentry->d_inode)->i_projid))) return -EXDEV; .... In particular that calls EXT4_I(): static inline struct ext4_inode_info *EXT4_I(struct inode *inode) { return container_of(inode, struct ext4_inode_info, vfs_inode); } Which is basically just direct unchecked pointer type conversion. So, calling ext4_link() on any non-ext4 created struct inode will blow up badly.

arnsholt 3y ago

I reread John Regehr's "The Problem with Friendly C" [0] a while back, and a comment towards the end made me think of the Rust in Linux effort. He speculates that "an influential group such as the Android team could create a friendly C dialect" which in turn can drive adoption and consensus on what a better C language should look like. Of course, we know that hasn't happened since 2015 when that article was written, but what has happened is the Android team's rewrite of the Binder in Rust. So I think the main thrust of the argument is correct: interested parties are converging on an answer to the challenges of writing secure C. It's just that rather than evolving C itself, the consensus that seems to be developing is that Rust is a better answer. 0: https://blog.regehr.org/archives/1287 https://blog.regehr.org/archives/1287

actionfromafar 3y ago
I think Rust is an improvement over C, but I also think it's unfortunate that there's nothing which is similar to C, but safer. C is incredibly easy to grok. (Even if the "idealized machine model" has grown a bit more involved with newer standards and optimizations. I'm probably not using the correct terminology but I'm referring to what happens with undefined behaviour.) Remind me again why we can't use Pascal? (Except that it's corny, looks funny and I scoffed at it as a youngun.) I wonder what "C with x" would look like, where x is: - real composable types - checked arrays - maybe some pointer checks? But no: - inheritance - objects And also very compatible with plain C, not sure if 100% language compatible would be a good idea. Keeping compatibility with C was one contributing reason C++ became so complicated. But keeping linker and header compatibility would be a godsend. Maybe realistically what I want is Zig, I realise it's the closest existing fit, even though it has metaprogramming and the syntax is a bit different from C. I should really learn it to see if I would be happy with it. Currently my go-to languages are C++ if I have the time, C# if I don't and Python if I just want to explore thoughts.
- bananapub 3y ago
  > C is incredibly easy to grok. what does this mean, though? writing good, secure C is so hard that literally no one can do it at scale. not Linus, not Theo, not even djb. why do you find "learning enough to write bad code isn't very hard" to be valuable? I'm not even really convinced that C is that useful as an approximation of "how computers work" anymore - caches and NUMA and speculative execution and mmap and large numbers of varying ability cpu cores and asynchronous code make C more of an approximation of "how the PDP worked in the 1970s" (with an abstraction layer that is an endless rolling nightmare of critical security issues), which I guess is a bit useful, but not as the thing we base the world on. and a C with array bounds checking and length-prefix strings would still have a shitty type system and no meta programming and not even an iteration protocol (or ability to add a nice one). > Remind me again why we can't use Pascal? I would guess it's because Unix-alikes (and for a while Windows, though I guess it had less C and more C++ earlier) ate the world and using the language (almost all of) the OS was written in had huge network effects.

chris_wot 3y ago

The problem with the rust developers creating an API based on a file system that is a “toy” (i.e. a limited subset of features) is that the API abstraction does not cater for all file systems. The advise provided is good - develop a file system module in rust for ext2 and then propose a new API.

mustache_kimono 3y ago
> The advice provided is good - develop a file system module in rust for ext2 and then propose a new API. Although I think the point re: the API is not really worth quibbling too much about, I think "Go rewrite ext2 in Rust" is pretty goofy advice, when the purpose is something like "so we will know what the API should look like." The minute someone implements a filesystem in Rust used by real people is when the rubber will meet the road. Rust filesystems may be different. Everyone is on notice -- APIs aren't stable in the kernel. They especially aren't stable when no one is using them. Something should be there to avoid 8 different implementations, but this endless -- go reimplement an existing facility, as an exercise, teaches us very little, and is mostly a waste of resources (which I'm beginning to believe is the point).