11 ms·

Committing to Rust for Kernel Code

203 points by wofo 3y ago

xyst 3y ago
Personally, I am very excited. Writing code in Rust is such a pleasure after years in corporate Java/C# shops. Hope to contribute patches in rust to kernel in the near future!
- neonsunset 3y ago
  C# is among the languages that influenced Rust's design :)
  - jimbob45 3y ago
    I wish it had. C# makes it effortless to write high-level and low-level code that lives together. Want pointers and manual allocation? It’s yours. Wanna ignore all that and write interfaces and Java-style classes? You can do that too and pretend that pointers don’t exist. Conversely, Rust never lets you do anything above a mid-level effortlessly. Traits barely get you halfway to typeclasses and you still have to mess with the borrow checker no matter what you’re doing. Add in lifetimes and even simple high-level code gets bogged down with extraneous low-level details.
    - LtdJorge 3y ago
      That's because it's not garbage collected
      - tadfisher 3y ago
        Surely someone has wrapped/ported Boehm and made this as accessible as ARC.
mustache_kimono 3y ago
> Linus Torvalds said that he was seeing a divide between the filesystem and driver maintainers. Developers on the filesystem side tend to be more conservative, while the driver world "is the wild west". Driver authors tend not to understand concurrency, he said, and a lot of the code there is broken and unfixable. So it is unsurprising that there is interest in bringing in a language that better supports the writing of correct and safe code. > Torvalds answered that, while he used to find problems in the LLVM Clang compiler, now he's more likely to find problems with GCC instead; he now builds with Clang. > At the conclusion, Torvalds pointed out that there have been problems over the years with GCC changes breaking the kernel; the same will surely happen with Rust, but it will be the same thing in the end. Based Linus Torvalds.
overflyer 3y ago
[flagged]
- ForkMeOnTinder 3y ago
  There are only two kinds of languages: the ones people complain about and the ones nobody uses.
thesuperbigfrog 3y ago
It is exciting to see Rust start to replace legacy C code in the Linux kernel. It is only the beginning, but everything seems to point to the start of a gradual migration. Reading about Asahi Lina's experience doing GPU driver development was intriguing: https://threadreaderapp.com/thread/1577667445719912450.html https://threadreaderapp.com/thread/1577667445719912450.html At $WORK we are converting a graphics processing application written in C to Rust and the experience has been similarly enlightening. Learning Rust has been a process, but the dividends are real and exciting.
- nequo 3y ago
  What have been the main lessons from your C to Rust migration? Was there anything that surprised you?
  - KineticLensman 3y ago
    Not GP, but for me it was having to think hard about what it means to actually ’own’ data, and learning to structure programs appropriately.
    - junon 3y ago
      This. If rust is giving you a hard time, you're probably designing something some subpar way. That's been my biggest takeaway switching to rust.
      - PartiallyTyped 3y ago
        I’ll disagree a bit here. There are situations where absence of (more) dynamic typing can make your life harder than it already is. An example of this is generating a JSON and only learning the correct schema at runtime. Or connecting to an external database with tables of unknown structures that you can’t type check. You can, of course deal with all these by wrapping the objects in some structure; but it’s just one more thing you need to keep in mind.
Laaas 3y ago
The issue with Rust is the horrible compile times. I actively avoid Rust projects because they take too long to compile. I wish rustc had an option for disabling monomorphization, a true -Os option.
- the8472 3y ago
  What are you compiling? I've "cargo install"-ed - which means deps had to be built too - two rust tools today on a 4c/8t laptop and they were done in maybe two minutes each. Building AUR packages of some C tools isn't much faster in comparison.
- deleted 3y ago
  [deleted]
- rthnbgrredf 3y ago
  How much longer does `rustc` typically take to compile the same programming tasks compared to `gcc`? Is it just "a bit" e.g. 50% slower or by an order of magnitude?
  - mostlylurks 3y ago
    It's not about the compiler, it's about the language. Rust, and C++ as well, have features that generally speaking require the compiler to do a significant amount of work to compile the program. Plain old C can easily compile more than an order of magnitude faster than an equivalent idiomatic Rust or C++ program, even if compiled with the same compiler suite (gcc).
  - fcantournet 3y ago
    You can install rust toolchain with https://rustup.rs/ https://rustup.rs/ in 1 minute, and git clone some project to see for yourself. It's impossible to answer your question directly, as it depends a lot on dependencies and language features.
  - llogiq 3y ago
    That very much depends on the code you're compiling. The factors that come into play are monomorphization (which means the compiler builds one copy of the code per type it is called for), procedural macros (which need to be fully compiled before being able to expand code using them), whether complex type shenanigans are used, etc. etc. Absent that, Rust will compile roughly as fast as C nowadays.
ndesaulniers 3y ago
> Torvalds answered that, while he used to find problems in the LLVM Clang compiler, now he's more likely to find problems with GCC instead; he now builds with Clang. https://github.com/ClangBuiltLinux/linux/issues https://github.com/ClangBuiltLinux/linux/issues is our bug tracker for known issues (a few are tracked in llvm's issue tracker). Bug reporters and future kernel hackers wanted! As I mentioned on mastodon, there's lots of bugs still to be fixed everywhere, but even if we don't fix them, providing competition in the toolchain space has been worth it to users.
- overflyer 3y ago
  I start taking LLVM serious when they give easy to follow documentation on how to compile LLVM completely GCC/binutils/gnu libc independent. I mean a single go to point with easy instructions. You have to Google everything and then you find out about EXPLICIT_LIBGCC_OPT_IN and you read unofficial responses to issues that describe how to use libunwind and compilert to replace libgcc_s. But all that stuff is out of date fast and never works. Even Rene Rebe who developed T2 ranted about this in 2019 and he couldn't completely figure it out without doing a shitton of workarounds and patches. You need to be fucking Stallmann or Torvalds or whatever computer scientist genius themselves to achieve this. Or Google who did this for Android and Fuchsia. My point is it is rocket science at this point. Fucking make that straightforward already dear LLVM project.
  - kwant_kiddo 3y ago
    compiling on mac is very easy/fluent experience for me. I guess OSX is still kinda first-class citizen for LLVM so maybe it does not count?
    - raggi 3y ago
      macOS LLVM is a bit of a special distribution. It's nice that there's a mostly well built LLVM distribution there, but the pain in the ass is that they ship a custom linker and _don't ship lld_, which means you can't readily use their distribution for a lot of common cross build targets - you need to go build at least an lld to get over the line.
askl 3y ago
duplicate of https://news.ycombinator.com/item?id=38381066 https://news.ycombinator.com/item?id=38381066
ziandrome22 3y ago
I remain unconvinced about the viability of Rust for many kinds of systems programming. I'm not sure how I feel about it in the Linux kernel, although I suppose it doesn't matter since it's likely to not penetrate very deeply so to speak. If there are any Rust experts around...what am I missing? The way I see it, Rust is still fundamentally designed to work at a higher level of abstraction than C, and is still mostly dependent on C (even C++ really, due to LLVM/Clang). And bare metal or "first compiler" support for bare metal doesn't seem like it's really intended to be a first class use case. At least that's my impression of the ecosystem from the little time I've had to play around with it. Is this mostly just a thing to get more young people interested in kernel development...allowing them to start out in less important areas and in a language they are passionate about? Or is this a serious proposal about the future of operating systems and other low level infrastructure code? Do you just program everything in unsafe mode? What about runtimes? It seems to me that Rust isn't even really intended to compete with C for the use cases in which C is dominant in 2023. Every indication is that for "serious Rust in production programming" it's mostly a C++ crowd. Whereas for myself and most of the C programmers I know, Zig has sort of filled that similar space and seems to take the concerns of C programmers more seriously and the team has an attitude more in line with the C culture than the Rust team does. I could spend hours writing examples of this but it's even apparent in the way the Zig team has handled its relationship with LLVM, where they seem very serious about trying to not accept it as fundamental to their language and eventually even eliminate the dependency on the C++ code...with Rust it doesn't seem like this is even on the minds of most of the users. It's purely a dependency for them and that isn't seen as being fundamentally at odds with the intended use cases. That is totally okay...but it ain't the C culture if we are to accept that such a thing exists.
- tester756 3y ago
  >It seems to me that Rust isn't even really intended to compete with C for the use cases in which C is dominant in 2023. Like? What other you guys do in C that isn't just integer oriented programming (using ints for everything)?
  - pas 3y ago
__turbobrew__ 3y ago
I wonder what would be worse: 1. Sticking with C for the rest of eternity for the linux kernel and accepting the baggage which comes with that 2. Migrating to rust and accepting the regressions which will inevitably appear as code is rewritten
- tester756 3y ago
  1, of course. Bugs can be fixed.
  - akira2501 3y ago
    Sure, as long as you don't introduce new emergent behavior, which in a kernel can be deadly.
  - bjackman 3y ago
    Memory safety bugs are currently being introduced to the Linux kernel faster than they are being fixed. I assume this is true for all major projects but for Linux there's hard evidence for this: https://syzkaller.appspot.com/upstream/graph/bugs https://syzkaller.appspot.com/upstream/graph/bugs So, yes, all bugs can be fixed. And yes, the world produces enough food to feed the entire population. But people are still going hungry.
    - IshKebab 3y ago
      All the more reason to switch to Rust surely?
      - bjackman 3y ago
        Yep that's what I'm saying
    - hgs3 3y ago
      > Memory safety bugs are currently being introduced to the Linux kernel faster than they are being fixed. I don't understand where all these memory bugs are coming from. Are programmers not testing their code? Are they not running sanitizers, fuzzers, and static analysis? Switching programming languages isn't going to fix broken development process.
janalsncm 3y ago
I do ML professionally, but for performance bottleneck data processing code I’ve converted a lot of scripts to Rust. It’s been a joy even if I can’t say I’m proficient. The combination of GPT4 and the Rust compiler really help to smooth out the learning curve. Some of my python scripts which used to take 24 hours to process data now can finish in under an hour. It’s like magic. Granted theres a development cost so I don’t convert everything but it almost feels like cheating watching it go. Another thing, it seems that the strictness of the language really cuts down on nasty runtime issues and silent bugs. Rust forces you to state explicitly what you want to do.

203 points by wofo 3y ago

xyst 3y ago

Personally, I am very excited. Writing code in Rust is such a pleasure after years in corporate Java/C# shops. Hope to contribute patches in rust to kernel in the near future!

neonsunset 3y ago
C# is among the languages that influenced Rust's design :)
- jimbob45 3y ago
  I wish it had. C# makes it effortless to write high-level and low-level code that lives together. Want pointers and manual allocation? It’s yours. Wanna ignore all that and write interfaces and Java-style classes? You can do that too and pretend that pointers don’t exist. Conversely, Rust never lets you do anything above a mid-level effortlessly. Traits barely get you halfway to typeclasses and you still have to mess with the borrow checker no matter what you’re doing. Add in lifetimes and even simple high-level code gets bogged down with extraneous low-level details.
  - LtdJorge 3y ago
    That's because it's not garbage collected
    - tadfisher 3y ago
      Surely someone has wrapped/ported Boehm and made this as accessible as ARC.

mustache_kimono 3y ago

> Linus Torvalds said that he was seeing a divide between the filesystem and driver maintainers. Developers on the filesystem side tend to be more conservative, while the driver world "is the wild west". Driver authors tend not to understand concurrency, he said, and a lot of the code there is broken and unfixable. So it is unsurprising that there is interest in bringing in a language that better supports the writing of correct and safe code. > Torvalds answered that, while he used to find problems in the LLVM Clang compiler, now he's more likely to find problems with GCC instead; he now builds with Clang. > At the conclusion, Torvalds pointed out that there have been problems over the years with GCC changes breaking the kernel; the same will surely happen with Rust, but it will be the same thing in the end. Based Linus Torvalds.

overflyer 3y ago

[flagged]

ForkMeOnTinder 3y ago
There are only two kinds of languages: the ones people complain about and the ones nobody uses.

thesuperbigfrog 3y ago

It is exciting to see Rust start to replace legacy C code in the Linux kernel. It is only the beginning, but everything seems to point to the start of a gradual migration. Reading about Asahi Lina's experience doing GPU driver development was intriguing: https://threadreaderapp.com/thread/1577667445719912450.html https://threadreaderapp.com/thread/1577667445719912450.html At $WORK we are converting a graphics processing application written in C to Rust and the experience has been similarly enlightening. Learning Rust has been a process, but the dividends are real and exciting.

nequo 3y ago
What have been the main lessons from your C to Rust migration? Was there anything that surprised you?
- KineticLensman 3y ago
  Not GP, but for me it was having to think hard about what it means to actually ’own’ data, and learning to structure programs appropriately.
  - junon 3y ago
    This. If rust is giving you a hard time, you're probably designing something some subpar way. That's been my biggest takeaway switching to rust.
    - PartiallyTyped 3y ago
      I’ll disagree a bit here. There are situations where absence of (more) dynamic typing can make your life harder than it already is. An example of this is generating a JSON and only learning the correct schema at runtime. Or connecting to an external database with tables of unknown structures that you can’t type check. You can, of course deal with all these by wrapping the objects in some structure; but it’s just one more thing you need to keep in mind.

Laaas 3y ago

The issue with Rust is the horrible compile times. I actively avoid Rust projects because they take too long to compile. I wish rustc had an option for disabling monomorphization, a true -Os option.

the8472 3y ago
What are you compiling? I've "cargo install"-ed - which means deps had to be built too - two rust tools today on a 4c/8t laptop and they were done in maybe two minutes each. Building AUR packages of some C tools isn't much faster in comparison.
deleted 3y ago
[deleted]
rthnbgrredf 3y ago
How much longer does `rustc` typically take to compile the same programming tasks compared to `gcc`? Is it just "a bit" e.g. 50% slower or by an order of magnitude?
- mostlylurks 3y ago
  It's not about the compiler, it's about the language. Rust, and C++ as well, have features that generally speaking require the compiler to do a significant amount of work to compile the program. Plain old C can easily compile more than an order of magnitude faster than an equivalent idiomatic Rust or C++ program, even if compiled with the same compiler suite (gcc).
- fcantournet 3y ago
  You can install rust toolchain with https://rustup.rs/ https://rustup.rs/ in 1 minute, and git clone some project to see for yourself. It's impossible to answer your question directly, as it depends a lot on dependencies and language features.
- llogiq 3y ago
  That very much depends on the code you're compiling. The factors that come into play are monomorphization (which means the compiler builds one copy of the code per type it is called for), procedural macros (which need to be fully compiled before being able to expand code using them), whether complex type shenanigans are used, etc. etc. Absent that, Rust will compile roughly as fast as C nowadays.

ndesaulniers 3y ago

> Torvalds answered that, while he used to find problems in the LLVM Clang compiler, now he's more likely to find problems with GCC instead; he now builds with Clang. https://github.com/ClangBuiltLinux/linux/issues https://github.com/ClangBuiltLinux/linux/issues is our bug tracker for known issues (a few are tracked in llvm's issue tracker). Bug reporters and future kernel hackers wanted! As I mentioned on mastodon, there's lots of bugs still to be fixed everywhere, but even if we don't fix them, providing competition in the toolchain space has been worth it to users.

overflyer 3y ago
I start taking LLVM serious when they give easy to follow documentation on how to compile LLVM completely GCC/binutils/gnu libc independent. I mean a single go to point with easy instructions. You have to Google everything and then you find out about EXPLICIT_LIBGCC_OPT_IN and you read unofficial responses to issues that describe how to use libunwind and compilert to replace libgcc_s. But all that stuff is out of date fast and never works. Even Rene Rebe who developed T2 ranted about this in 2019 and he couldn't completely figure it out without doing a shitton of workarounds and patches. You need to be fucking Stallmann or Torvalds or whatever computer scientist genius themselves to achieve this. Or Google who did this for Android and Fuchsia. My point is it is rocket science at this point. Fucking make that straightforward already dear LLVM project.
- kwant_kiddo 3y ago
  compiling on mac is very easy/fluent experience for me. I guess OSX is still kinda first-class citizen for LLVM so maybe it does not count?
  - raggi 3y ago
    macOS LLVM is a bit of a special distribution. It's nice that there's a mostly well built LLVM distribution there, but the pain in the ass is that they ship a custom linker and _don't ship lld_, which means you can't readily use their distribution for a lot of common cross build targets - you need to go build at least an lld to get over the line.

askl 3y ago

duplicate of https://news.ycombinator.com/item?id=38381066 https://news.ycombinator.com/item?id=38381066

ziandrome22 3y ago

I remain unconvinced about the viability of Rust for many kinds of systems programming. I'm not sure how I feel about it in the Linux kernel, although I suppose it doesn't matter since it's likely to not penetrate very deeply so to speak. If there are any Rust experts around...what am I missing? The way I see it, Rust is still fundamentally designed to work at a higher level of abstraction than C, and is still mostly dependent on C (even C++ really, due to LLVM/Clang). And bare metal or "first compiler" support for bare metal doesn't seem like it's really intended to be a first class use case. At least that's my impression of the ecosystem from the little time I've had to play around with it. Is this mostly just a thing to get more young people interested in kernel development...allowing them to start out in less important areas and in a language they are passionate about? Or is this a serious proposal about the future of operating systems and other low level infrastructure code? Do you just program everything in unsafe mode? What about runtimes? It seems to me that Rust isn't even really intended to compete with C for the use cases in which C is dominant in 2023. Every indication is that for "serious Rust in production programming" it's mostly a C++ crowd. Whereas for myself and most of the C programmers I know, Zig has sort of filled that similar space and seems to take the concerns of C programmers more seriously and the team has an attitude more in line with the C culture than the Rust team does. I could spend hours writing examples of this but it's even apparent in the way the Zig team has handled its relationship with LLVM, where they seem very serious about trying to not accept it as fundamental to their language and eventually even eliminate the dependency on the C++ code...with Rust it doesn't seem like this is even on the minds of most of the users. It's purely a dependency for them and that isn't seen as being fundamentally at odds with the intended use cases. That is totally okay...but it ain't the C culture if we are to accept that such a thing exists.

tester756 3y ago
>It seems to me that Rust isn't even really intended to compete with C for the use cases in which C is dominant in 2023. Like? What other you guys do in C that isn't just integer oriented programming (using ints for everything)?
- pas 3y ago

__turbobrew__ 3y ago

I wonder what would be worse: 1. Sticking with C for the rest of eternity for the linux kernel and accepting the baggage which comes with that 2. Migrating to rust and accepting the regressions which will inevitably appear as code is rewritten

tester756 3y ago
1, of course. Bugs can be fixed.
- akira2501 3y ago
  Sure, as long as you don't introduce new emergent behavior, which in a kernel can be deadly.
- bjackman 3y ago
  Memory safety bugs are currently being introduced to the Linux kernel faster than they are being fixed. I assume this is true for all major projects but for Linux there's hard evidence for this: https://syzkaller.appspot.com/upstream/graph/bugs https://syzkaller.appspot.com/upstream/graph/bugs So, yes, all bugs can be fixed. And yes, the world produces enough food to feed the entire population. But people are still going hungry.
  - IshKebab 3y ago
    All the more reason to switch to Rust surely?
    - bjackman 3y ago
      Yep that's what I'm saying
  - hgs3 3y ago
    > Memory safety bugs are currently being introduced to the Linux kernel faster than they are being fixed. I don't understand where all these memory bugs are coming from. Are programmers not testing their code? Are they not running sanitizers, fuzzers, and static analysis? Switching programming languages isn't going to fix broken development process.

janalsncm 3y ago

I do ML professionally, but for performance bottleneck data processing code I’ve converted a lot of scripts to Rust. It’s been a joy even if I can’t say I’m proficient. The combination of GPT4 and the Rust compiler really help to smooth out the learning curve. Some of my python scripts which used to take 24 hours to process data now can finish in under an hour. It’s like magic. Granted theres a development cost so I don’t convert everything but it almost feels like cheating watching it go. Another thing, it seems that the strictness of the language really cuts down on nasty runtime issues and silent bugs. Rust forces you to state explicitly what you want to do.