5 ms·

This will be downvoted, but I'm a bit skeptical of the "integrate Rust into big C projects" trend. It seems to me that it adds a lot of complexity: now there ar

This will be downvoted, but I'm a bit skeptical of the "integrate Rust into big C projects" trend. It seems to me that it adds a lot of complexity: now there are two toolchains you must understand in order to contribute. There will invariably be subtle bugs at the interface between the two languages. Yes, memory and type-safety are wonderful, but they are not the be all and end all.

gregwebs 5y ago
Does there need to be subtle bugs at the interface if Rust exposes the interface as a C ABI? I think the idea of this article is that you don’t need to understand two tool chains but instead can focus on the harder task of understanding two languages.
- Const-me 5y ago
  Yes. Rust does not consume C headers. For every function and data structure of the API surface you gonna need two language projections, C header and Rust source file. For large API surfaces, especially when both sides of the FFI evolve over time, bugs are inevitable. The problem is not specific to Rust. It affects all languages, except C++ and Objective-C which are backward compatible.
  - steveklabnik 5y ago
    While Rust itself may not directly, tools like bindgen take in those headers and produce the needed glue. https://cxx.rs/ https://cxx.rs/ is another example of a tool that's used in these sorts of cases. You do not have to write these things, nor keep track of changes, by hand.
    - Const-me 5y ago
      I wonder why so many Rust FFI wrappers don’t use these tools? The most popular C library in the world is probably the standard one. Yet in https://github.com/rust-lang/libc https://github.com/rust-lang/libc I don’t see any glue automatically generated from C headers, everything’s written manually. Other examples: https://github.com/servo/rust-png/ https://github.com/servo/rust-png/ https://github.com/bozaro/lz4-rs https://github.com/bozaro/lz4-rs
      - jcranmer 5y ago
        The libc crate's test does actually parse the C header files to generate tests for conformance. There are a few big reasons to consider avoiding parsing the C headers directly: * The C headers might use features in C that are not compatible with the C parser or Rust ABI itself. (C unions used to be a big headache, and bitfields probably still are.). Manually specifying only what you want to import avoids that issue entirely. * As an extended note to the above, many libraries (particularly your system's implementation of libc) are actually not written in any standard C dialect but instead have a lot of compiler-specific things tacked onto it that make it even more difficult to parse and handle correctly than regular C/C++. * Parsing the C headers requires having them installed and being able to find them correctly. When you start dealing with cross-compilation toolchains, this can be a major hassle.
      - steveklabnik 5y ago
        The first two are older than these tools, not sure about lz4. Not 100% sure why but I would imagine that’s at least a reason.
  - JoshTriplett 5y ago
    > Rust does not consume C headers. Yet, at least. I hope one day it can, transparently.
OmarShehata 5y ago
The reason I'm optimistic about this trend is that it's been a common thing in game development for a long time (low level language for execution speed, high level scripting language for development speed) and it works really well. Having to maintain this interface between the language does come with its own set of problems but is well worth the tradeoff especially if you can spend most of your time in one language or the other.
- bosswipe 5y ago
  That's not what's happening here (Android already has low level C and high level Java). What's happening here is two low level languages side by side.
pcwalton 5y ago
Sometimes complexity is worth it for the engineering benefits. We could all be running our software on beautifully simple vintage 1980s microcontrollers: the 6502 was an incredibly elegant and simple design! But then we'd be stuck with computers that can't multiply in hardware or add two floating point numbers or prevent a misbehaving application from taking down the whole system.
pjmlp 5y ago
Unless one starts with another CFront, naturally moving away from how C compilers see the world has some cost, no matter the language.
wyldfire 5y ago
You have good reason to be wary, it is added complexity that should not be taken lightly. The reason to pay this cost is that the alternatives are: (1) do nothing: keep developing solely in C, (2) rewrite the big C project in Rust in its entirety. The cost of option (1) is significant: memory safety errors contribute to 70% of defects not detected by test [1]. Even with static+dynamic checking tools, developers keep introducing bugs into C projects that aren't possible in safe Rust [2]. The cost of option (2) is significant risk. Big C projects have grown over time and have many implicit requirements already satisfied. Discovering all of those incidental bugs and features that are necessary takes a long time and scales with the size of the project. [1] https://msrc-blog.microsoft.com/2019/07/16/a-proactive-approach-to-more-secure-code/ https://msrc-blog.microsoft.com/2019/07/16/a-proactive-appro... [2] http://blog.vmsplice.net/2020/08/why-qemu-should-move-from-c-to-rust.html http://blog.vmsplice.net/2020/08/why-qemu-should-move-from-c...
- kjs3 5y ago
  I don't think the only options are "rewrite in Rust" or "don't". There are other options than Rust.
  - bestouff 5y ago
    Not with the same performance and memory safety guarantees.
    - kjs3 5y ago
      Prove it. This is going to come as a wild shock to you, but Rust isn't the only language that provides this. There have, in fact, been memory safe, performant languages for decades. But we do live it a bubble here where the norm is anything that hasn't been hyped by a pack of fanbois on HN the last 2 years doesn't exist and "an untruth repeated enough times...".
      - steveklabnik 5y ago
        It’s generally expected that the person making the positive claim demonstrate it. You’re asking them to prove something they don’t think exists. (I have some ideas of what you may say but am curious! You clearly have strong feelings here.)
zaphar 5y ago
I would perhaps argue, especially for projects like Android, that memory safety should be considered table stakes going forward. The class of risks that are mitigated or eliminated by memory safety are well worth the effort. There isn't a great excuse to not prioritize memory safety when it's an option.
jchw 5y ago
If the project is a pure C project using a build system that can't integrate Rust very easily, or it would come at a great cost, then I can see this being a totally valid concern. However, AOSP is massive. It is HUGE. And complicated to build. I really, really doubt this addition will make a huge deal. This is the same for many of the huge projects adopting Rust. The worst one so far is the Linux kernel IMO, because at least that is mostly C and GNU Make. And yet even there, that still isn't that bad. Also, as far as bugs at the interface, I'm pretty sure it's a wash. C and Rust are ABI compatible to the extent that matters. The "subtle bugs" are most likely going to be ones that could've existed without the additional complexity of a new language and an interface between them. It's no worse than say, JNI, which doesn't seem to be a serious issue. I'm pretty sure the benefits of Rust can outweigh this cost. As long as you can build the project, you don't really need to understand 2 toolchains to contribute unless your contribution lands across a boundary between 2 toolchains. On a lot of really large projects this may already be the case anyways.
dang 5y ago
Please don't downvote-bait. It's tedious and against the site guidelines. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
throwaway894345 5y ago
I think it makes the most sense as part of a strategy to incrementally migrate a codebase from C to Rust, or else if there is a clean separation between components such that people are working in one ecosystem or the other but rarely both. In whichever case, we should be careful not to overstate the complexity of integrating Rust and C. Specifically, bringing in any given C dependency means integrating its bespoke build system (and those of each transitive dependency) with that of your project (or more likely, you punt on reproducible builds and instead push the dependency hell onto every contributor) which doesn't strike me as considerably less work than integrating the Rust toolchain (and since Rust has sane, standard build/dependency tooling, it's pretty much a one-time cost rather than a once-per-dependency cost).

gregwebs 5y ago

Does there need to be subtle bugs at the interface if Rust exposes the interface as a C ABI? I think the idea of this article is that you don’t need to understand two tool chains but instead can focus on the harder task of understanding two languages.

Const-me 5y ago
Yes. Rust does not consume C headers. For every function and data structure of the API surface you gonna need two language projections, C header and Rust source file. For large API surfaces, especially when both sides of the FFI evolve over time, bugs are inevitable. The problem is not specific to Rust. It affects all languages, except C++ and Objective-C which are backward compatible.
- steveklabnik 5y ago
  While Rust itself may not directly, tools like bindgen take in those headers and produce the needed glue. https://cxx.rs/ https://cxx.rs/ is another example of a tool that's used in these sorts of cases. You do not have to write these things, nor keep track of changes, by hand.
  - Const-me 5y ago
    I wonder why so many Rust FFI wrappers don’t use these tools? The most popular C library in the world is probably the standard one. Yet in https://github.com/rust-lang/libc https://github.com/rust-lang/libc I don’t see any glue automatically generated from C headers, everything’s written manually. Other examples: https://github.com/servo/rust-png/ https://github.com/servo/rust-png/ https://github.com/bozaro/lz4-rs https://github.com/bozaro/lz4-rs
    - jcranmer 5y ago
      The libc crate's test does actually parse the C header files to generate tests for conformance. There are a few big reasons to consider avoiding parsing the C headers directly: * The C headers might use features in C that are not compatible with the C parser or Rust ABI itself. (C unions used to be a big headache, and bitfields probably still are.). Manually specifying only what you want to import avoids that issue entirely. * As an extended note to the above, many libraries (particularly your system's implementation of libc) are actually not written in any standard C dialect but instead have a lot of compiler-specific things tacked onto it that make it even more difficult to parse and handle correctly than regular C/C++. * Parsing the C headers requires having them installed and being able to find them correctly. When you start dealing with cross-compilation toolchains, this can be a major hassle.
    - steveklabnik 5y ago
      The first two are older than these tools, not sure about lz4. Not 100% sure why but I would imagine that’s at least a reason.
- JoshTriplett 5y ago
  > Rust does not consume C headers. Yet, at least. I hope one day it can, transparently.

OmarShehata 5y ago

The reason I'm optimistic about this trend is that it's been a common thing in game development for a long time (low level language for execution speed, high level scripting language for development speed) and it works really well. Having to maintain this interface between the language does come with its own set of problems but is well worth the tradeoff especially if you can spend most of your time in one language or the other.

bosswipe 5y ago
That's not what's happening here (Android already has low level C and high level Java). What's happening here is two low level languages side by side.

pcwalton 5y ago

Sometimes complexity is worth it for the engineering benefits. We could all be running our software on beautifully simple vintage 1980s microcontrollers: the 6502 was an incredibly elegant and simple design! But then we'd be stuck with computers that can't multiply in hardware or add two floating point numbers or prevent a misbehaving application from taking down the whole system.

pjmlp 5y ago

Unless one starts with another CFront, naturally moving away from how C compilers see the world has some cost, no matter the language.

wyldfire 5y ago

You have good reason to be wary, it is added complexity that should not be taken lightly. The reason to pay this cost is that the alternatives are: (1) do nothing: keep developing solely in C, (2) rewrite the big C project in Rust in its entirety. The cost of option (1) is significant: memory safety errors contribute to 70% of defects not detected by test [1]. Even with static+dynamic checking tools, developers keep introducing bugs into C projects that aren't possible in safe Rust [2]. The cost of option (2) is significant risk. Big C projects have grown over time and have many implicit requirements already satisfied. Discovering all of those incidental bugs and features that are necessary takes a long time and scales with the size of the project. [1] https://msrc-blog.microsoft.com/2019/07/16/a-proactive-approach-to-more-secure-code/ https://msrc-blog.microsoft.com/2019/07/16/a-proactive-appro... [2] http://blog.vmsplice.net/2020/08/why-qemu-should-move-from-c-to-rust.html http://blog.vmsplice.net/2020/08/why-qemu-should-move-from-c...

kjs3 5y ago
I don't think the only options are "rewrite in Rust" or "don't". There are other options than Rust.
- bestouff 5y ago
  Not with the same performance and memory safety guarantees.
  - kjs3 5y ago
    Prove it. This is going to come as a wild shock to you, but Rust isn't the only language that provides this. There have, in fact, been memory safe, performant languages for decades. But we do live it a bubble here where the norm is anything that hasn't been hyped by a pack of fanbois on HN the last 2 years doesn't exist and "an untruth repeated enough times...".
    - steveklabnik 5y ago
      It’s generally expected that the person making the positive claim demonstrate it. You’re asking them to prove something they don’t think exists. (I have some ideas of what you may say but am curious! You clearly have strong feelings here.)

zaphar 5y ago

I would perhaps argue, especially for projects like Android, that memory safety should be considered table stakes going forward. The class of risks that are mitigated or eliminated by memory safety are well worth the effort. There isn't a great excuse to not prioritize memory safety when it's an option.

jchw 5y ago

If the project is a pure C project using a build system that can't integrate Rust very easily, or it would come at a great cost, then I can see this being a totally valid concern. However, AOSP is massive. It is HUGE. And complicated to build. I really, really doubt this addition will make a huge deal. This is the same for many of the huge projects adopting Rust. The worst one so far is the Linux kernel IMO, because at least that is mostly C and GNU Make. And yet even there, that still isn't that bad. Also, as far as bugs at the interface, I'm pretty sure it's a wash. C and Rust are ABI compatible to the extent that matters. The "subtle bugs" are most likely going to be ones that could've existed without the additional complexity of a new language and an interface between them. It's no worse than say, JNI, which doesn't seem to be a serious issue. I'm pretty sure the benefits of Rust can outweigh this cost. As long as you can build the project, you don't really need to understand 2 toolchains to contribute unless your contribution lands across a boundary between 2 toolchains. On a lot of really large projects this may already be the case anyways.

dang 5y ago

Please don't downvote-bait. It's tedious and against the site guidelines. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html

throwaway894345 5y ago

I think it makes the most sense as part of a strategy to incrementally migrate a codebase from C to Rust, or else if there is a clean separation between components such that people are working in one ecosystem or the other but rarely both. In whichever case, we should be careful not to overstate the complexity of integrating Rust and C. Specifically, bringing in any given C dependency means integrating its bespoke build system (and those of each transitive dependency) with that of your project (or more likely, you punt on reproducible builds and instead push the dependency hell onto every contributor) which doesn't strike me as considerably less work than integrating the Rust toolchain (and since Rust has sane, standard build/dependency tooling, it's pretty much a one-time cost rather than a once-per-dependency cost).