6 ms·

Integrating Rust into the Android Open Source Project

92 points by theafh 5y ago

gautamcgoel 5y ago
This will be downvoted, but I'm a bit skeptical of the "integrate Rust into big C projects" trend. It seems to me that it adds a lot of complexity: now there are two toolchains you must understand in order to contribute. There will invariably be subtle bugs at the interface between the two languages. Yes, memory and type-safety are wonderful, but they are not the be all and end all.
- gregwebs 5y ago
  Does there need to be subtle bugs at the interface if Rust exposes the interface as a C ABI? I think the idea of this article is that you don’t need to understand two tool chains but instead can focus on the harder task of understanding two languages.
  - Const-me 5y ago
    Yes. Rust does not consume C headers. For every function and data structure of the API surface you gonna need two language projections, C header and Rust source file. For large API surfaces, especially when both sides of the FFI evolve over time, bugs are inevitable. The problem is not specific to Rust. It affects all languages, except C++ and Objective-C which are backward compatible.
    - steveklabnik 5y ago
      While Rust itself may not directly, tools like bindgen take in those headers and produce the needed glue. https://cxx.rs/ https://cxx.rs/ is another example of a tool that's used in these sorts of cases. You do not have to write these things, nor keep track of changes, by hand.
      - Const-me 5y ago
        I wonder why so many Rust FFI wrappers don’t use these tools? The most popular C library in the world is probably the standard one. Yet in https://github.com/rust-lang/libc https://github.com/rust-lang/libc I don’t see any glue automatically generated from C headers, everything’s written manually. Other examples: https://github.com/servo/rust-png/ https://github.com/servo/rust-png/ https://github.com/bozaro/lz4-rs https://github.com/bozaro/lz4-rs
        jcranmer 5y ago
        The libc crate's test does actually parse the C header files to generate tests for conformance. There are a few big reasons to consider avoiding parsing the C headers directly: * The C headers might use features in C that are not compatible with the C parser or Rust ABI itself. (C unions used to be a big headache, and bitfields probably still are.). Manually specifying only what you want to import avoids that issue entirely. * As an extended note to the above, many libraries (particularly your system's implementation of libc) are actually not written in any standard C dialect but instead have a lot of compiler-specific things tacked onto it that make it even more difficult to parse and handle correctly than regular C/C++. * Parsing the C headers requires having them installed and being able to find them correctly. When you start dealing with cross-compilation toolchains, this can be a major hassle.
        steveklabnik 5y ago
        The first two are older than these tools, not sure about lz4. Not 100% sure why but I would imagine that’s at least a reason.
    - JoshTriplett 5y ago
      > Rust does not consume C headers. Yet, at least. I hope one day it can, transparently.
- OmarShehata 5y ago
  The reason I'm optimistic about this trend is that it's been a common thing in game development for a long time (low level language for execution speed, high level scripting language for development speed) and it works really well. Having to maintain this interface between the language does come with its own set of problems but is well worth the tradeoff especially if you can spend most of your time in one language or the other.
  - bosswipe 5y ago
    That's not what's happening here (Android already has low level C and high level Java). What's happening here is two low level languages side by side.
- pcwalton 5y ago
  Sometimes complexity is worth it for the engineering benefits. We could all be running our software on beautifully simple vintage 1980s microcontrollers: the 6502 was an incredibly elegant and simple design! But then we'd be stuck with computers that can't multiply in hardware or add two floating point numbers or prevent a misbehaving application from taking down the whole system.
- pjmlp 5y ago
  Unless one starts with another CFront, naturally moving away from how C compilers see the world has some cost, no matter the language.
- wyldfire 5y ago
  You have good reason to be wary, it is added complexity that should not be taken lightly. The reason to pay this cost is that the alternatives are: (1) do nothing: keep developing solely in C, (2) rewrite the big C project in Rust in its entirety. The cost of option (1) is significant: memory safety errors contribute to 70% of defects not detected by test [1]. Even with static+dynamic checking tools, developers keep introducing bugs into C projects that aren't possible in safe Rust [2]. The cost of option (2) is significant risk. Big C projects have grown over time and have many implicit requirements already satisfied. Discovering all of those incidental bugs and features that are necessary takes a long time and scales with the size of the project. [1] https://msrc-blog.microsoft.com/2019/07/16/a-proactive-approach-to-more-secure-code/ https://msrc-blog.microsoft.com/2019/07/16/a-proactive-appro... [2] http://blog.vmsplice.net/2020/08/why-qemu-should-move-from-c-to-rust.html http://blog.vmsplice.net/2020/08/why-qemu-should-move-from-c...
  - kjs3 5y ago
    I don't think the only options are "rewrite in Rust" or "don't". There are other options than Rust.
    - bestouff 5y ago
      Not with the same performance and memory safety guarantees.
      - kjs3 5y ago
        Prove it. This is going to come as a wild shock to you, but Rust isn't the only language that provides this. There have, in fact, been memory safe, performant languages for decades. But we do live it a bubble here where the norm is anything that hasn't been hyped by a pack of fanbois on HN the last 2 years doesn't exist and "an untruth repeated enough times...".
        steveklabnik 5y ago
        It’s generally expected that the person making the positive claim demonstrate it. You’re asking them to prove something they don’t think exists. (I have some ideas of what you may say but am curious! You clearly have strong feelings here.)
- zaphar 5y ago
  I would perhaps argue, especially for projects like Android, that memory safety should be considered table stakes going forward. The class of risks that are mitigated or eliminated by memory safety are well worth the effort. There isn't a great excuse to not prioritize memory safety when it's an option.
- jchw 5y ago
  If the project is a pure C project using a build system that can't integrate Rust very easily, or it would come at a great cost, then I can see this being a totally valid concern. However, AOSP is massive. It is HUGE. And complicated to build. I really, really doubt this addition will make a huge deal. This is the same for many of the huge projects adopting Rust. The worst one so far is the Linux kernel IMO, because at least that is mostly C and GNU Make. And yet even there, that still isn't that bad. Also, as far as bugs at the interface, I'm pretty sure it's a wash. C and Rust are ABI compatible to the extent that matters. The "subtle bugs" are most likely going to be ones that could've existed without the additional complexity of a new language and an interface between them. It's no worse than say, JNI, which doesn't seem to be a serious issue. I'm pretty sure the benefits of Rust can outweigh this cost. As long as you can build the project, you don't really need to understand 2 toolchains to contribute unless your contribution lands across a boundary between 2 toolchains. On a lot of really large projects this may already be the case anyways.
- dang 5y ago
  Please don't downvote-bait. It's tedious and against the site guidelines. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
- throwaway894345 5y ago
  I think it makes the most sense as part of a strategy to incrementally migrate a codebase from C to Rust, or else if there is a clean separation between components such that people are working in one ecosystem or the other but rarely both. In whichever case, we should be careful not to overstate the complexity of integrating Rust and C. Specifically, bringing in any given C dependency means integrating its bespoke build system (and those of each transitive dependency) with that of your project (or more likely, you punt on reproducible builds and instead push the dependency hell onto every contributor) which doesn't strike me as considerably less work than integrating the Rust toolchain (and since Rust has sane, standard build/dependency tooling, it's pretty much a one-time cost rather than a once-per-dependency cost).
thesuperbigfrog 5y ago
>> Dynamic linkage by default >> By default, the Rust ecosystem assumes that crates will be statically linked into binaries. The usual benefits of dynamic libraries are upgrades (whether for security or functionality) and decreased memory usage. Rust’s lack of a stable binary interface and usage of cross-crate information flow prevents upgrading libraries without upgrading all dependent code. Even when the same crate is used by two different programs on the system, it is unlikely to be provided by the same shared object4 due to the precision with which Rust identifies its crates. This makes Rust binaries more portable but also results in larger disk and memory footprints. >> This is problematic for Android devices where resources like memory and disk usage must be carefully managed because statically linking all crates into Rust binaries would result in excessive code duplication (especially in the standard library). However, our situation is also different from the standard host environment: we build Android using global decisions about dependencies. This means that nearly every crate is shareable between all users of that crate. Thus, we opt to link crates dynamically by default for device targets. This reduces the overall memory footprint of Rust in Android by allowing crates to be reused across multiple binaries which depend on them. >> Since this is unusual in the Rust community, not all third-party crates support dynamic compilation. Sometimes we must carry small patches while we work with upstream maintainers to add support. Having a dynamic linking option for Rust would be a tremendous improvement. It would help to solve vendoring issues and hopefully improve interop with existing systems and programming languages. Good dynamic linking support and enhanced interop with C would make it much easier to do a piece-wise replacement of C components with Rust components, allowing for a gradual migration in places where it makes sense to do so.
esarbe 5y ago
That's very exciting. Rust really seems to establish itself as a serious contender for the low-level niche that until so far has almost exclusively been occupied by C. I wonder how much of an obstacle the lack of the stable ABI and the inability to transparently use C headers still are.
flakiness 5y ago
One good thing about Android in this perspective is that many sub-components of it have their own processes (binary), especially around HAL, after Project Treble [1]. This means that you can rely on the process boundary as the language boundary to start small. Also note that Android is more C++ than C. And C++-Rust integration is something Rust has already done in Firefox. The road is paved, at least partially. So I have high hope here in long term. [1] https://android-developers.googleblog.com/2018/05/faster-adoption-with-project-treble.html https://android-developers.googleblog.com/2018/05/faster-ado...
- bsder 5y ago
  > And C++-Rust integration is something Rust has already done in Firefox. I don't think this is really true. The simple fact that Rust can't overload functions causes significant grief in a C++/Rust interface, for example. Personally, I now see C++ as an evolutionary dead end--yes, it's going to continue to exist for a bit until we get a language that is ready for game programmers can use. After that, the fall of C++ will be fast. This is especially true since we are now back to lots of projects side-stepping the C++ standard libraries. If you're going to do that, why use C++ in the first place? The simplicity of C is much better suited to being a lingua franca, and much better languages are appearing that can finally be the "Successor to C".
  - pjmlp 5y ago
    If only, https://www.khronos.org/sycl/ https://www.khronos.org/sycl/ https://developer.nvidia.com/cuda-zone https://developer.nvidia.com/cuda-zone https://devblogs.microsoft.com/directx/ https://devblogs.microsoft.com/directx/ https://developer.apple.com/metal/ https://developer.apple.com/metal/ https://os.mbed.com/ https://os.mbed.com/ https://www.arduino.cc/ https://www.arduino.cc/ https://www.unrealengine.com/en-US/ https://www.unrealengine.com/en-US/ https://www.crytek.com/cryengine https://www.crytek.com/cryengine https://www.misra.org.uk/Activities/MISRAC/tabid/171/Default.aspx https://www.misra.org.uk/Activities/MISRAC/tabid/171/Default... https://www.snsystems.com/ https://www.snsystems.com/ https://www.codeplay.com/ https://www.codeplay.com/ https://docs.microsoft.com/en-us/windows/apps/winui/ https://docs.microsoft.com/en-us/windows/apps/winui/ https://www.qt.io/ https://www.qt.io/ This is the kind of tooling and ecosystems that Rust still needs to build, besides a plain compiler + cargo.
mleonhard 5y ago
I wish Google would contribute fixes to the problems instead of working around them. Specifically, they could add Cargo incremental builds, reproducible build mode, and build script sandboxing (run build.rs inside WASM).

92 points by theafh 5y ago

This will be downvoted, but I'm a bit skeptical of the "integrate Rust into big C projects" trend. It seems to me that it adds a lot of complexity: now there are two toolchains you must understand in order to contribute. There will invariably be subtle bugs at the interface between the two languages. Yes, memory and type-safety are wonderful, but they are not the be all and end all.

gregwebs 5y ago
Does there need to be subtle bugs at the interface if Rust exposes the interface as a C ABI? I think the idea of this article is that you don’t need to understand two tool chains but instead can focus on the harder task of understanding two languages.
- Const-me 5y ago
  Yes. Rust does not consume C headers. For every function and data structure of the API surface you gonna need two language projections, C header and Rust source file. For large API surfaces, especially when both sides of the FFI evolve over time, bugs are inevitable. The problem is not specific to Rust. It affects all languages, except C++ and Objective-C which are backward compatible.
  - steveklabnik 5y ago
    While Rust itself may not directly, tools like bindgen take in those headers and produce the needed glue. https://cxx.rs/ https://cxx.rs/ is another example of a tool that's used in these sorts of cases. You do not have to write these things, nor keep track of changes, by hand.
    - Const-me 5y ago
      I wonder why so many Rust FFI wrappers don’t use these tools? The most popular C library in the world is probably the standard one. Yet in https://github.com/rust-lang/libc https://github.com/rust-lang/libc I don’t see any glue automatically generated from C headers, everything’s written manually. Other examples: https://github.com/servo/rust-png/ https://github.com/servo/rust-png/ https://github.com/bozaro/lz4-rs https://github.com/bozaro/lz4-rs
      - jcranmer 5y ago
        The libc crate's test does actually parse the C header files to generate tests for conformance. There are a few big reasons to consider avoiding parsing the C headers directly: * The C headers might use features in C that are not compatible with the C parser or Rust ABI itself. (C unions used to be a big headache, and bitfields probably still are.). Manually specifying only what you want to import avoids that issue entirely. * As an extended note to the above, many libraries (particularly your system's implementation of libc) are actually not written in any standard C dialect but instead have a lot of compiler-specific things tacked onto it that make it even more difficult to parse and handle correctly than regular C/C++. * Parsing the C headers requires having them installed and being able to find them correctly. When you start dealing with cross-compilation toolchains, this can be a major hassle.
      - steveklabnik 5y ago
        The first two are older than these tools, not sure about lz4. Not 100% sure why but I would imagine that’s at least a reason.
  - JoshTriplett 5y ago
    > Rust does not consume C headers. Yet, at least. I hope one day it can, transparently.
OmarShehata 5y ago
The reason I'm optimistic about this trend is that it's been a common thing in game development for a long time (low level language for execution speed, high level scripting language for development speed) and it works really well. Having to maintain this interface between the language does come with its own set of problems but is well worth the tradeoff especially if you can spend most of your time in one language or the other.
- bosswipe 5y ago
  That's not what's happening here (Android already has low level C and high level Java). What's happening here is two low level languages side by side.
pcwalton 5y ago
Sometimes complexity is worth it for the engineering benefits. We could all be running our software on beautifully simple vintage 1980s microcontrollers: the 6502 was an incredibly elegant and simple design! But then we'd be stuck with computers that can't multiply in hardware or add two floating point numbers or prevent a misbehaving application from taking down the whole system.
pjmlp 5y ago
Unless one starts with another CFront, naturally moving away from how C compilers see the world has some cost, no matter the language.
wyldfire 5y ago
You have good reason to be wary, it is added complexity that should not be taken lightly. The reason to pay this cost is that the alternatives are: (1) do nothing: keep developing solely in C, (2) rewrite the big C project in Rust in its entirety. The cost of option (1) is significant: memory safety errors contribute to 70% of defects not detected by test [1]. Even with static+dynamic checking tools, developers keep introducing bugs into C projects that aren't possible in safe Rust [2]. The cost of option (2) is significant risk. Big C projects have grown over time and have many implicit requirements already satisfied. Discovering all of those incidental bugs and features that are necessary takes a long time and scales with the size of the project. [1] https://msrc-blog.microsoft.com/2019/07/16/a-proactive-approach-to-more-secure-code/ https://msrc-blog.microsoft.com/2019/07/16/a-proactive-appro... [2] http://blog.vmsplice.net/2020/08/why-qemu-should-move-from-c-to-rust.html http://blog.vmsplice.net/2020/08/why-qemu-should-move-from-c...
- kjs3 5y ago
  I don't think the only options are "rewrite in Rust" or "don't". There are other options than Rust.
  - bestouff 5y ago
    Not with the same performance and memory safety guarantees.
    - kjs3 5y ago
      Prove it. This is going to come as a wild shock to you, but Rust isn't the only language that provides this. There have, in fact, been memory safe, performant languages for decades. But we do live it a bubble here where the norm is anything that hasn't been hyped by a pack of fanbois on HN the last 2 years doesn't exist and "an untruth repeated enough times...".
      - steveklabnik 5y ago
        It’s generally expected that the person making the positive claim demonstrate it. You’re asking them to prove something they don’t think exists. (I have some ideas of what you may say but am curious! You clearly have strong feelings here.)
zaphar 5y ago
I would perhaps argue, especially for projects like Android, that memory safety should be considered table stakes going forward. The class of risks that are mitigated or eliminated by memory safety are well worth the effort. There isn't a great excuse to not prioritize memory safety when it's an option.
jchw 5y ago
If the project is a pure C project using a build system that can't integrate Rust very easily, or it would come at a great cost, then I can see this being a totally valid concern. However, AOSP is massive. It is HUGE. And complicated to build. I really, really doubt this addition will make a huge deal. This is the same for many of the huge projects adopting Rust. The worst one so far is the Linux kernel IMO, because at least that is mostly C and GNU Make. And yet even there, that still isn't that bad. Also, as far as bugs at the interface, I'm pretty sure it's a wash. C and Rust are ABI compatible to the extent that matters. The "subtle bugs" are most likely going to be ones that could've existed without the additional complexity of a new language and an interface between them. It's no worse than say, JNI, which doesn't seem to be a serious issue. I'm pretty sure the benefits of Rust can outweigh this cost. As long as you can build the project, you don't really need to understand 2 toolchains to contribute unless your contribution lands across a boundary between 2 toolchains. On a lot of really large projects this may already be the case anyways.
dang 5y ago
Please don't downvote-bait. It's tedious and against the site guidelines. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
throwaway894345 5y ago
I think it makes the most sense as part of a strategy to incrementally migrate a codebase from C to Rust, or else if there is a clean separation between components such that people are working in one ecosystem or the other but rarely both. In whichever case, we should be careful not to overstate the complexity of integrating Rust and C. Specifically, bringing in any given C dependency means integrating its bespoke build system (and those of each transitive dependency) with that of your project (or more likely, you punt on reproducible builds and instead push the dependency hell onto every contributor) which doesn't strike me as considerably less work than integrating the Rust toolchain (and since Rust has sane, standard build/dependency tooling, it's pretty much a one-time cost rather than a once-per-dependency cost).

thesuperbigfrog 5y ago

>> Dynamic linkage by default >> By default, the Rust ecosystem assumes that crates will be statically linked into binaries. The usual benefits of dynamic libraries are upgrades (whether for security or functionality) and decreased memory usage. Rust’s lack of a stable binary interface and usage of cross-crate information flow prevents upgrading libraries without upgrading all dependent code. Even when the same crate is used by two different programs on the system, it is unlikely to be provided by the same shared object4 due to the precision with which Rust identifies its crates. This makes Rust binaries more portable but also results in larger disk and memory footprints. >> This is problematic for Android devices where resources like memory and disk usage must be carefully managed because statically linking all crates into Rust binaries would result in excessive code duplication (especially in the standard library). However, our situation is also different from the standard host environment: we build Android using global decisions about dependencies. This means that nearly every crate is shareable between all users of that crate. Thus, we opt to link crates dynamically by default for device targets. This reduces the overall memory footprint of Rust in Android by allowing crates to be reused across multiple binaries which depend on them. >> Since this is unusual in the Rust community, not all third-party crates support dynamic compilation. Sometimes we must carry small patches while we work with upstream maintainers to add support. Having a dynamic linking option for Rust would be a tremendous improvement. It would help to solve vendoring issues and hopefully improve interop with existing systems and programming languages. Good dynamic linking support and enhanced interop with C would make it much easier to do a piece-wise replacement of C components with Rust components, allowing for a gradual migration in places where it makes sense to do so.

esarbe 5y ago

That's very exciting. Rust really seems to establish itself as a serious contender for the low-level niche that until so far has almost exclusively been occupied by C. I wonder how much of an obstacle the lack of the stable ABI and the inability to transparently use C headers still are.

flakiness 5y ago

One good thing about Android in this perspective is that many sub-components of it have their own processes (binary), especially around HAL, after Project Treble [1]. This means that you can rely on the process boundary as the language boundary to start small. Also note that Android is more C++ than C. And C++-Rust integration is something Rust has already done in Firefox. The road is paved, at least partially. So I have high hope here in long term. [1] https://android-developers.googleblog.com/2018/05/faster-adoption-with-project-treble.html https://android-developers.googleblog.com/2018/05/faster-ado...

bsder 5y ago
> And C++-Rust integration is something Rust has already done in Firefox. I don't think this is really true. The simple fact that Rust can't overload functions causes significant grief in a C++/Rust interface, for example. Personally, I now see C++ as an evolutionary dead end--yes, it's going to continue to exist for a bit until we get a language that is ready for game programmers can use. After that, the fall of C++ will be fast. This is especially true since we are now back to lots of projects side-stepping the C++ standard libraries. If you're going to do that, why use C++ in the first place? The simplicity of C is much better suited to being a lingua franca, and much better languages are appearing that can finally be the "Successor to C".
- pjmlp 5y ago
  If only, https://www.khronos.org/sycl/ https://www.khronos.org/sycl/ https://developer.nvidia.com/cuda-zone https://developer.nvidia.com/cuda-zone https://devblogs.microsoft.com/directx/ https://devblogs.microsoft.com/directx/ https://developer.apple.com/metal/ https://developer.apple.com/metal/ https://os.mbed.com/ https://os.mbed.com/ https://www.arduino.cc/ https://www.arduino.cc/ https://www.unrealengine.com/en-US/ https://www.unrealengine.com/en-US/ https://www.crytek.com/cryengine https://www.crytek.com/cryengine https://www.misra.org.uk/Activities/MISRAC/tabid/171/Default.aspx https://www.misra.org.uk/Activities/MISRAC/tabid/171/Default... https://www.snsystems.com/ https://www.snsystems.com/ https://www.codeplay.com/ https://www.codeplay.com/ https://docs.microsoft.com/en-us/windows/apps/winui/ https://docs.microsoft.com/en-us/windows/apps/winui/ https://www.qt.io/ https://www.qt.io/ This is the kind of tooling and ecosystems that Rust still needs to build, besides a plain compiler + cargo.

mleonhard 5y ago

I wish Google would contribute fixes to the problems instead of working around them. Specifically, they could add Cargo incremental builds, reproducible build mode, and build script sandboxing (run build.rs inside WASM).