5 ms·

Can't you just cross compile? Wouldn't you have to do that anyway for some parts of the system? What's the point of bootstrapping?

monocasa 6y ago
When I say "systems", I should have been a little more specific. There's a bunch of corporate/.gov envs that don't allow binaries to enter their system. All new code has to be compiled from source, on the target system (or at least on that side of the process firewall, it might be an airgapped cluster that can share code). They have C and C++ compilers that they've been compiling from source since the dawn of time, but they don't have a rust compiler. This gives them a mechanism to begin using rust within their process.
- dataflow 6y ago
  Wow I see. Could they not use the initial Rust compilers (and however those were bootstrapped? I have no idea how) to get a Rust compiler going? I'm guessing it's just too painful?
  - spijdar 6y ago
    Bootstrapping rust is really, really painful as is. The rust compiler itself tends to require a recent version of the rust compiler to work. So to bootstrap from the original versions of rustc (which were written in OCaml) you'd have to incrementally step up really slowly. Being able to target a pretty recent version of Rust with a compiler written in C would be so, so useful for these purposes.
    - Blikkentrekker 6y ago
      Certainly that could be automated quite easily. I would assume that on such compilation farms that these systems generally use, this could be done very quickly. Perhaps there would be an interest for the Rustc team to provide a recursive automated setup that is capable of compiling the latest Rustc from OCaml, and finally from C as well since one must follow a similar process with OCaml.
      - LegionMammal978 6y ago
        The automation for this is actually trickier than it would first appear: especially with the earlier snapshots, a number of patches are needed to compile the LLVM interfaces written in C++. Also, due to linking issues, some snapshots require a stage2 compiler, while others can only use a stage1 compiler. I agree that it would be trivial once you have the chain in place, but that chain can only be built through trial and error.
        Blikkentrekker 6y ago
        And no one has done this before? Methinks that building this chain by trial and error is rather trivial compared to the actual work in building a compiler.
        sanxiyn 6y ago
        We are discussing mrustc here. The reason it hasn't been done yet is that nearly everyone interested in this agrees mrustc is a better bootstrapping path than starting from the last OCaml version.
        Blikkentrekker 6y ago
        That it's a better bootstrapping process for the user has no relationship with the ease of development for the developer. Typically, they are inversely correlated.
        qqii 6y ago
        Guix performed the bootstrap from mrustc 0.8 for rust 1.19 all the way to rust 1.28 in 2018: https://guix.gnu.org/en/blog/2018/bootstrapping-rust/ https://guix.gnu.org/en/blog/2018/bootstrapping-rust/ Since them they've bootstrapped this all the way to 1.50: https://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/rust.scm#n1454 https://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages... The work done here is non trivial, this chain of versions are required for the final binary to be reproducible.
      - 6y ago
  - monocasa 6y ago
    Yeah, rustc dogfoods new features, so it's _very_ painful. It'd probably be close to 100 or so compilers to go back to the ocaml that it started from.
    - LegionMammal978 6y ago
      I've actually been trying this for real. Right now, I'm at the 82-compiler mark, including the ~6 LLVM versions needed. For a sense of scale, the most recent compiler in this chain is the 2011-12-07 snapshot, which is still older than Rust 0.1 (2012-01-20). In most cases, additional compilers are needed only to resolve syntax errors, but there are occasional LLVM-related segfaults that create most of my headaches.
      - wh33zle 6y ago
        Is your work publicly available somewhere?
        LegionMammal978 6y ago
        The scripts are still pretty messy right now. I'll probably clean them up a bit and upload them once I reach 0.1.
  - shakow 6y ago
    > I'm guessing it's just too painful? Indeed. Although it is being worked on, the whole bootstrap process of rustc is currently a major hassle, requiring to start from the oldest CaML versions of the compiler up to the most recent ones.
  - maccam94 6y ago
    The earliest versions of the rust compiler were written in Ocaml, which is also unlikely to have been built in a trusted way by these organizations. mrustc exists to be a way to bootstrap trusted builds of rustc.
- skybrian 6y ago
  But there are no binaries being copied. They allow hand-written C to be copied, but not machine-generated C?
  - monocasa 6y ago
    Without mrustc you'd need to trust the rustc binary, or build around a hundred versions of rust to go back to a rustc that itself didn't depend on rust.
- CameronNemo 6y ago
  Do all these entities operate independently? Is there no internal web of trust they could use to distribute rebuilt binaries?
  - monocasa 6y ago
    Yeah, they operate independently. No new binaries, period.
    - __d 6y ago
      Which makes sense: compartmentalise. Otherwise, a failure of one domain would compromise all the others. I hope there's some automated assistance for scanning inbound source code too. Imagine reviewing everything, line by line, millions of them.
  - deleted 6y ago
    [deleted]
- Blikkentrekker 6y ago
  How did they get the original C compiler there then without allowing binaries? Did they originally write one in assembler? I remember reading an interesting thought experiment that traced history that investigated what would happen if Dennis Ritchie had put malicious code in the first C compiler that was designed to detect whether the compiler compiled a compiler, and then copied the malicious code into it. It concluded that tracing the history, that if this were to have happened, then GCC and Clang and many other programming languages would have said malicious code in their compilers that do not show up in the source code with none the wiser.
  - monocasa 6y ago
    Epoch creation is covered by a totally different set of process rules, for the reasons you've stated. These rules are mainly intended to make it much harder for someone to gain access to the system if they weren't there when the system initially was blocked off from the world.
  - twic 6y ago
    If understand Dennis Ritchie's explanation [1] correctly, he started with a compiler for the B language, itself written in B, and modified it to accept a somewhat different language, which he called C. Before the B compiler written in B, there was a B compiler written in something else. I couldn't figure out what from that document. [1] https://www.bell-labs.com/usr/dmr/www/chist.html https://www.bell-labs.com/usr/dmr/www/chist.html
- josephg 6y ago
  Couldn’t they run mrustc on the rust source code on another machine to generate C code for rustc, then copy the C code to their internal systems and compile that into a working rust compiler? Or would that not count as “compiling from source”?
  - danappelxx 6y ago
    I believe that’s exactly what they were suggesting :)
- xiphias2 6y ago
  I see, thanks for the explanation, now the project makes a lot of sense :) I guess you didn't want to advertise the real reason of the project.
dan-robertson 6y ago
One way to trust your computer system more is to have very few binary dependencies. Ie the goal is to have some source files and hopefully if you start with (eg) a trusted C compiler and linker you can work your way up to the whole system with deterministic results. This is supposed to be a way to prove that the compiler wasn’t doing anything funky like injecting some vulnerability into the code it compiled and injecting that vulnerability injection into any compiler it compiled.

monocasa 6y ago

When I say "systems", I should have been a little more specific. There's a bunch of corporate/.gov envs that don't allow binaries to enter their system. All new code has to be compiled from source, on the target system (or at least on that side of the process firewall, it might be an airgapped cluster that can share code). They have C and C++ compilers that they've been compiling from source since the dawn of time, but they don't have a rust compiler. This gives them a mechanism to begin using rust within their process.

dataflow 6y ago
Wow I see. Could they not use the initial Rust compilers (and however those were bootstrapped? I have no idea how) to get a Rust compiler going? I'm guessing it's just too painful?
- spijdar 6y ago
  Bootstrapping rust is really, really painful as is. The rust compiler itself tends to require a recent version of the rust compiler to work. So to bootstrap from the original versions of rustc (which were written in OCaml) you'd have to incrementally step up really slowly. Being able to target a pretty recent version of Rust with a compiler written in C would be so, so useful for these purposes.
  - Blikkentrekker 6y ago
    Certainly that could be automated quite easily. I would assume that on such compilation farms that these systems generally use, this could be done very quickly. Perhaps there would be an interest for the Rustc team to provide a recursive automated setup that is capable of compiling the latest Rustc from OCaml, and finally from C as well since one must follow a similar process with OCaml.
    - LegionMammal978 6y ago
      The automation for this is actually trickier than it would first appear: especially with the earlier snapshots, a number of patches are needed to compile the LLVM interfaces written in C++. Also, due to linking issues, some snapshots require a stage2 compiler, while others can only use a stage1 compiler. I agree that it would be trivial once you have the chain in place, but that chain can only be built through trial and error.
      - Blikkentrekker 6y ago
        And no one has done this before? Methinks that building this chain by trial and error is rather trivial compared to the actual work in building a compiler.
        sanxiyn 6y ago
        We are discussing mrustc here. The reason it hasn't been done yet is that nearly everyone interested in this agrees mrustc is a better bootstrapping path than starting from the last OCaml version.
        Blikkentrekker 6y ago
        That it's a better bootstrapping process for the user has no relationship with the ease of development for the developer. Typically, they are inversely correlated.
        qqii 6y ago
        Guix performed the bootstrap from mrustc 0.8 for rust 1.19 all the way to rust 1.28 in 2018: https://guix.gnu.org/en/blog/2018/bootstrapping-rust/ https://guix.gnu.org/en/blog/2018/bootstrapping-rust/ Since them they've bootstrapped this all the way to 1.50: https://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/rust.scm#n1454 https://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages... The work done here is non trivial, this chain of versions are required for the final binary to be reproducible.
    - 6y ago
- monocasa 6y ago
  Yeah, rustc dogfoods new features, so it's _very_ painful. It'd probably be close to 100 or so compilers to go back to the ocaml that it started from.
  - LegionMammal978 6y ago
    I've actually been trying this for real. Right now, I'm at the 82-compiler mark, including the ~6 LLVM versions needed. For a sense of scale, the most recent compiler in this chain is the 2011-12-07 snapshot, which is still older than Rust 0.1 (2012-01-20). In most cases, additional compilers are needed only to resolve syntax errors, but there are occasional LLVM-related segfaults that create most of my headaches.
    - wh33zle 6y ago
      Is your work publicly available somewhere?
      - LegionMammal978 6y ago
        The scripts are still pretty messy right now. I'll probably clean them up a bit and upload them once I reach 0.1.
- shakow 6y ago
  > I'm guessing it's just too painful? Indeed. Although it is being worked on, the whole bootstrap process of rustc is currently a major hassle, requiring to start from the oldest CaML versions of the compiler up to the most recent ones.
- maccam94 6y ago
  The earliest versions of the rust compiler were written in Ocaml, which is also unlikely to have been built in a trusted way by these organizations. mrustc exists to be a way to bootstrap trusted builds of rustc.
skybrian 6y ago
But there are no binaries being copied. They allow hand-written C to be copied, but not machine-generated C?
- monocasa 6y ago
  Without mrustc you'd need to trust the rustc binary, or build around a hundred versions of rust to go back to a rustc that itself didn't depend on rust.
CameronNemo 6y ago
Do all these entities operate independently? Is there no internal web of trust they could use to distribute rebuilt binaries?
- monocasa 6y ago
  Yeah, they operate independently. No new binaries, period.
  - __d 6y ago
    Which makes sense: compartmentalise. Otherwise, a failure of one domain would compromise all the others. I hope there's some automated assistance for scanning inbound source code too. Imagine reviewing everything, line by line, millions of them.
- deleted 6y ago
  [deleted]
Blikkentrekker 6y ago
How did they get the original C compiler there then without allowing binaries? Did they originally write one in assembler? I remember reading an interesting thought experiment that traced history that investigated what would happen if Dennis Ritchie had put malicious code in the first C compiler that was designed to detect whether the compiler compiled a compiler, and then copied the malicious code into it. It concluded that tracing the history, that if this were to have happened, then GCC and Clang and many other programming languages would have said malicious code in their compilers that do not show up in the source code with none the wiser.
- monocasa 6y ago
  Epoch creation is covered by a totally different set of process rules, for the reasons you've stated. These rules are mainly intended to make it much harder for someone to gain access to the system if they weren't there when the system initially was blocked off from the world.
- twic 6y ago
  If understand Dennis Ritchie's explanation [1] correctly, he started with a compiler for the B language, itself written in B, and modified it to accept a somewhat different language, which he called C. Before the B compiler written in B, there was a B compiler written in something else. I couldn't figure out what from that document. [1] https://www.bell-labs.com/usr/dmr/www/chist.html https://www.bell-labs.com/usr/dmr/www/chist.html
josephg 6y ago
Couldn’t they run mrustc on the rust source code on another machine to generate C code for rustc, then copy the C code to their internal systems and compile that into a working rust compiler? Or would that not count as “compiling from source”?
- danappelxx 6y ago
  I believe that’s exactly what they were suggesting :)
xiphias2 6y ago
I see, thanks for the explanation, now the project makes a lot of sense :) I guess you didn't want to advertise the real reason of the project.

dan-robertson 6y ago

One way to trust your computer system more is to have very few binary dependencies. Ie the goal is to have some source files and hopefully if you start with (eg) a trusted C compiler and linker you can work your way up to the whole system with deterministic results. This is supposed to be a way to prove that the compiler wasn’t doing anything funky like injecting some vulnerability into the code it compiled and injecting that vulnerability injection into any compiler it compiled.