8 ms·

Bootstrapping Rust

194 points by darnir 8y ago

rusbus 8y ago
The "bootstrapable.org" project that OP refers to is an interesting practical result from the infamous "Reflections on Trusting Trust."[1] If you have the source compiler, which is itself bootstrapped from source, then you effectively sidestep the problem brought up in the paper -- someone sneaks in a boobytrapped compiler somewhere in the process resulting in a chain of tainted compilers. This is the kind of work that seems pretty thankless, but I'm glad someone is doing it. [1] https://www.archive.ece.cmu.edu/~ganger/712.fall02/papers/p761-thompson.pdf https://www.archive.ece.cmu.edu/~ganger/712.fall02/papers/p7...
- WalterGR 8y ago
  If you have the source compiler, which is itself bootstrapped from source, then you effectively sidestep the problem brought up in the paper How does this work? Is the source compiler bootstrapped from its source, such that the source compiler compiles the source of the source compiler and that results in the source compiler? I think I’m getting confused because “source compiler” describes what a compiler does anyway, so I’m guessing that “source” in “source compiler” is used as in origin and not as in textual code, rather than a redundancy? So which source is the source compiler compiling?
  - coldtea 8y ago
    The source compiler could be in assembler, for one.
    - WalterGR 8y ago
      Compiling or assembling, how does this work around the problem described in Reflections on Trusting Trust?
      - Cyph0n 8y ago
        I don’t see it either. The paper specifically talked about modifying the source of the “root” compiler, and then using it to compile the system compiler. This can be generalized to N compilers, so to truly claim that your compiler is safe, you’ll need to verify the source of all compilers in the chain. I’m not an expert, so please correct me if I’m off here!
        coldtea 8y ago
        Assembler is pretty trivial to translate to machine code (and can be even written so that it's 1 to 1 per line). Theoretically (not very practical due to size) you could translate the compiler text that you see to machine executable code yourself, looking at some opcode reference.
        WalterGR 8y ago
        While that’s true, I don’t believe that’s the approach we’re discussing here.
        WalterGR 8y ago
        Apologies, maybe that is the approach we’re talking about here, and I’ve simply confused myself. :) OP: If you have the source compiler, which is itself bootstrapped from source, then you effectively sidestep the problem brought up in the paper. So does that translate to English as: You have the source code for a compiler. You’ve ensured that the source code isn’t tainted in any way. If you translate that source code into an executable binary by hand, then you know that binary isn’t tainted. And subsequently, that binary won’t taint its output. ? Hmm. That just doesn’t seem interesting. And, as you say, “Theoretically... you could...”. Emphasis mine...
        WalterGR 8y ago
        I think I’ve resolved my confusion (see my updated response to coldtea.) I think you’re 100% correct. This result just doesn’t seem interesting. Not only do you have to verify the source of all compilers in the chain, you need to ensure that - when that code is compiled - it’s the exact source that was verified. And even then, how do you know that the disk firmware actually gives you that non-tainted compiler when you ask for it? But: I guess it’s turtles all the way down. This at least theoretically improves how much of the chain (from atoms on a disk to the binaries produced by your Nth compiler) is non-tainted.
- colejohnson66 8y ago
  “someone sneaks in a boobytrapped compiler somewhere...” Especially one that spits out a white supremacy message every once in a while: https://www.quora.com/What-is-a-coders-worst-nightmare/answer/Mick-Stute https://www.quora.com/What-is-a-coders-worst-nightmare/answe...
  - unhammer 8y ago
    Wow. But it sounds almost too good to be true; anyone know if it's for real?
    - Drdrdrq 8y ago
      I have no idea if it's true, but it is really not that difficult to achieve in a setting like this, where binaries are compiled from the sources that are available on the system itself. Even creating a compiler that poisons itself is not that difficult. It is the idea itself which is genius, and (unfortunately) completely doable.
- WalterGR 8y ago
  The "bootstrapable.org" project that OP refers to... “Strappable” has 2 P’s... Link: https://bootstrappable.org/ https://bootstrappable.org/
ximeng 8y ago
So bootstrap chain here is g++->mrustc->several iterations of rust. (Rather than original bootstrap chain via ocaml.) Bootstrap for g++ is presumably something like machine code->asm->c->g++. And the overall goal is something like shortening or simplifying the chain from machine code to rust compiler. Ideally I suppose this would be something like machine code->proto rust->rust compiler. Haskell seems to have a relatively good pipeline, with clean division between core and non-core. https://ghc.haskell.org/trac/ghc/wiki/Commentary https://ghc.haskell.org/trac/ghc/wiki/Commentary
- reirob 8y ago
  I was just yesterday reading that bootstrapping Haskell seems to have unanswered questions too: https://www.reddit.com/r/haskell/comments/a600zp/thoughts_on_bootstrapping_ghc/ https://www.reddit.com/r/haskell/comments/a600zp/thoughts_on...
- clort 8y ago
  The nice(r) thing about bootstrapping GCC is that there are many alternative implementations of C and C++ already existing.
  - ploxiln 8y ago
    And you can build GCC-8 with GCC-4.8, whereas rust seems to require at least the previous version of rust. It would be reasonable for rust 1.19 to be able to build the latest release for the next couple of years ... rust 1.19 is less than 18 months old! Surely rust was an OK language 18 months ago?
    - steveklabnik 8y ago
      Rust’s standard library makes heavy use of unstable features that may change from release to release. We’ve been slowing down the requirements over time; the current policy is that you can build any stable with the previous release. Maybe someday we’ll slow it down further, but there’s no plans to do so any time soon.
    - 8y ago
yoklov 8y ago
> There are plans to extend mrustc to support newer Rust, but it turned out to be difficult. Was some feature added to rust 1.20.0 that was particularly difficult to implement? Or is this just a 'have to stop somewhere' situation.
- steveklabnik 8y ago
  My understanding is that the goal was to break the bootstrap chain, and so once that was done, there wasn’t a ton of reasons to keep working on it. mrustc is effectively written entirely by one person: https://github.com/thepowersgang/mrustc/graphs/contributors https://github.com/thepowersgang/mrustc/graphs/contributors It's extremely impressive, but I can also understand why trying to keep up doesn't make a ton of sense.
  - Twirrim 8y ago
    Presumably the drawback here is that rust is going to take an increasing amount of time to bootstrap on these platforms. You're releasing about 8 a year (excluding bug fix releases), each of which is necessary to compile the next one, and so on down the line? That sounds like it's going to get extremely nasty, really quickly.
    - mmastrac 8y ago
      Once you've bootstrapped a single platform a certain number of times with the exact same compiler output through a diverse-enough set of toolchains, you can start to trust that the entire toolchain (including hardware) is secure and simply release hashes of those diversely-compiled binaries as trusted roots for faster rebuilds (ie: nightlies).
    - steveklabnik 8y ago
      That’s assuming that someone wants to fully bootstrap. Only a very, very small number of people actually do this, and since this is all about trust anyway, it all depends on your level of paranoia. mrustc has already bootstrapped a byte-identical rustc from the mainline, that in and of itself is good enough for many. Even Debian didn’t do a full bootstrap from the OCaml days. It’s all about trust. If you want to be mega paranoid, then yeah it’s gonna be a lot of work. But that always is. This whole thing is only an issue for a very small number of people. Those people are important! But it’s a tradeoff, like everything.
deleted 8y ago
[deleted]
deleted 8y ago
[deleted]
andrewchambers 8y ago
writing mrustc in C++ seems like such a mistake. It compiles rust to C, so why not write it in rust? then compile itself to C. Another idea, just compile rustc to web assembly then use https://github.com/WebAssembly/wabt/tree/master/wasm2c https://github.com/WebAssembly/wabt/tree/master/wasm2c to convert it to your bootstrap source.
- nine_k 8y ago
  > why not. write it in rust? Exactly to avoid having any rust compilers in the chain.
  - mrob 8y ago
    There are many Rust compilers in the bootstrap chain. The point is to make every step in the chain human-readable. Auto-generated C is not "source code" in the sense of "the preferred form of the work for making modifications to it" (the GPL's definition of source code). A malicious code generator could hide a trusting trust attack in the generated code in such a way that it would be difficult to find. True source code is easier to audit.
unhammer 8y ago
https://dwheeler.com/trusting-trust/ https://dwheeler.com/trusting-trust/ is the page on Diverse Double-Compiling as a counter to the trusting trust attack

194 points by darnir 8y ago

The "bootstrapable.org" project that OP refers to is an interesting practical result from the infamous "Reflections on Trusting Trust."[1] If you have the source compiler, which is itself bootstrapped from source, then you effectively sidestep the problem brought up in the paper -- someone sneaks in a boobytrapped compiler somewhere in the process resulting in a chain of tainted compilers. This is the kind of work that seems pretty thankless, but I'm glad someone is doing it. [1] https://www.archive.ece.cmu.edu/~ganger/712.fall02/papers/p761-thompson.pdf https://www.archive.ece.cmu.edu/~ganger/712.fall02/papers/p7...

WalterGR 8y ago
If you have the source compiler, which is itself bootstrapped from source, then you effectively sidestep the problem brought up in the paper How does this work? Is the source compiler bootstrapped from its source, such that the source compiler compiles the source of the source compiler and that results in the source compiler? I think I’m getting confused because “source compiler” describes what a compiler does anyway, so I’m guessing that “source” in “source compiler” is used as in origin and not as in textual code, rather than a redundancy? So which source is the source compiler compiling?
- coldtea 8y ago
  The source compiler could be in assembler, for one.
  - WalterGR 8y ago
    Compiling or assembling, how does this work around the problem described in Reflections on Trusting Trust?
    - Cyph0n 8y ago
      I don’t see it either. The paper specifically talked about modifying the source of the “root” compiler, and then using it to compile the system compiler. This can be generalized to N compilers, so to truly claim that your compiler is safe, you’ll need to verify the source of all compilers in the chain. I’m not an expert, so please correct me if I’m off here!
      - coldtea 8y ago
        Assembler is pretty trivial to translate to machine code (and can be even written so that it's 1 to 1 per line). Theoretically (not very practical due to size) you could translate the compiler text that you see to machine executable code yourself, looking at some opcode reference.
        WalterGR 8y ago
        While that’s true, I don’t believe that’s the approach we’re discussing here.
        WalterGR 8y ago
        Apologies, maybe that is the approach we’re talking about here, and I’ve simply confused myself. :) OP: If you have the source compiler, which is itself bootstrapped from source, then you effectively sidestep the problem brought up in the paper. So does that translate to English as: You have the source code for a compiler. You’ve ensured that the source code isn’t tainted in any way. If you translate that source code into an executable binary by hand, then you know that binary isn’t tainted. And subsequently, that binary won’t taint its output. ? Hmm. That just doesn’t seem interesting. And, as you say, “Theoretically... you could...”. Emphasis mine...
      - WalterGR 8y ago
        I think I’ve resolved my confusion (see my updated response to coldtea.) I think you’re 100% correct. This result just doesn’t seem interesting. Not only do you have to verify the source of all compilers in the chain, you need to ensure that - when that code is compiled - it’s the exact source that was verified. And even then, how do you know that the disk firmware actually gives you that non-tainted compiler when you ask for it? But: I guess it’s turtles all the way down. This at least theoretically improves how much of the chain (from atoms on a disk to the binaries produced by your Nth compiler) is non-tainted.
colejohnson66 8y ago
“someone sneaks in a boobytrapped compiler somewhere...” Especially one that spits out a white supremacy message every once in a while: https://www.quora.com/What-is-a-coders-worst-nightmare/answer/Mick-Stute https://www.quora.com/What-is-a-coders-worst-nightmare/answe...
- unhammer 8y ago
  Wow. But it sounds almost too good to be true; anyone know if it's for real?
  - Drdrdrq 8y ago
    I have no idea if it's true, but it is really not that difficult to achieve in a setting like this, where binaries are compiled from the sources that are available on the system itself. Even creating a compiler that poisons itself is not that difficult. It is the idea itself which is genius, and (unfortunately) completely doable.
WalterGR 8y ago
The "bootstrapable.org" project that OP refers to... “Strappable” has 2 P’s... Link: https://bootstrappable.org/ https://bootstrappable.org/

ximeng 8y ago

So bootstrap chain here is g++->mrustc->several iterations of rust. (Rather than original bootstrap chain via ocaml.) Bootstrap for g++ is presumably something like machine code->asm->c->g++. And the overall goal is something like shortening or simplifying the chain from machine code to rust compiler. Ideally I suppose this would be something like machine code->proto rust->rust compiler. Haskell seems to have a relatively good pipeline, with clean division between core and non-core. https://ghc.haskell.org/trac/ghc/wiki/Commentary https://ghc.haskell.org/trac/ghc/wiki/Commentary

reirob 8y ago
I was just yesterday reading that bootstrapping Haskell seems to have unanswered questions too: https://www.reddit.com/r/haskell/comments/a600zp/thoughts_on_bootstrapping_ghc/ https://www.reddit.com/r/haskell/comments/a600zp/thoughts_on...
clort 8y ago
The nice(r) thing about bootstrapping GCC is that there are many alternative implementations of C and C++ already existing.
- ploxiln 8y ago
  And you can build GCC-8 with GCC-4.8, whereas rust seems to require at least the previous version of rust. It would be reasonable for rust 1.19 to be able to build the latest release for the next couple of years ... rust 1.19 is less than 18 months old! Surely rust was an OK language 18 months ago?
  - steveklabnik 8y ago
    Rust’s standard library makes heavy use of unstable features that may change from release to release. We’ve been slowing down the requirements over time; the current policy is that you can build any stable with the previous release. Maybe someday we’ll slow it down further, but there’s no plans to do so any time soon.
  - 8y ago

yoklov 8y ago

> There are plans to extend mrustc to support newer Rust, but it turned out to be difficult. Was some feature added to rust 1.20.0 that was particularly difficult to implement? Or is this just a 'have to stop somewhere' situation.

steveklabnik 8y ago
My understanding is that the goal was to break the bootstrap chain, and so once that was done, there wasn’t a ton of reasons to keep working on it. mrustc is effectively written entirely by one person: https://github.com/thepowersgang/mrustc/graphs/contributors https://github.com/thepowersgang/mrustc/graphs/contributors It's extremely impressive, but I can also understand why trying to keep up doesn't make a ton of sense.
- Twirrim 8y ago
  Presumably the drawback here is that rust is going to take an increasing amount of time to bootstrap on these platforms. You're releasing about 8 a year (excluding bug fix releases), each of which is necessary to compile the next one, and so on down the line? That sounds like it's going to get extremely nasty, really quickly.
  - mmastrac 8y ago
    Once you've bootstrapped a single platform a certain number of times with the exact same compiler output through a diverse-enough set of toolchains, you can start to trust that the entire toolchain (including hardware) is secure and simply release hashes of those diversely-compiled binaries as trusted roots for faster rebuilds (ie: nightlies).
  - steveklabnik 8y ago
    That’s assuming that someone wants to fully bootstrap. Only a very, very small number of people actually do this, and since this is all about trust anyway, it all depends on your level of paranoia. mrustc has already bootstrapped a byte-identical rustc from the mainline, that in and of itself is good enough for many. Even Debian didn’t do a full bootstrap from the OCaml days. It’s all about trust. If you want to be mega paranoid, then yeah it’s gonna be a lot of work. But that always is. This whole thing is only an issue for a very small number of people. Those people are important! But it’s a tradeoff, like everything.

deleted 8y ago

[deleted]

deleted 8y ago

[deleted]

andrewchambers 8y ago

writing mrustc in C++ seems like such a mistake. It compiles rust to C, so why not write it in rust? then compile itself to C. Another idea, just compile rustc to web assembly then use https://github.com/WebAssembly/wabt/tree/master/wasm2c https://github.com/WebAssembly/wabt/tree/master/wasm2c to convert it to your bootstrap source.

nine_k 8y ago
> why not. write it in rust? Exactly to avoid having any rust compilers in the chain.
- mrob 8y ago
  There are many Rust compilers in the bootstrap chain. The point is to make every step in the chain human-readable. Auto-generated C is not "source code" in the sense of "the preferred form of the work for making modifications to it" (the GPL's definition of source code). A malicious code generator could hide a trusting trust attack in the generated code in such a way that it would be difficult to find. True source code is easier to audit.

unhammer 8y ago

https://dwheeler.com/trusting-trust/ https://dwheeler.com/trusting-trust/ is the page on Diverse Double-Compiling as a counter to the trusting trust attack