4 ms·

But writing a Rust compiler in Rust is already done.

by remram 2y ago

deleted 2y ago
[deleted]
wyager 2y ago
How do you compile that on a new platform?
- macjohnmcc 2y ago
  Cross compilation.
- projektfu 2y ago
  One way would be to have an intermediate target that is easily recompiled or run on any hardware. https://ziglang.org/news/goodbye-cpp/ https://ziglang.org/news/goodbye-cpp/
  - maxdamantus 2y ago
    But that doesn't conform to the "Descent Principle" described in the article. I haven't really been following Zig, but I still felt slightly disappointed when I learnt that they were just replacing a source-based bootstrapping compiler with a binary blob that someone generated and added to the source tree. The thing that makes me uncomfortable with that approach is that if a certain kind of bug (or virus! [0]) is found in the compiler, it's possible that you have to fix the bug in multiple versions to rebootstrap, in case the bug (or virus!) manages to persist itself into the compilation output. The Dozer article talks about the eventual goal of removing all generated files from the rustc source tree, ie undoing what Zig recently decided to do. If everything is reliably built from source, you can just fix any bugs by editing the current source files. [0] https://wiki.c2.com/?TheKenThompsonHack https://wiki.c2.com/?TheKenThompsonHack
    - projektfu 2y ago
      I think there is too much mysticism here in believing that the bootstrapping phases will offer any particular guarantees. Without essentially a formal proof that the output of the compiler is what you expect, you will have to manually inspect every aspect of every output phase of any bootstrapping process. OK, so you decide to use Compcert C. You now have a proof that your object code is what your C code asked for. Do you have a formal proof of your C code? Have you proved that you have not allowed any surprises? If not, what is your Rust compiler? Junk piled on top of junk, from this standpoint. On the other hand, you could have a verified WASM (or other VM) runner. That verified runner could run the output of a formally verified compiler (which Rustc is not). The trusted base is actually quite small if you had a fully specified language with a verified compiler. But you have to start with that trusted base, and something like a compiler written in C is not really enough to get you there. Oh, and why do we trust QBE?
      - maxdamantus 2y ago
        > Without essentially a formal proof that the output of the compiler is what you expect, you will have to manually inspect every aspect of every output phase of any bootstrapping process. And why would it be easier to manually inspect (prove correct) the output of every phase than to manually inspect (prove correct) the source code? The compiled code will often lose important information about code structure, how abstractions are used, include optimisations, etc. I usually trust my ability to understand source code better than my ability to understand the compiled code.
        projektfu 2y ago
        But you cannot trust the compiler, you said.
        maxdamantus 2y ago
        That's not what I said. I've implied that it's hard to trust the output of some unknown compiler (eg, the "zig1.wasm" blob) and that it's easier to trust source code. The Dozer article explains, under "The Descent Principle", how rustc will eventually be buildable using only source code [0] (other than a "512-byte binary seed" which implements a trivial hex interpreter). You still need to trust a computer to run everything on, though in theory it should be possible to gain trust by running it on multiple computers and checking that the result is the same (this is why any useful system bootstrapping project should also be reproducible [1]). [0] https://github.com/fosslinux/live-bootstrap https://github.com/fosslinux/live-bootstrap [1] https://bootstrappable.org/best-practices.html https://bootstrappable.org/best-practices.html
- remram 2y ago
  Cross-compilation. There is no requirement of being able to run the compiler on the platform to compile for that platform. It is much easier to add support for a platform to the compiler backend than to write a new, full compiler with its own, new bootstrapping method.
kelnos 2y ago
Sure, but the small compiler that you write in C can't compile rustc. So you write a new Rust compiler that uses much simpler Rust that the small compiler in C can compile. Then that new Rust compiler can compile rustc. And since that new Rust compiler might not have much of an optimizer (if it even has one at all), then you recompile rustc with your just-compiled rustc.
- remram 2y ago
  No that makes no sense to me. Or are we pretending cross-compilation doesn't exist?
  - returningfory2 2y ago
    The post is about solving a specific same-architecture bootstrapping problem. Cross-compilation is irrelevant to this discussion.
  - jdbdbebe 2y ago
    There are two kinds of bootstrapping: * Bootstrapping a language on a new architecture using an existing architecture. With modern compilers this is usually done using cross compilation * Bootstrapping a language on an architecture _without_ using another architecture The latter is mostly done for theoretical purposes like reproducibility like reflections on trusting trust
  - RandomThoughts3 2y ago
    Yes, they are because they want to target systems which explicitely disallow cross-compilation like Debian. Yes, I think it's silly too but other people disagree and they are free to work on whatever they want. Do I think it's a mostly pointless waste of time? Obviously, I do. Still, I guess there are worst ones. Note that the Rust project does use cross-compilation for the ports it supports itself and considering the amount of time they use features only available in the current version of rustc in rustc, I guess it's safe to assume they share my opinion on the usefulness of keeping Rust bootstrappable.
  - starkrights 2y ago
bastawhiz 2y ago
Paring down a rust compiler in rust to only use a subset of rust features might not be a big lift. Then you only need to build a rust compiler (in C) that implements the features used by the pared-down rust compiler rather than the full language. Pypy, for instance, implements RPython, which is a valid subset of Python. The compiler is written in RPython. The compiler code is limited on features, but it only needs to implement what RPython includes.

But writing a Rust compiler in Rust is already done.

by remram 2y ago

But writing a Rust compiler in Rust is already done.

deleted 2y ago

[deleted]

wyager 2y ago

How do you compile that on a new platform?

macjohnmcc 2y ago
Cross compilation.
projektfu 2y ago
One way would be to have an intermediate target that is easily recompiled or run on any hardware. https://ziglang.org/news/goodbye-cpp/ https://ziglang.org/news/goodbye-cpp/
- maxdamantus 2y ago
  But that doesn't conform to the "Descent Principle" described in the article. I haven't really been following Zig, but I still felt slightly disappointed when I learnt that they were just replacing a source-based bootstrapping compiler with a binary blob that someone generated and added to the source tree. The thing that makes me uncomfortable with that approach is that if a certain kind of bug (or virus! [0]) is found in the compiler, it's possible that you have to fix the bug in multiple versions to rebootstrap, in case the bug (or virus!) manages to persist itself into the compilation output. The Dozer article talks about the eventual goal of removing all generated files from the rustc source tree, ie undoing what Zig recently decided to do. If everything is reliably built from source, you can just fix any bugs by editing the current source files. [0] https://wiki.c2.com/?TheKenThompsonHack https://wiki.c2.com/?TheKenThompsonHack
  - projektfu 2y ago
    I think there is too much mysticism here in believing that the bootstrapping phases will offer any particular guarantees. Without essentially a formal proof that the output of the compiler is what you expect, you will have to manually inspect every aspect of every output phase of any bootstrapping process. OK, so you decide to use Compcert C. You now have a proof that your object code is what your C code asked for. Do you have a formal proof of your C code? Have you proved that you have not allowed any surprises? If not, what is your Rust compiler? Junk piled on top of junk, from this standpoint. On the other hand, you could have a verified WASM (or other VM) runner. That verified runner could run the output of a formally verified compiler (which Rustc is not). The trusted base is actually quite small if you had a fully specified language with a verified compiler. But you have to start with that trusted base, and something like a compiler written in C is not really enough to get you there. Oh, and why do we trust QBE?
    - maxdamantus 2y ago
      > Without essentially a formal proof that the output of the compiler is what you expect, you will have to manually inspect every aspect of every output phase of any bootstrapping process. And why would it be easier to manually inspect (prove correct) the output of every phase than to manually inspect (prove correct) the source code? The compiled code will often lose important information about code structure, how abstractions are used, include optimisations, etc. I usually trust my ability to understand source code better than my ability to understand the compiled code.
      - projektfu 2y ago
        But you cannot trust the compiler, you said.
        maxdamantus 2y ago
        That's not what I said. I've implied that it's hard to trust the output of some unknown compiler (eg, the "zig1.wasm" blob) and that it's easier to trust source code. The Dozer article explains, under "The Descent Principle", how rustc will eventually be buildable using only source code [0] (other than a "512-byte binary seed" which implements a trivial hex interpreter). You still need to trust a computer to run everything on, though in theory it should be possible to gain trust by running it on multiple computers and checking that the result is the same (this is why any useful system bootstrapping project should also be reproducible [1]). [0] https://github.com/fosslinux/live-bootstrap https://github.com/fosslinux/live-bootstrap [1] https://bootstrappable.org/best-practices.html https://bootstrappable.org/best-practices.html
remram 2y ago
Cross-compilation. There is no requirement of being able to run the compiler on the platform to compile for that platform. It is much easier to add support for a platform to the compiler backend than to write a new, full compiler with its own, new bootstrapping method.

kelnos 2y ago

Sure, but the small compiler that you write in C can't compile rustc. So you write a new Rust compiler that uses much simpler Rust that the small compiler in C can compile. Then that new Rust compiler can compile rustc. And since that new Rust compiler might not have much of an optimizer (if it even has one at all), then you recompile rustc with your just-compiled rustc.

remram 2y ago
No that makes no sense to me. Or are we pretending cross-compilation doesn't exist?
- returningfory2 2y ago
  The post is about solving a specific same-architecture bootstrapping problem. Cross-compilation is irrelevant to this discussion.
- jdbdbebe 2y ago
  There are two kinds of bootstrapping: * Bootstrapping a language on a new architecture using an existing architecture. With modern compilers this is usually done using cross compilation * Bootstrapping a language on an architecture _without_ using another architecture The latter is mostly done for theoretical purposes like reproducibility like reflections on trusting trust
- RandomThoughts3 2y ago
  Yes, they are because they want to target systems which explicitely disallow cross-compilation like Debian. Yes, I think it's silly too but other people disagree and they are free to work on whatever they want. Do I think it's a mostly pointless waste of time? Obviously, I do. Still, I guess there are worst ones. Note that the Rust project does use cross-compilation for the ports it supports itself and considering the amount of time they use features only available in the current version of rustc in rustc, I guess it's safe to assume they share my opinion on the usefulness of keeping Rust bootstrappable.
- starkrights 2y ago

bastawhiz 2y ago

Paring down a rust compiler in rust to only use a subset of rust features might not be a big lift. Then you only need to build a rust compiler (in C) that implements the features used by the pared-down rust compiler rather than the full language. Pypy, for instance, implements RPython, which is a valid subset of Python. The compiler is written in RPython. The compiler code is limited on features, but it only needs to implement what RPython includes.