10 ms·

Mrustc: a Rust compiler written in C++

182 points by mabynogy 9y ago

steveklabnik 9y ago
Lots of discussion and questions answered by the author on the Rust subreddit the other day https://www.reddit.com/r/rust/comments/718gbh/mrustc_alternative_rust_compiler_written_in_c/ https://www.reddit.com/r/rust/comments/718gbh/mrustc_alterna...
fyrerise 9y ago
Neat project. It looks like this was largely written by one person, and I'm fairly in awe at anyone who can take a big project like a compiler this far alone. Isn't there a bit of cognitive dissonance in believing that Rust as a language is an important idea (i.e. by the additional code safety and code maintainability that it conveys), but then simultaneously making the effort to rewrite the current Rust-implemented compiler in C++? C++ is fast, but aside from a shared value around performance, it has fairly little in common with the ideas that Rust is built on.
- CUViper 9y ago
  By not self-hosting in Rust, it should be easier to bootstrap on a new target. C++ is ubiquitous -- what other language would you choose?
  - yjftsjthsd-h 9y ago
    If bootstrapping/portability is the goal, probably C.
    - damnstraight 9y ago
      Conveniently there are already C++ compilers bootstrappable with C.
      - tavert 9y ago
        Any that are both open-source and C++11 compliant? Guess you can still build g++-4.6 with just C, then a newer g++ from that, but it's a bit of a pain.
  - fyrerise 9y ago
    > what other language would you choose? As far as I can tell the goal of the project isn't to target more platforms (Rust targets quite a few by way of LLVM), so I don't think I'd choose any other language, including C++. Having a compiler and standard library written in the language that it compiles has some huge benefits for increasing the pool of possible contributors.
    - kibwen 9y ago
      > As far as I can tell the goal of the project isn't to target more platforms (Rust targets quite a few by way of LLVM) There appears to be an attempt to leverage this project to target ESP8266, which AFAIK LLVM does not support: https://github.com/emosenkis/esp-rs https://github.com/emosenkis/esp-rs
      - kbenson 9y ago
        Interesting. Along with with your other comment about the borrow checker, I guess you could develop using (or occasionally check against) the rustc compiler for borrow checker correctness, and deploy using mrustc. That's pretty cool.
    - tatterdemalion 9y ago
      > Having a compiler and standard library written in the language that it compiles has some huge benefits for increasing the pool of possible contributors. But this isn't the official compiler, this is someone's personal project?
      - fyrerise 9y ago
        > But this isn't the official compiler, this is someone's personal project? True, but compilers are complicated machines and Rust is still changing at a fairly frantic rate. The author seems to be doing quite a good job of development today, but if it has any hope of staying current, it probably needs to think about how to increase its bus factor (something happens like changing jobs, starting a family, or they just become interested in something else, and a single person suddenly has less time to contribute).
  - deleted 9y ago
    [deleted]
  - xioxox 9y ago
    If the goal is to break the dependency cycle, a higher level language like Python would make development much easier. C++ is powerful, but not as rapid to develop in.
    - xioxox 9y ago
      Why the downvotes? It's clearly easier to make a compiler in a higher level language (Python is just an example, but Lisps are suited to this kind of thing). For example, text parsing is easier in Perl/Ruby/Python/Swift/etc. As someone who knows C++, more thought is required to do the same thing as in a higher level language, although it runs much faster. If you just wanted to bootstrap the compiler, then you'd choose the easiest route to that. It could also be easier to read and understand than a C++ compiler.
  - pjmlp 9y ago
    Ada, for example. There is an open source version available in every system supported by gcc. However C++17 is quite safe, provided one doesn't follow "write C with C++ compiler" idiom.
    - pcwalton 9y ago
      > However C++17 is quite safe, provided one doesn't follow "write C with C++ compiler" idiom. No, it's not. There is no use-after-free protection among other things. It doesn't matter how much you write code like C or not: it's just not safe.
      - rtpg 9y ago
        safety is a spectrum, I think declaring it unsafe entirely might not be conducive to more interesting points
      - pjmlp 9y ago
        Then better stop using LLVM on Rust. Until then, those comments only annoy those of us that happen to like Rust, but don't find it mature enough to replace C++ on the use cases we happen to care about.
        pcwalton 9y ago
        I didn't say you can't write useful software in C++. I said that C++ software is not, in general, memory safe. Sometimes the benefits of a particular piece of software (for example, having an excellent production-grade optimizer) outweigh its drawbacks (for example, not being memory-safe). I don't think you'd find a single LLVM developer who would claim that LLVM is memory safe. Giving invalid IR to LLVM and not running the verifier frequently segfaults it, for example...
        pjmlp 9y ago
        I do agree with that, but that is exactly why I always mention the "C with C++ compiler" idiom, which is usually followed by your remark.
- mrob 9y ago
  Multiple implementations of a compiler lets you implement the "Diverse Double-Compiling"[0] countermeasure to the famous "Reflections on Trusting Trust"[1] attack. You wouldn't necessarily use the C++ implementation in production, but it still improves the security of the Rust language just by existing. [0] https://www.dwheeler.com/trusting-trust/dissertation/html/wheeler-trusting-trust-ddc.html#0.Abstract https://www.dwheeler.com/trusting-trust/dissertation/html/wh... (previous HN discussion at: https://news.ycombinator.com/item?id=12666923 https://news.ycombinator.com/item?id=12666923 ) [1] http://www.ece.cmu.edu/~ganger/712.fall02/papers/p761-thompson.pdf http://www.ece.cmu.edu/~ganger/712.fall02/papers/p761-thomps...
  - Manishearth 9y ago
    DDC is irrelevant here, DDC is an argument to not write the second compiler in C++ and write it in Rust too. Having a Rust compiler in C++ is a mitigation to the trusting trust attack, period. You don't need DDC for this. DDC is necessary when you have two self hosted compilers (e.g. GCC and clang). Here we have one self-hosted compiler (rustc), and one in another language (C++). To mitigate trusting trust in rustc, use mrustc to compile rustc, and then use that rustc to compile itself, and now you have a trusted binary (provided you trust your C++ compiler. you can fix this by DDCing the C++ compilers)
- santaclaus 9y ago
  The Rust compiler infrastructure is built on LLVM (a C++ project), so this doesn’t seem terribly weird.
- santaclaus 9y ago
  > C++ is fast, but aside from a shared value around performance, it has fairly little in common with the ideas that Rust is built on. The core guidelines are very similar to Rust, in spirit and intent.
  - steveklabnik 9y ago
    ... but the details are very, very different. Unless I've missed significant revisions, data races and concurrency are a non-goal of the Core Guidelines, but are central to Rust. That said, I always welcome tooling to make C++ safer; the end game is making programs better, not language partisanship!
    - swift 9y ago
      I will say this for C++: post C++11, it’s one of the few languages in widespread use to have an explicitly defined memory model. The people working on C++ definitely do care about concurrency and parallelism. I’d still choose Rust over C++ for that kind of program any time I was given the choice, though. =)
      - steveklabnik 9y ago
        Absolutely, I'm not saying they don't care; it's that solving that problem is an explicit non-goal of the GSL work. The C++ committee is clearly working on concurrency related things, I was reading the various coroutines TSes recently in fact, as we're working on similar things in Rust.
    - duneroadrunner 9y ago
      SaferCPlusPlus[1] is the library that's probably closer to Rust "in spirit and intent". And more importantly, safety effectiveness. And it does address the data race issue. [1] shameless plug: https://github.com/duneroadrunner/SaferCPlusPlus https://github.com/duneroadrunner/SaferCPlusPlus
- r0br0dz 9y ago
  Im in awe too how single handedly some folks have that drive to keep hitting the keyboard and put all the brain's power into a model defined by electricity and binary. Truly amazing! Bravo
kibwen 9y ago
This was recently posted to /r/rust with the engagement of the author (mutabah) if you'd like to ask questions or get involved. In particular here's him answering why he started this project: https://www.reddit.com/r/rust/comments/718gbh/mrustc_alternative_rust_compiler_written_in_c/dn90vo1/?context=10000 https://www.reddit.com/r/rust/comments/718gbh/mrustc_alterna... One of the interesting things about this project is that because its initial goal is to validate the reproducibility of the main Rust compiler it doesn't actually need to implement a borrow checker, as the borrow checker strictly rejects programs and has no role in code generation.
- grok2 9y ago
  If you don't really need the borrow checker to compile Rust, can the original Rust compiler have a mode where it doesn't do the borrow checking? That might make it easier to ease into Rust, because right now, "fighting" the borrow-checker is the hardest problem with learning Rust.
  - steveklabnik 9y ago
    The borrow checker is needed for soundness; this means mrustc will miscompile things. Assuming the input is sound, mrustc will do the right thing. Ignoring the borrow checker will not make things magically work.
  - __s 9y ago
    Learning Rust without borrowck is throwing the baby out with the bath water
    - nickrio 9y ago
      How about "throwing the baby out but keeps the bath water"? Which is more appropriate for this scene I think. LOL
  - keeperofdakeys 9y ago
    Removing the borrow checker would allow you to compile broken code (eg. use after free, data races). However given code that compiles on rustc - passes the borrow checker - you don't need to reverify the borrow checking to get correct code from mrustc.
raimue 9y ago
Maybe this compiler could help with bootstrapping the official rust compiler. At the moment, it seems to be quite a mess for packagers. Rust is written in Rust, but also needs cargo to build, which is also written in Rust. To break this circular dependency chain, you need to have pre-compiled binaries first in order to bootstrap the build process.
- steveklabnik 9y ago
  That's one of the goals, yeah. The author is working on Cargo now.
- mitchty 9y ago
  This is true of a lot of languages, go, haskell (even more once shake becomes the build system for it) etc... It makes porting things... interesting.
  - steveklabnik 9y ago
    Usually you port by cross compiling an initial compiler, and bootstrapping from there.
    - binarycrusader 9y ago
      Speaking from personal experience, that can be quite painful. Being able to build rust and cargo, even if only for bootstrapping when porting, would truly be a wondrous thing.
      - fmap 9y ago
        Why, we could always use emscripten to compile rustc to JavaScript and use that for bootstrapping! On a more serious note, ghc at least solved this problem by allowing you to compile Haskell to C (-fvia-C). Writing a naive C code generator from whatever your backend is using is probably not that hard.
        9y ago
ndesaulniers 9y ago
It's a major milestone for a compiled language to have it be self hosting. Another is having a competing implementation of the compiler/runtime. That's legit; cool project and kudos to the beautiful Rust ecosystem.
alexlarsson 9y ago
IF the goal is bootstrapping another approach would be to use the llvm c backend to generate a version of rustc and cargo that can be built by a c compiler.
- kiriakasis 9y ago
  IIRC the c backend for llvm was deprecated quite a while ago
topbanana 9y ago
Curious why this wasn't written in Rust?
- Ygg2 9y ago
  Rust compiler is already written in Rust, which means it can't be deployed to some environments that C++ could. This compiler in essence "bridges" the gap. Sorta.
phkahler 9y ago
It may have been nice to build it on GCC instead of LLVM. First, the existing Rust compiler uses LLVM so this won't be a fully independent implementation. Second, it would provide GCC with a Rust front end.

182 points by mabynogy 9y ago

Lots of discussion and questions answered by the author on the Rust subreddit the other day https://www.reddit.com/r/rust/comments/718gbh/mrustc_alternative_rust_compiler_written_in_c/ https://www.reddit.com/r/rust/comments/718gbh/mrustc_alterna...

fyrerise 9y ago

Neat project. It looks like this was largely written by one person, and I'm fairly in awe at anyone who can take a big project like a compiler this far alone. Isn't there a bit of cognitive dissonance in believing that Rust as a language is an important idea (i.e. by the additional code safety and code maintainability that it conveys), but then simultaneously making the effort to rewrite the current Rust-implemented compiler in C++? C++ is fast, but aside from a shared value around performance, it has fairly little in common with the ideas that Rust is built on.

CUViper 9y ago
By not self-hosting in Rust, it should be easier to bootstrap on a new target. C++ is ubiquitous -- what other language would you choose?
- yjftsjthsd-h 9y ago
  If bootstrapping/portability is the goal, probably C.
  - damnstraight 9y ago
    Conveniently there are already C++ compilers bootstrappable with C.
    - tavert 9y ago
      Any that are both open-source and C++11 compliant? Guess you can still build g++-4.6 with just C, then a newer g++ from that, but it's a bit of a pain.
- fyrerise 9y ago
  > what other language would you choose? As far as I can tell the goal of the project isn't to target more platforms (Rust targets quite a few by way of LLVM), so I don't think I'd choose any other language, including C++. Having a compiler and standard library written in the language that it compiles has some huge benefits for increasing the pool of possible contributors.
  - kibwen 9y ago
    > As far as I can tell the goal of the project isn't to target more platforms (Rust targets quite a few by way of LLVM) There appears to be an attempt to leverage this project to target ESP8266, which AFAIK LLVM does not support: https://github.com/emosenkis/esp-rs https://github.com/emosenkis/esp-rs
    - kbenson 9y ago
      Interesting. Along with with your other comment about the borrow checker, I guess you could develop using (or occasionally check against) the rustc compiler for borrow checker correctness, and deploy using mrustc. That's pretty cool.
  - tatterdemalion 9y ago
    > Having a compiler and standard library written in the language that it compiles has some huge benefits for increasing the pool of possible contributors. But this isn't the official compiler, this is someone's personal project?
    - fyrerise 9y ago
      > But this isn't the official compiler, this is someone's personal project? True, but compilers are complicated machines and Rust is still changing at a fairly frantic rate. The author seems to be doing quite a good job of development today, but if it has any hope of staying current, it probably needs to think about how to increase its bus factor (something happens like changing jobs, starting a family, or they just become interested in something else, and a single person suddenly has less time to contribute).
- deleted 9y ago
  [deleted]
- xioxox 9y ago
  If the goal is to break the dependency cycle, a higher level language like Python would make development much easier. C++ is powerful, but not as rapid to develop in.
  - xioxox 9y ago
    Why the downvotes? It's clearly easier to make a compiler in a higher level language (Python is just an example, but Lisps are suited to this kind of thing). For example, text parsing is easier in Perl/Ruby/Python/Swift/etc. As someone who knows C++, more thought is required to do the same thing as in a higher level language, although it runs much faster. If you just wanted to bootstrap the compiler, then you'd choose the easiest route to that. It could also be easier to read and understand than a C++ compiler.
- pjmlp 9y ago
  Ada, for example. There is an open source version available in every system supported by gcc. However C++17 is quite safe, provided one doesn't follow "write C with C++ compiler" idiom.
  - pcwalton 9y ago
    > However C++17 is quite safe, provided one doesn't follow "write C with C++ compiler" idiom. No, it's not. There is no use-after-free protection among other things. It doesn't matter how much you write code like C or not: it's just not safe.
    - rtpg 9y ago
      safety is a spectrum, I think declaring it unsafe entirely might not be conducive to more interesting points
    - pjmlp 9y ago
      Then better stop using LLVM on Rust. Until then, those comments only annoy those of us that happen to like Rust, but don't find it mature enough to replace C++ on the use cases we happen to care about.
      - pcwalton 9y ago
        I didn't say you can't write useful software in C++. I said that C++ software is not, in general, memory safe. Sometimes the benefits of a particular piece of software (for example, having an excellent production-grade optimizer) outweigh its drawbacks (for example, not being memory-safe). I don't think you'd find a single LLVM developer who would claim that LLVM is memory safe. Giving invalid IR to LLVM and not running the verifier frequently segfaults it, for example...
        pjmlp 9y ago
        I do agree with that, but that is exactly why I always mention the "C with C++ compiler" idiom, which is usually followed by your remark.
mrob 9y ago
Multiple implementations of a compiler lets you implement the "Diverse Double-Compiling"[0] countermeasure to the famous "Reflections on Trusting Trust"[1] attack. You wouldn't necessarily use the C++ implementation in production, but it still improves the security of the Rust language just by existing. [0] https://www.dwheeler.com/trusting-trust/dissertation/html/wheeler-trusting-trust-ddc.html#0.Abstract https://www.dwheeler.com/trusting-trust/dissertation/html/wh... (previous HN discussion at: https://news.ycombinator.com/item?id=12666923 https://news.ycombinator.com/item?id=12666923 ) [1] http://www.ece.cmu.edu/~ganger/712.fall02/papers/p761-thompson.pdf http://www.ece.cmu.edu/~ganger/712.fall02/papers/p761-thomps...
- Manishearth 9y ago
  DDC is irrelevant here, DDC is an argument to not write the second compiler in C++ and write it in Rust too. Having a Rust compiler in C++ is a mitigation to the trusting trust attack, period. You don't need DDC for this. DDC is necessary when you have two self hosted compilers (e.g. GCC and clang). Here we have one self-hosted compiler (rustc), and one in another language (C++). To mitigate trusting trust in rustc, use mrustc to compile rustc, and then use that rustc to compile itself, and now you have a trusted binary (provided you trust your C++ compiler. you can fix this by DDCing the C++ compilers)
santaclaus 9y ago
The Rust compiler infrastructure is built on LLVM (a C++ project), so this doesn’t seem terribly weird.
santaclaus 9y ago
> C++ is fast, but aside from a shared value around performance, it has fairly little in common with the ideas that Rust is built on. The core guidelines are very similar to Rust, in spirit and intent.
- steveklabnik 9y ago
  ... but the details are very, very different. Unless I've missed significant revisions, data races and concurrency are a non-goal of the Core Guidelines, but are central to Rust. That said, I always welcome tooling to make C++ safer; the end game is making programs better, not language partisanship!
  - swift 9y ago
    I will say this for C++: post C++11, it’s one of the few languages in widespread use to have an explicitly defined memory model. The people working on C++ definitely do care about concurrency and parallelism. I’d still choose Rust over C++ for that kind of program any time I was given the choice, though. =)
    - steveklabnik 9y ago
      Absolutely, I'm not saying they don't care; it's that solving that problem is an explicit non-goal of the GSL work. The C++ committee is clearly working on concurrency related things, I was reading the various coroutines TSes recently in fact, as we're working on similar things in Rust.
  - duneroadrunner 9y ago
    SaferCPlusPlus[1] is the library that's probably closer to Rust "in spirit and intent". And more importantly, safety effectiveness. And it does address the data race issue. [1] shameless plug: https://github.com/duneroadrunner/SaferCPlusPlus https://github.com/duneroadrunner/SaferCPlusPlus
r0br0dz 9y ago
Im in awe too how single handedly some folks have that drive to keep hitting the keyboard and put all the brain's power into a model defined by electricity and binary. Truly amazing! Bravo

kibwen 9y ago

This was recently posted to /r/rust with the engagement of the author (mutabah) if you'd like to ask questions or get involved. In particular here's him answering why he started this project: https://www.reddit.com/r/rust/comments/718gbh/mrustc_alternative_rust_compiler_written_in_c/dn90vo1/?context=10000 https://www.reddit.com/r/rust/comments/718gbh/mrustc_alterna... One of the interesting things about this project is that because its initial goal is to validate the reproducibility of the main Rust compiler it doesn't actually need to implement a borrow checker, as the borrow checker strictly rejects programs and has no role in code generation.

grok2 9y ago
If you don't really need the borrow checker to compile Rust, can the original Rust compiler have a mode where it doesn't do the borrow checking? That might make it easier to ease into Rust, because right now, "fighting" the borrow-checker is the hardest problem with learning Rust.
- steveklabnik 9y ago
  The borrow checker is needed for soundness; this means mrustc will miscompile things. Assuming the input is sound, mrustc will do the right thing. Ignoring the borrow checker will not make things magically work.
- __s 9y ago
  Learning Rust without borrowck is throwing the baby out with the bath water
  - nickrio 9y ago
    How about "throwing the baby out but keeps the bath water"? Which is more appropriate for this scene I think. LOL
- keeperofdakeys 9y ago
  Removing the borrow checker would allow you to compile broken code (eg. use after free, data races). However given code that compiles on rustc - passes the borrow checker - you don't need to reverify the borrow checking to get correct code from mrustc.

raimue 9y ago

Maybe this compiler could help with bootstrapping the official rust compiler. At the moment, it seems to be quite a mess for packagers. Rust is written in Rust, but also needs cargo to build, which is also written in Rust. To break this circular dependency chain, you need to have pre-compiled binaries first in order to bootstrap the build process.

steveklabnik 9y ago
That's one of the goals, yeah. The author is working on Cargo now.
mitchty 9y ago
This is true of a lot of languages, go, haskell (even more once shake becomes the build system for it) etc... It makes porting things... interesting.
- steveklabnik 9y ago
  Usually you port by cross compiling an initial compiler, and bootstrapping from there.
  - binarycrusader 9y ago
    Speaking from personal experience, that can be quite painful. Being able to build rust and cargo, even if only for bootstrapping when porting, would truly be a wondrous thing.
    - fmap 9y ago
      Why, we could always use emscripten to compile rustc to JavaScript and use that for bootstrapping! On a more serious note, ghc at least solved this problem by allowing you to compile Haskell to C (-fvia-C). Writing a naive C code generator from whatever your backend is using is probably not that hard.
      - 9y ago

ndesaulniers 9y ago

It's a major milestone for a compiled language to have it be self hosting. Another is having a competing implementation of the compiler/runtime. That's legit; cool project and kudos to the beautiful Rust ecosystem.

alexlarsson 9y ago

IF the goal is bootstrapping another approach would be to use the llvm c backend to generate a version of rustc and cargo that can be built by a c compiler.

kiriakasis 9y ago
IIRC the c backend for llvm was deprecated quite a while ago

topbanana 9y ago

Curious why this wasn't written in Rust?

Ygg2 9y ago
Rust compiler is already written in Rust, which means it can't be deployed to some environments that C++ could. This compiler in essence "bridges" the gap. Sorta.

phkahler 9y ago

It may have been nice to build it on GCC instead of LLVM. First, the existing Rust compiler uses LLVM so this won't be a fully independent implementation. Second, it would provide GCC with a Rust front end.