4 ms·

Something I've always wondered about compilers written in their own language.... What is your process for compiling a new compiler? Let's say you make a code c

Something I've always wondered about compilers written in their own language.... What is your process for compiling a new compiler? Let's say you make a code change to the compiler. You have a compiled version of the previous compiler you can run to compile the new compiler. But, by definition, the new compiler is different from the old one. Do you re-run the compilation with the new compiler? How many times?

moonchild 4y ago
> How many times? Once.
atorodius 4y ago
I think the keyword to look for is „bootstrapping compilers“
WalterBright 4y ago
1. Compile the new compiler with the old compiler 2. Compile the new compiler with the result of (1) 3. Compile the new compiler with the result of (2) 4. Verify that (2) and (3) produce identical results
- titzer 4y ago
  Indeed, this is part of Virgil's default test run (and presumably most self-hosted compilers'). If the question is about adding new features to the language, then the process is: 1. Add the new feature to the (source of the) new compiler in a way that doesn't break any existing feature. 2. Cement the new feature in with extensive tests. 3. Bootstrap the new compiler and stable-rev it (in Virgil, that means checking in the new compiler's binary into the repo). 4. Work on other things for a while; either optimizations in the compiler or applications, to shake out bugs. 5. Bootstrap and stable-rev again. 6. Gently start using the new feature in the compiler source itself.
cpeterso 4y ago
Rust’s documentation describes how new versions of the rustc compiler are bootstrapped: https://rustc-dev-guide.rust-lang.org/buildings/bootstrapping.html https://rustc-dev-guide.rust-lang.org/buildings/bootstrappin... Stage 0: The stage0 compiler is usually the current beta rustc compiler. States 1: The rustc source code is then compiled with the stage0 compiler to produce the stage1 compiler. Stage 2: We then rebuild our stage1 compiler with itself to produce the stage2 compiler. In theory, the stage1 compiler is functionally identical to the stage2 compiler, but in practice there are subtle differences. The stage2 compiler is the one distributed with rustup and all other install methods. State 3: To sanity check our new compiler, we can build the libraries with the stage2 compiler. The result ought to be identical to before, unless something has broken.
- LoganDark 4y ago
  To expand on this, the "subtle differences" are things like optimizations introduced/tweaks in the new compiler that can only be taken advantage of by creating stage2. (since stage1 was compiled with the old compiler)
giancarlostoro 4y ago
I think where you might be getting confused is you are assuming the new feature is some syntax they'll just automatically start using in the new compiler immediately. Typically you wait till the compiler you added that new feature to is released, then you can consider how you would refactor your compiler code, and then it should compile since you already have your new syntactic sugar or whatever. I hope that makes more sense to you. I think what Walter Bright answered was good too, but I think it helps to remind oneself, that just because your new compiler code implements something new, doesnt mean you have to use it the second you want to compile it, so it wont matter until the new compiler is ready, then you consider adding new syntax or features to compiler code base.
- disqard 4y ago
  Thank you for adding this explanation! It helps catch the mistaken assumption one might make, that "adding X does not imply the compiler's source code also immediately starts using X".
BatteryMountain 4y ago
It's called bootstrapping. It's the hard part of any new language, esp lower level ones. Your first iteration will be very manual/low level, until your compiler gets sophisticated enough to compile itself, granted your trust it enough to do the right thing. It can take many iterations to get to the point that you trust your compiler works well enough for daily use to build itself and then you can slowly start discarding the older pieces from earlier iterations. On some level I think Rust will become a major player for building compilers with (and obviously drivers), and since it is a portable executable and safe/predictable, there is a good chance the the compiler dev won't need to switch to his own language to compile itself, unless ofcoarse a point of pride, some specific functionality that rust cannot do or if the person just don't like rust. Compiler development is a different beast altogether from most forms of programming, and I highly recommend you build a basic one as a hobby project. It will let you appreciate the shoulders of giants we are standing. Same goes for 3D/physics engines, audio/signal processing and so on. Building a basic filesystem or database that supports indexes and a strict schema that has some form of relational theory in it is also highly enlightening and will dispell the magic of sql engines (and make you appreciate those that came before you and their struggle to get where we are today).

moonchild 4y ago

> How many times? Once.

atorodius 4y ago

I think the keyword to look for is „bootstrapping compilers“

WalterBright 4y ago

1. Compile the new compiler with the old compiler 2. Compile the new compiler with the result of (1) 3. Compile the new compiler with the result of (2) 4. Verify that (2) and (3) produce identical results

titzer 4y ago
Indeed, this is part of Virgil's default test run (and presumably most self-hosted compilers'). If the question is about adding new features to the language, then the process is: 1. Add the new feature to the (source of the) new compiler in a way that doesn't break any existing feature. 2. Cement the new feature in with extensive tests. 3. Bootstrap the new compiler and stable-rev it (in Virgil, that means checking in the new compiler's binary into the repo). 4. Work on other things for a while; either optimizations in the compiler or applications, to shake out bugs. 5. Bootstrap and stable-rev again. 6. Gently start using the new feature in the compiler source itself.

cpeterso 4y ago

Rust’s documentation describes how new versions of the rustc compiler are bootstrapped: https://rustc-dev-guide.rust-lang.org/buildings/bootstrapping.html https://rustc-dev-guide.rust-lang.org/buildings/bootstrappin... Stage 0: The stage0 compiler is usually the current beta rustc compiler. States 1: The rustc source code is then compiled with the stage0 compiler to produce the stage1 compiler. Stage 2: We then rebuild our stage1 compiler with itself to produce the stage2 compiler. In theory, the stage1 compiler is functionally identical to the stage2 compiler, but in practice there are subtle differences. The stage2 compiler is the one distributed with rustup and all other install methods. State 3: To sanity check our new compiler, we can build the libraries with the stage2 compiler. The result ought to be identical to before, unless something has broken.

LoganDark 4y ago
To expand on this, the "subtle differences" are things like optimizations introduced/tweaks in the new compiler that can only be taken advantage of by creating stage2. (since stage1 was compiled with the old compiler)

giancarlostoro 4y ago

I think where you might be getting confused is you are assuming the new feature is some syntax they'll just automatically start using in the new compiler immediately. Typically you wait till the compiler you added that new feature to is released, then you can consider how you would refactor your compiler code, and then it should compile since you already have your new syntactic sugar or whatever. I hope that makes more sense to you. I think what Walter Bright answered was good too, but I think it helps to remind oneself, that just because your new compiler code implements something new, doesnt mean you have to use it the second you want to compile it, so it wont matter until the new compiler is ready, then you consider adding new syntax or features to compiler code base.

disqard 4y ago
Thank you for adding this explanation! It helps catch the mistaken assumption one might make, that "adding X does not imply the compiler's source code also immediately starts using X".

BatteryMountain 4y ago

It's called bootstrapping. It's the hard part of any new language, esp lower level ones. Your first iteration will be very manual/low level, until your compiler gets sophisticated enough to compile itself, granted your trust it enough to do the right thing. It can take many iterations to get to the point that you trust your compiler works well enough for daily use to build itself and then you can slowly start discarding the older pieces from earlier iterations. On some level I think Rust will become a major player for building compilers with (and obviously drivers), and since it is a portable executable and safe/predictable, there is a good chance the the compiler dev won't need to switch to his own language to compile itself, unless ofcoarse a point of pride, some specific functionality that rust cannot do or if the person just don't like rust. Compiler development is a different beast altogether from most forms of programming, and I highly recommend you build a basic one as a hobby project. It will let you appreciate the shoulders of giants we are standing. Same goes for 3D/physics engines, audio/signal processing and so on. Building a basic filesystem or database that supports indexes and a strict schema that has some form of relational theory in it is also highly enlightening and will dispell the magic of sql engines (and make you appreciate those that came before you and their struggle to get where we are today).