7 ms·

Looking at this, I can now see why a test function nested in another function (see [0] and [1]) doesn’t work. I’ve tried dabbling with the Rust compiler to see

Looking at this, I can now see why a test function nested in another function (see [0] and [1]) doesn’t work. I’ve tried dabbling with the Rust compiler to see if I could help fix this, but it takes 30 minutes to compile rustc and I couldn’t figure out how to reduce those compile times (each time I changed 1 line and wanted to test, I had to wait 30 minutes to build). How do people work on rustc (or internal Rust libraries, like libsyntax) without waiting for 30 minute builds after every minor change? [0]: https://github.com/rust-lang/rust/issues/36629 https://github.com/rust-lang/rust/issues/36629 [1]: https://github.com/rust-lang/rfcs/issues/612 https://github.com/rust-lang/rfcs/issues/612

estebank 8y ago
./x.py test src/test/ui --stage 1 And coffee, as that already takes around 5 to 20 minutes depending on what part of the compiler you're touching.
djrenren 8y ago
Hi author here, I won't lie, it's rough. I definitely built this whole website and wrote the blog during my "compile breaks". On the bright side, typechecking is super fast so I only really have to wait when making algorithmic changes. Still, especially for anything that does code generation, you'll be tweaking the algorithm a good deal.
- antpls 8y ago
  Would building the compiler on cloud instances help? If the compiler is built in Rust, I guess it would efficiently use several dozens of cores and gigabytes of RAM
- hinkley 8y ago
  There's a cargo watch command that tries to do the same thing that other languages with slow compilation or test frameworks do, and that's to run all your tests the minute you save. This narrows the time between when you have something to test and when you remember to test it. It saves you wall clock time if not CPU time. (I highly recommend combining watch with an editor or IDE that saves all buffers at the same time, instead of one at a time).
  - djrenren 8y ago
    I don't believe this would work with the rustc compiler because, although it uses Cargo to build, it has a special process that bootstraps the compiler by building itself.
mark-simulacrum 8y ago
Depending on what you're doing, there's often a few different approaches to reducing compile times. First of all, if you're going to be running (small) tests, then it may be beneficial to enable incremental with `--incremental` or `-i` to x.py. This does make the resulting compiler slower, but does make the build itself faster. If you're making type-level changes that aren't going to alter functionality, or want to just make sure the whole compiler will build, you can run `x.py check -i` to get the equivalent of `cargo check` outside of the compiler tree. If you know ahead of time that you're not changing the code generation/ABI of the compiler (metadata, mangling, that sort of thing) then you can also do something like `x.py --keep-stage 0` and work entirely in stage 2 (you wouldn't pass --stage 1 with this option). Alternatively, you can pass `--keep-stage 1` and run the stage 1 tests; this will save you a rebuild of std and test which is often unnecessary after a compiler change. However, even with these suggestions/hints, building the Rust compiler can be rather slow.
gameswithgo 8y ago
compiling gcc takes even longer
- readittwice 8y ago
  but incremental builds only take a few seconds when bootstrap is disabled...
  - djrenren 8y ago
    You can do the same on rustc by just doing --keep-stage 0. This doesn't work with with codegen-like stuff but for a lot of changes it works.
WalterBright 8y ago
A debug build of the D compiler on my (ordinary) machine takes 24 seconds. The optimized build takes 132 seconds.
- dgzxzgzx 8y ago
  The level of optimization they are not doing must be staggering.
  - WalterBright 8y ago
    Not really. The two main reasons LLVM and gcc do a better optimization job is they have a better function inliner and loop unroller. Neither are particularly computation intensive. DMD does all the usual data flow analysis optimizations - strength reduction, copy propagation, loop unrolling, dead store elimination, live range analysis, etc. https://github.com/DigitalMars/Compiler/blob/master/dm/src/dmc/gflow.d https://github.com/DigitalMars/Compiler/blob/master/dm/src/d... The optimizer / code generator is the same one that Digital Mars C++ uses, which was developed in the 1980's (!). Up until 2000 or so, the code generated was about the same as gcc, though DMC++ ran several times faster. Since 2000, all my attention has been on the D front end, and so the optimizer wasn't getting the devoted attention it had earlier. Just recently I finished converting the DMD optimizer from C to D, and am doing the code generator. This should make the code much more tractable.
    - pcwalton 8y ago
      Respectfully—and I truly mean this!—this is nowhere near enough to attain parity with GCC and LLVM. I don't see any SSA form, to name just one of many issues. GCC 4.0, which introduced the SSA-based GIMPLE, was released in 2004, which is roughly around the same time you mentioned that the optimization quality started to diverge. This is not an accident. At the time, lots of compilers thought that they could continue to compete with GCC without doing SSA-based optimizations. GCC crushed them all one by one.
      - WalterBright 8y ago

estebank 8y ago

./x.py test src/test/ui --stage 1 And coffee, as that already takes around 5 to 20 minutes depending on what part of the compiler you're touching.

djrenren 8y ago

Hi author here, I won't lie, it's rough. I definitely built this whole website and wrote the blog during my "compile breaks". On the bright side, typechecking is super fast so I only really have to wait when making algorithmic changes. Still, especially for anything that does code generation, you'll be tweaking the algorithm a good deal.

antpls 8y ago
Would building the compiler on cloud instances help? If the compiler is built in Rust, I guess it would efficiently use several dozens of cores and gigabytes of RAM
hinkley 8y ago
There's a cargo watch command that tries to do the same thing that other languages with slow compilation or test frameworks do, and that's to run all your tests the minute you save. This narrows the time between when you have something to test and when you remember to test it. It saves you wall clock time if not CPU time. (I highly recommend combining watch with an editor or IDE that saves all buffers at the same time, instead of one at a time).
- djrenren 8y ago
  I don't believe this would work with the rustc compiler because, although it uses Cargo to build, it has a special process that bootstraps the compiler by building itself.

mark-simulacrum 8y ago

Depending on what you're doing, there's often a few different approaches to reducing compile times. First of all, if you're going to be running (small) tests, then it may be beneficial to enable incremental with `--incremental` or `-i` to x.py. This does make the resulting compiler slower, but does make the build itself faster. If you're making type-level changes that aren't going to alter functionality, or want to just make sure the whole compiler will build, you can run `x.py check -i` to get the equivalent of `cargo check` outside of the compiler tree. If you know ahead of time that you're not changing the code generation/ABI of the compiler (metadata, mangling, that sort of thing) then you can also do something like `x.py --keep-stage 0` and work entirely in stage 2 (you wouldn't pass --stage 1 with this option). Alternatively, you can pass `--keep-stage 1` and run the stage 1 tests; this will save you a rebuild of std and test which is often unnecessary after a compiler change. However, even with these suggestions/hints, building the Rust compiler can be rather slow.

gameswithgo 8y ago

compiling gcc takes even longer

readittwice 8y ago
but incremental builds only take a few seconds when bootstrap is disabled...
- djrenren 8y ago
  You can do the same on rustc by just doing --keep-stage 0. This doesn't work with with codegen-like stuff but for a lot of changes it works.

WalterBright 8y ago

A debug build of the D compiler on my (ordinary) machine takes 24 seconds. The optimized build takes 132 seconds.

dgzxzgzx 8y ago
The level of optimization they are not doing must be staggering.
- WalterBright 8y ago
  Not really. The two main reasons LLVM and gcc do a better optimization job is they have a better function inliner and loop unroller. Neither are particularly computation intensive. DMD does all the usual data flow analysis optimizations - strength reduction, copy propagation, loop unrolling, dead store elimination, live range analysis, etc. https://github.com/DigitalMars/Compiler/blob/master/dm/src/dmc/gflow.d https://github.com/DigitalMars/Compiler/blob/master/dm/src/d... The optimizer / code generator is the same one that Digital Mars C++ uses, which was developed in the 1980's (!). Up until 2000 or so, the code generated was about the same as gcc, though DMC++ ran several times faster. Since 2000, all my attention has been on the D front end, and so the optimizer wasn't getting the devoted attention it had earlier. Just recently I finished converting the DMD optimizer from C to D, and am doing the code generator. This should make the code much more tractable.
  - pcwalton 8y ago
    Respectfully—and I truly mean this!—this is nowhere near enough to attain parity with GCC and LLVM. I don't see any SSA form, to name just one of many issues. GCC 4.0, which introduced the SSA-based GIMPLE, was released in 2004, which is roughly around the same time you mentioned that the optimization quality started to diverge. This is not an accident. At the time, lots of compilers thought that they could continue to compete with GCC without doing SSA-based optimizations. GCC crushed them all one by one.
    - WalterBright 8y ago