3 ms·

I think this post has a clearly missing solution that is (essentially) a first stage in a build system. Yes, C++ can be slow to build. But it's slow to build b

I think this post has a clearly missing solution that is (essentially) a first stage in a build system. Yes, C++ can be slow to build. But it's slow to build because of the includes and the code generation (and we still have to do code generation). How much time does the author think it takes to pipe code through the preprocessor (for defines, not includes since in this case there are no includes), and tokenize? I just checked the dependency files of a fairly large project (GNU Radio), and each cc file transitively depends on 414 files on average. In total in order to compile its 1339 cc files the compiler needs to open and read over half a million files. So just in terms of data to read on this first pass it's faster by a factor over 400x. Add to that that you don't have to do anything in this pass except look for module dependencies, I bet it's several thousand times faster than a compile. So in other words, if it takes you an hour to build your big project, are you saying you don't have one single second left over to calculate the dependency tree as a first pass? And you only have to do it once (and then re-do files as source timestamps change).

WalterBright 5y ago
D just reads and compiles each module only once, no matter how many times it is imported, or whatever rat's nest the import graph is. Modules could be precompiled, but I found that it was faster (and easier) to just recompile them. (And not having a preprocessor eliminates several passes.)
munch117 5y ago
If the compiler and the build system is an integrated product, then right, this shouldn't be too hard. But that's not usually the case. If they're not, what's the interface for the initial build step? Is it still something like "gcc -MD"? But "gcc -MD" can't just spit out a list of files. It will have to output not only file names but also module names (both referenced and declared), because resolving module names to file names requires global knowledge. That means every build tool that wants to work with C++ must be changed to take in this extended information and resolve module dependencies. Because otherwise, if there's even a single module in your 1339 file project, then current tooling breaks. Which will put pressure on developers to not add that first module, slowing adoption.
- knorker 5y ago
  Sure, probably a little bit more work. But I think building a tree of 1339 items is quick enough even on a PDP-11, right? Or is there a step to this I'm missing? Can't they be analyzed individually and then joined? Conceptually this merging isn't much different from the fact that gcc -MD needs to output data formatted in a way that Make understands. But fair enough. I feel like I'm for the second time am saying "just add another stage to the build". Still. "Dead on arrival" is a bit much.
  - munch117 5y ago
    Make doesn't understand gcc -MD output. Make has stayed basically the same for half a century. There's nothing language-specific in make. For example, when C++ was invented, nothing needed to be added to make, because the general mechanism could easily support adding a cfront step that compiled .cxx files into .c. It looks like that's changing now. Yesterday's make will not be able to build tomorrow's C++.
    - knorker 5y ago
      > Make doesn't understand gcc -MD output. How do you mean? It's literally Makefile syntax that's being output, and it's directly included by the Makefile as-is.
      - munch117 5y ago
        You're right, that wasn't well phrased. What I meant is that make doesn't have special code to handle "gcc -MD" output. gcc is adapted to the needs of make, not the other way around.
        knorker 5y ago
        I'm not going to try to design the best way to do the equivalent for C++ modules here in a HN comment, but it doesn't seem like a particularly hard problem. Of course it could be exactly because I've not seriously written a compiler or build system that I mistake the problem for not being hard. But it shouldn't be a problem for explicit dependency build systems like Bazel. And people have written automatic Bazel dependency tooling, so it really looks to me like "what's the best way to do this" rather than "we can't do this", which the article implies. Yes, gcc produces dependency information while compiling, and is adapted for Make. But in theory this could be done by `./configure` or something in a dedicated script. It'd not even be hard. And it's super quick: cpp -I. src/foo.cc | grep '^# ' | sed 's/^[^"]"//;s/"[^"]$//' | sort | uniq | grep -v \< It's not without precedence to have a pre-stage. Just look at the Ninja build system.

WalterBright 5y ago

D just reads and compiles each module only once, no matter how many times it is imported, or whatever rat's nest the import graph is. Modules could be precompiled, but I found that it was faster (and easier) to just recompile them. (And not having a preprocessor eliminates several passes.)

munch117 5y ago

If the compiler and the build system is an integrated product, then right, this shouldn't be too hard. But that's not usually the case. If they're not, what's the interface for the initial build step? Is it still something like "gcc -MD"? But "gcc -MD" can't just spit out a list of files. It will have to output not only file names but also module names (both referenced and declared), because resolving module names to file names requires global knowledge. That means every build tool that wants to work with C++ must be changed to take in this extended information and resolve module dependencies. Because otherwise, if there's even a single module in your 1339 file project, then current tooling breaks. Which will put pressure on developers to not add that first module, slowing adoption.

knorker 5y ago
Sure, probably a little bit more work. But I think building a tree of 1339 items is quick enough even on a PDP-11, right? Or is there a step to this I'm missing? Can't they be analyzed individually and then joined? Conceptually this merging isn't much different from the fact that gcc -MD needs to output data formatted in a way that Make understands. But fair enough. I feel like I'm for the second time am saying "just add another stage to the build". Still. "Dead on arrival" is a bit much.
- munch117 5y ago
  Make doesn't understand gcc -MD output. Make has stayed basically the same for half a century. There's nothing language-specific in make. For example, when C++ was invented, nothing needed to be added to make, because the general mechanism could easily support adding a cfront step that compiled .cxx files into .c. It looks like that's changing now. Yesterday's make will not be able to build tomorrow's C++.
  - knorker 5y ago
    > Make doesn't understand gcc -MD output. How do you mean? It's literally Makefile syntax that's being output, and it's directly included by the Makefile as-is.
    - munch117 5y ago
      You're right, that wasn't well phrased. What I meant is that make doesn't have special code to handle "gcc -MD" output. gcc is adapted to the needs of make, not the other way around.
      - knorker 5y ago
        I'm not going to try to design the best way to do the equivalent for C++ modules here in a HN comment, but it doesn't seem like a particularly hard problem. Of course it could be exactly because I've not seriously written a compiler or build system that I mistake the problem for not being hard. But it shouldn't be a problem for explicit dependency build systems like Bazel. And people have written automatic Bazel dependency tooling, so it really looks to me like "what's the best way to do this" rather than "we can't do this", which the article implies. Yes, gcc produces dependency information while compiling, and is adapted for Make. But in theory this could be done by `./configure` or something in a dedicated script. It'd not even be hard. And it's super quick: cpp -I. src/foo.cc | grep '^# ' | sed 's/^[^"]"//;s/"[^"]$//' | sort | uniq | grep -v \< It's not without precedence to have a pre-stage. Just look at the Ninja build system.