4 ms·

Back in the 90s, I implemented precompiled headers for my C++ compiler (Symantec C++). They were very much like modules. There were two modes of operation: 1.

Back in the 90s, I implemented precompiled headers for my C++ compiler (Symantec C++). They were very much like modules. There were two modes of operation: 1. all the .h files were compiled, and emitted as a binary that could be rolled in all at once 2. each .h file created its own precompiled header. Sounds like modules, right? Anyhow, I learned a lot, mostly that without semantic improvements to C++, while it made compilation much faster, it was too sensitive to breakage. This experience was rolled into the design of D modules, which work like a champ. They were everything I wanted modules to be. In particular, The semantic meaning of the module is completely independent of wherever it is imported from. Anyhow, C++ is welcome to adopt the D design of modules. C++ would get modules that have 25 years of use, and are very satisfying. Yes, I do understand that the C preprocessor macros are a problem. My recommendation is, find language solutions to replace the preprocessor. C++ is most of the way there, just finish the job and relegate the preprocessor to the dustbin.

fooker 1y ago
What has to change in C++ templates for this to work? It seems particularly tricky to define a template in a module and then instantiate it or specialize it somewhere else.
- WalterBright 1y ago
  In order to make things work smoothly, the module has to have its own namespace, and a namespace that is closed. D also has an `alias` feature, where you can do things like: alias Q = abc.T; where from then on, `abc.T` can be referred to simply as `Q`. This also eliminates a large chunk of purpose behind the preprocessor.
  - fooker 1y ago
    Neat, closed namespaces sound great! C++ has adapted the ‘using’ keyword now to seem fairly similar to alias, but can’t completely subsume macros unfortunately.
    - WalterBright 1y ago
      Closed namespaces means the module can be reliably imported anywhere without changing its semantic meaning.
  - Gibbon1 1y ago
    I've really wanted that in C for a long time. It seems like a very trivial thing as well.
    - WalterBright 1y ago
      Yeah, it's actually easy to implement, too. It replaces the preprocesser: #define Q abc.T with hygiene. Once you get used to it, it has all kinds of nice uses.
      - StilesCrisis 1y ago
        C++ code generally uses `using` for typedefs. Macro-based typedefs are exceptionally rare and frowned upon.
- WanderPanda 1y ago
  I think modularization of templates is really hard. Best thing I can think of is a cache e.g. for signatures. But then again this is basically what the mangling already does anyways in my understanding.
  - fooker 1y ago
    Not just signatures, you need the whole AST more or less. This seems incredibly wasteful, but of course still marginal better than just #including code which is the alternative.
- jjmarr 1y ago
  Including or importing a templated class/function should not require bringing in the definition. That's why #includes and imports are so expensive, as we have to parse the entire definition to determine if template instantiations will work. For normal functions or classes, we have forward declarations. Something similar needs to exist for templates.
  - WalterBright 1y ago
    And it doesn't in D. We call such modules ".di files", which consist only of declarations. D does not require names in global scope to be declared lexically before they are used. C++ only does this for class/struct scopes. For example: int bar() { foo(); } int foo() { bar(); } compiles and runs (and runs, and runs, and runs!!!).
    - jjmarr 1y ago
      > D does not require names in global scope to be declared lexically before they are used. C++ only does this for class/struct scopes. But how do you handle a template substitution failure? In C++: template<typename T> auto bar(T x, T y) { return x + y;} The compiler has no idea whether bar(1, 2); will compile unless it parses the full definition. I don't understand how the compiler can avoid parsing the full definition. The expensive bit in my experience isn't parsing the declaration, it's parsing the definition. Typically redundantly over thousands of source files for identical types.
Calavar 1y ago
Did anyone reach out to you for input during the modules standardization process? D seems like the most obvious prior art, but the modules standardization process seems like it was especially cursed
- WalterBright 1y ago
  Nobody from C++ reached out to me for the modules. Herb Sutter, Andrei Alexandrescu and myself once submitted an official proposal for "static if" for C++, based on the huge success it has had in D. We received a vehement rejection. It demotivated me from submitting further proposals. ("static if" replaces the C preprocessor #if/#ifdef/#ifndef constructions.) C++ has gone on to adopt many features of D, but usually with modifications that make them less useful.
  - jjmarr 1y ago
    static if was more or less added in C++17 under the name `if constexpr`. It's not exactly the same since the discarded statement is still checked if not dependent on a template, but like most things in C++, it's similar enough to footgun yourself.
    - sfpotter 1y ago
      "if constexpr" introduces a new scope, while "static if" does not. A major divergence, enough to make the features very, very different in terms of what they can actually be used for.
      - WalterBright 1y ago
        That was a giant mistake on C++'s part, as you cannot do this: static if (feature) { int bar() { betty(); }} ... lots of code ... static if (feature) bar(); Forcing a new scope cuts the utility of it about in half, and there's no way around it. But if you need a scope with D's static if: static if (expression) {{ int x; foo(x); }} the extra { } will do it. But, as it turns out, this is rarely desirable.
cogman10 1y ago
> just finish the job and relegate the preprocessor to the dustbin. Yup, I think this is the core of the problem with C++. The standards committee has drawn a bad line that makes encoding the modules basically impossible. Other languages with good module systems and fast incremental builds don't allow for preprocessor style craziness without some pretty strict boundaries. Even languages that have gotten it somewhat wrong (such as rust with it's proc macros) have bound where and how that sort of metaprogramming can take place. Even if the preprocessor isn't dustbined, it should be excluded from the module system. Metaprogramming should be a feature of the language with clear interfaces and interactions. For example, in Java the annotation processor is ultimately what triggers code generation capabilities. No annotation, no metaprogramming. It's not perfect, but it's a lot better than the C/C++'s free for all macro system. Or the other option is the go route. Don't make the compiler generate code, instead have the build system be responsible for code generation (calling code generators). That would be miles better as it'd allow devs to opt in to that slowdown when they need it.
- will4274 1y ago
  I actually think this attitude is the reason modules are so slowly adopted. The committee did exactly what you suggest and relegated the preprocessor to the dustbin and modules are seeing slow adoption. The truth is 98% of the preprocessor is fine - it's ifdefs for platforms and defines of constants and inline functions that are defined exactly once and never redefined. Because modules supports none of this, that means we can't modulize Windows.h. Or zlib. Or gtest. The committee should have remembered that one of the selling points of C++ is C compatibility, and figured out a way to get modules to work with 98% or the preprocessor and forbid only the nasty 2%.
  - cogman10 1y ago
    > The committee did exactly what you suggest and relegated the preprocessor to the dustbin Importantly, no they did not. They put a boundary on the preprocessor containing preprocessing within the module definition and not the code importing the module. And that's where a lot of the loss and compatibility problems have come into play. That's why you can't, for example, share a module between builds. Because the ifdefs that built the module in the first place may have changed from one build to the next. It was good that the committee bound the preprocessor, but they simply didn't go far enough. C++ is making strides to adding the language features needed to dustbin modules. A lot of the work of consteval can replace a lot of what the preprocessor is doing. > Because modules supports none of this, that means we can't modulize Windows.h. Or zlib. Or gtest. And see, that's the issue. Modules do actually support all this. We can in fact modularize windows.h, zlib, or gtest. The issue is the `windows.module` still has to be rebuilt with every application that imports it because those `ifdefs` could evaluate differently depending on what env variables the build system sets before building. The module can't be just a simple AST built once. Maybe once for a project, but that's about it. And that's the rub. Change anything that causes the module to recompile and you spend exactly the same time you'd spend on precompiled headers.
gary_0 1y ago
I can't think of a C++ project I've worked on that didn't rely on being able to include C headers and have things usually just work. Are there ways of banning C macros from "modular" C++ without breaking that? (Many would find it unacceptable if you had to go through every C dependency and write/generate some sort of wrapper.)
- WalterBright 1y ago
  D resolved this problem by creating D versions of the C system headers. Yes, this was tedious, but we do it for each of our supported platforms. But we can't do it for various C libraries. This created a problem for us, as it is indeed tedious for users. We created a repository where people shared their conversions, but it was still inadequate. The solution was to build a C compiler into the D compiler. Now, you can simply "import" a C .h file. It works surprisingly well. Sure, some things don't work, as C programmers cannot resist put some really crazy stuff in the .h files. The solution to that problem turned out be we discovered that the D compiler was able to create D modules from C code. Then, the user could tweak by hand the nutburger bits.
  - fooker 1y ago
    > The solution was to build a C compiler into the D compiler. This is the same solution that Apple chose for Swift <-> Objective C interop. I wonder if someone at Apple was inspired by this decision in D!
agent327 1y ago
C++ modules aren't influenced by macros on import, nor do they export any macros, so I'm curious what the problem is?
repstosb 1y ago
I've often wondered how the evolution of C and C++ might have been different if a more capable preprocessor (in particular, with more flexible recursive expansion and a better grammar for pattern matching) had caught on. The C++ template engine can be used to work around some of those limits, but always awkwardly, not least due to the way you need Knuth's arrow notation to express the growth in compiler error message volume with template complexity. By the time C++ came out we already had tools like m4 and awk with far more capability than cpp. It's pretty ridiculous that everything else about computing has radically changed since 1970 except the preprocessor and its memory-driven constraints.

fooker 1y ago

What has to change in C++ templates for this to work? It seems particularly tricky to define a template in a module and then instantiate it or specialize it somewhere else.

WalterBright 1y ago
In order to make things work smoothly, the module has to have its own namespace, and a namespace that is closed. D also has an `alias` feature, where you can do things like: alias Q = abc.T; where from then on, `abc.T` can be referred to simply as `Q`. This also eliminates a large chunk of purpose behind the preprocessor.
- fooker 1y ago
  Neat, closed namespaces sound great! C++ has adapted the ‘using’ keyword now to seem fairly similar to alias, but can’t completely subsume macros unfortunately.
  - WalterBright 1y ago
    Closed namespaces means the module can be reliably imported anywhere without changing its semantic meaning.
- Gibbon1 1y ago
  I've really wanted that in C for a long time. It seems like a very trivial thing as well.
  - WalterBright 1y ago
    Yeah, it's actually easy to implement, too. It replaces the preprocesser: #define Q abc.T with hygiene. Once you get used to it, it has all kinds of nice uses.
    - StilesCrisis 1y ago
      C++ code generally uses `using` for typedefs. Macro-based typedefs are exceptionally rare and frowned upon.
WanderPanda 1y ago
I think modularization of templates is really hard. Best thing I can think of is a cache e.g. for signatures. But then again this is basically what the mangling already does anyways in my understanding.
- fooker 1y ago
  Not just signatures, you need the whole AST more or less. This seems incredibly wasteful, but of course still marginal better than just #including code which is the alternative.
jjmarr 1y ago
Including or importing a templated class/function should not require bringing in the definition. That's why #includes and imports are so expensive, as we have to parse the entire definition to determine if template instantiations will work. For normal functions or classes, we have forward declarations. Something similar needs to exist for templates.
- WalterBright 1y ago
  And it doesn't in D. We call such modules ".di files", which consist only of declarations. D does not require names in global scope to be declared lexically before they are used. C++ only does this for class/struct scopes. For example: int bar() { foo(); } int foo() { bar(); } compiles and runs (and runs, and runs, and runs!!!).
  - jjmarr 1y ago
    > D does not require names in global scope to be declared lexically before they are used. C++ only does this for class/struct scopes. But how do you handle a template substitution failure? In C++: template<typename T> auto bar(T x, T y) { return x + y;} The compiler has no idea whether bar(1, 2); will compile unless it parses the full definition. I don't understand how the compiler can avoid parsing the full definition. The expensive bit in my experience isn't parsing the declaration, it's parsing the definition. Typically redundantly over thousands of source files for identical types.

Calavar 1y ago

Did anyone reach out to you for input during the modules standardization process? D seems like the most obvious prior art, but the modules standardization process seems like it was especially cursed

WalterBright 1y ago
Nobody from C++ reached out to me for the modules. Herb Sutter, Andrei Alexandrescu and myself once submitted an official proposal for "static if" for C++, based on the huge success it has had in D. We received a vehement rejection. It demotivated me from submitting further proposals. ("static if" replaces the C preprocessor #if/#ifdef/#ifndef constructions.) C++ has gone on to adopt many features of D, but usually with modifications that make them less useful.
- jjmarr 1y ago
  static if was more or less added in C++17 under the name `if constexpr`. It's not exactly the same since the discarded statement is still checked if not dependent on a template, but like most things in C++, it's similar enough to footgun yourself.
  - sfpotter 1y ago
    "if constexpr" introduces a new scope, while "static if" does not. A major divergence, enough to make the features very, very different in terms of what they can actually be used for.
    - WalterBright 1y ago
      That was a giant mistake on C++'s part, as you cannot do this: static if (feature) { int bar() { betty(); }} ... lots of code ... static if (feature) bar(); Forcing a new scope cuts the utility of it about in half, and there's no way around it. But if you need a scope with D's static if: static if (expression) {{ int x; foo(x); }} the extra { } will do it. But, as it turns out, this is rarely desirable.

cogman10 1y ago

> just finish the job and relegate the preprocessor to the dustbin. Yup, I think this is the core of the problem with C++. The standards committee has drawn a bad line that makes encoding the modules basically impossible. Other languages with good module systems and fast incremental builds don't allow for preprocessor style craziness without some pretty strict boundaries. Even languages that have gotten it somewhat wrong (such as rust with it's proc macros) have bound where and how that sort of metaprogramming can take place. Even if the preprocessor isn't dustbined, it should be excluded from the module system. Metaprogramming should be a feature of the language with clear interfaces and interactions. For example, in Java the annotation processor is ultimately what triggers code generation capabilities. No annotation, no metaprogramming. It's not perfect, but it's a lot better than the C/C++'s free for all macro system. Or the other option is the go route. Don't make the compiler generate code, instead have the build system be responsible for code generation (calling code generators). That would be miles better as it'd allow devs to opt in to that slowdown when they need it.

will4274 1y ago
I actually think this attitude is the reason modules are so slowly adopted. The committee did exactly what you suggest and relegated the preprocessor to the dustbin and modules are seeing slow adoption. The truth is 98% of the preprocessor is fine - it's ifdefs for platforms and defines of constants and inline functions that are defined exactly once and never redefined. Because modules supports none of this, that means we can't modulize Windows.h. Or zlib. Or gtest. The committee should have remembered that one of the selling points of C++ is C compatibility, and figured out a way to get modules to work with 98% or the preprocessor and forbid only the nasty 2%.
- cogman10 1y ago
  > The committee did exactly what you suggest and relegated the preprocessor to the dustbin Importantly, no they did not. They put a boundary on the preprocessor containing preprocessing within the module definition and not the code importing the module. And that's where a lot of the loss and compatibility problems have come into play. That's why you can't, for example, share a module between builds. Because the ifdefs that built the module in the first place may have changed from one build to the next. It was good that the committee bound the preprocessor, but they simply didn't go far enough. C++ is making strides to adding the language features needed to dustbin modules. A lot of the work of consteval can replace a lot of what the preprocessor is doing. > Because modules supports none of this, that means we can't modulize Windows.h. Or zlib. Or gtest. And see, that's the issue. Modules do actually support all this. We can in fact modularize windows.h, zlib, or gtest. The issue is the `windows.module` still has to be rebuilt with every application that imports it because those `ifdefs` could evaluate differently depending on what env variables the build system sets before building. The module can't be just a simple AST built once. Maybe once for a project, but that's about it. And that's the rub. Change anything that causes the module to recompile and you spend exactly the same time you'd spend on precompiled headers.

gary_0 1y ago

I can't think of a C++ project I've worked on that didn't rely on being able to include C headers and have things usually just work. Are there ways of banning C macros from "modular" C++ without breaking that? (Many would find it unacceptable if you had to go through every C dependency and write/generate some sort of wrapper.)

WalterBright 1y ago
D resolved this problem by creating D versions of the C system headers. Yes, this was tedious, but we do it for each of our supported platforms. But we can't do it for various C libraries. This created a problem for us, as it is indeed tedious for users. We created a repository where people shared their conversions, but it was still inadequate. The solution was to build a C compiler into the D compiler. Now, you can simply "import" a C .h file. It works surprisingly well. Sure, some things don't work, as C programmers cannot resist put some really crazy stuff in the .h files. The solution to that problem turned out be we discovered that the D compiler was able to create D modules from C code. Then, the user could tweak by hand the nutburger bits.
- fooker 1y ago
  > The solution was to build a C compiler into the D compiler. This is the same solution that Apple chose for Swift <-> Objective C interop. I wonder if someone at Apple was inspired by this decision in D!

agent327 1y ago

C++ modules aren't influenced by macros on import, nor do they export any macros, so I'm curious what the problem is?

repstosb 1y ago

I've often wondered how the evolution of C and C++ might have been different if a more capable preprocessor (in particular, with more flexible recursive expansion and a better grammar for pattern matching) had caught on. The C++ template engine can be used to work around some of those limits, but always awkwardly, not least due to the way you need Knuth's arrow notation to express the growth in compiler error message volume with template complexity. By the time C++ came out we already had tools like m4 and awk with far more capability than cpp. It's pretty ridiculous that everything else about computing has radically changed since 1970 except the preprocessor and its memory-driven constraints.