32 ms·

Trying out C++20's modules with Clang and Make

90 points by vmsp 3y ago

maccard 3y ago
I'm super disappointed with modules. It's 2023, we've pushed out a new c++ standard since modules were standardised, and they're not usable. They're not supported in cmake (unless you set CMAKE_EXPERIMENTAL_CXX_MODULE_CMAKE_API to 2182bf5c-ef0d-489a-91da-49dbc3090d2a if you're on the same version of cmake as I am). My experience is that no IDE's support them properly, so either you need to ifdef out for intellisense, or go back to no tooling support. The format breaks every update, so you can't distribute precompiled modules. Last but definitely not least, they're slower than headers. So, we've got a feature that we started talking about 11 years ago, standardised 3 years ago, not implemented fully, has no build system support, and is a downgrade over the existing solution from every real world use case I've seen. What a mess.
- Davidbrcz 3y ago
  That's the pinacle of C++ development !
- shrimp_emoji 3y ago
  It's still brand new, in C++ feature terms. I'm not surprised by, and would totally expect, all of the problems you cited.
  - maccard 3y ago
    I think we've been talking about this and trying to implement it for long enough that "it's still new" stopped being an excuse 2 years ago.
- vblanco 3y ago
  They are not slower than headers. Ive been looking into it because modular STL is such a big win. On my little toy project i have .cpp files compiling in 0.05 seconds while doing import std. Downside is that at the moment you cant mix normal header STL with module STL in the same project (msvc), so its for cleanroom small projects only. I expect the second you can reliably use that almost everyone will switch overnight just from how fast of a speed boost it gives on the STL vs even precompiled headers.
  - cwzwarich 3y ago
    The one way in which they are slower than headers is that they create longer dependency chains of translation units, whereas with headers you unlock more parallelism at the beginning of the build process, but much of it is duplicated work.
  - klipt 3y ago
    I assume templates can only be partly preprocessed (parsed?) but not fully pre compiled, since final code depends on the template types?
    - vblanco 3y ago
      Yes, but template code is all on headers, so it gets parsed every single time its included on some compile unit. With modules this only happens once so its a huge speed upgrade in pretty much all cases.
      - moregrist 3y ago
        Whenever I’ve profiled compile times, parsing accounts for relatively little of the time, while the vast majority of the time is spent in the optimizer. So at least for my projects it’s a modest (maybe 10-20%) speed up, not the order of magnitude speed up I was hoping for. Thus C++ compile times will remain abysmal.
        dagmx 3y ago
        For some template heavy code bases I’ve been in, going to PCH has cut my compile times to less than half. I assume modules will have a similar benefit in those particular repositories, but obviously YMMV
    - jcelerier 3y ago
      Depends on the compiler, clang is able to pre-instantiate templates and generate debug info as part of its pch system - (for instance most likely you have some std::vector<int> which can be instantiated somewhere in a transitively included header). In my projects enabling the relevant flags gave pretty nice speedups.
- IAmLiterallyAB 3y ago
  The progress is frustratingly slow. My understanding is GCC and Clang still haven't finished implementing them fully. Last I read they are still making significant changes, for example moving to the strong ownership model for symbols. Once they're done, hopefully the build system and IDE support will follow quickly. MSVC seems to be in much better shape.
  - humanrebar 3y ago
    The progress is mostly unfunded. It's not that the work is especially slow (maybe it is a little?). Mostly it's that barely anyone is working on it. This is especially frustrating for organizations ostensibly paying vendors for high quality compilers.
    - Gibbon1 3y ago
      It's frustrating because consider how many hours are lost to compile times over the whole industry. I have no idea how many people are slopping C++ code. So lets just calculate it per 1000. Assume modules save an average of 10 minutes a day. 1000 X 250 days/year X 0.167 hours/day -> 41,666 hours a year per thousand coders. Or 21 man years. Yeah you'd think it'd be worth funding heavily.
      - kccqzy 3y ago
        When you use a good build system like Bazel modules do not save 10 minutes a day. The time saving is negligible. That's why large C++ shops do not care about modules and only volunteers are working on them.
        maccard 3y ago
        Not everyone uses bazel. I would wager that a vanishingly small amount of people use bazel. If people using other things was enough to not let things be standardized we wouldn't have asio, ranges, fmt. Precompiled headers also exist, so why would we standardise modules?
- aseipp 3y ago
  Module support will be stabilized in CMake 3.28, actually: https://www.reddit.com/r/cpp/comments/16y9qv2/cmake_c_modules_support_in_328/ https://www.reddit.com/r/cpp/comments/16y9qv2/cmake_c_module... The other points (mainly integration points) are definitely valid though.
  - maccard 3y ago
    That's really great to hear. Having proper build system support will finally let us use this in anger and report actual issues upstream.
    - mathstuf 3y ago
      Please also report relevant issues to CMake itself. We can't fix what we don't hear about :) . FD: CMake developer.
      - maccard 3y ago
        I absolutely will. Im usually not cutting edge enough to catch stuff, by the time I find it it's usually already reported.
- humanrebar 3y ago
  > I'm super disappointed with modules. It's 2023, we've pushed out a new c++ standard since modules were standardised, and they're not usable. Important question: Who are you disappointed in? Keep in mind that the ISO committee does not employ engineers and for international treaty reasons, it reasonably cannot. Point being, who should be doing this work? Are you talking to them about your expectations? Are you providing actual support to implementations that are public goods (i.e., open source compilers)?
  - david2ndaccount 3y ago
    The ISO committee should not be in the business of inventing language features out of whole cloth. That is not how “standardization” works. They should only standardize existing practice.
    - deleted 3y ago
      [deleted]
    - rewmie 3y ago
      > The ISO committee should not be in the business of inventing language features out of whole cloth. That is not how “standardization” works. They should only standardize existing practice. How do you expect to standardize a feature that does not exist yet but the whole community is demanding? It sounds like the C++ standardization committee designed a feature following a process where all stakeholders had a say. Is this worse than being force-fed a vendor-specific solution?
      - david2ndaccount 3y ago
        I do not expect standardization of features that do not exist.
        rewmie 3y ago
        > I do not expect standardization of features that do not exist. But that makes absolutely no sense, doesn't it? I mean, think about it for a moment. Specifying a standard behavior is not about picking a flawed winner from a list of ad-hoc implementation or settle with the least common denominator. Specifying a standard behavior is about providing the best possible solution that meets the design requirements following the input from the whole community.
- IshKebab 3y ago
  My feeling is that the desire for it has somewhat waned because so many people that care about the elegance of programming languages and use C++ have just moved to Rust. There are still plenty of people using C++ of course, but it certainly feels like more of a dead end than it did before Rust. Why bother putting a mountain of effort into maybe slightly improving it when in 10-20 years you won't be using it anyway?
  - deterministic 3y ago
    The number of Rust developers is a drop in the ocean compared with C++ developers. There are more than 5 million C++ developers out there starting new C++ projects every single day. I am one of them.
    - steveklabnik 3y ago
      Not that I disagree that there are a large number of C++ programmers out there, but where did you get that number?
    - IshKebab 3y ago
      Yeah obviously that's true now because C++ has been around for literally decades and had basically 0 competition for all that time.
      - deterministic 3y ago
        And it will continue to be true for a very long time since the world runs on software written in C and C++ and that needs to be maintained and improved.
- baybal2 3y ago
  [dead]
- matheusmoreira 3y ago
  > The format breaks every update, so you can't distribute precompiled modules. When will they fix this? People keep using C to this day because these unstable binary interfaces make it so no one can reuse software. Can't write a library in almost any other language and expect it to just work. To this day, C is the lowest common denominator of software because of stuff like this.
  - mathstuf 3y ago
    Unlikely to be fixed. The module files are basically AST dumps and no compiler keeps that stable over time. Even flag selections make incompatible BMIs. So if one TU uses C++20 and another uses C++23, they'll each need their own BMI of anything they import. FD: CMake developer that implemented the modules support
    - maccard 3y ago
      If module files are pretty much memory dumps that makes it all the more frustrating that it's taken this long, given thats what precompiled headers essentially are.
      - mathstuf 3y ago
        There are new rules about "reachability" and "module linkage". Not to mention things like methods defined within the class declaration are no longer as-if `inline` when within a module. It's not just "a faster preprocessor" mode; there are real rules associated with module boundaries.
  - leni536 3y ago
    C++ has stable binary interfaces just like C.
    - matheusmoreira 3y ago
      I've seen compiled C++ code turn out to be incompatible with code produced by different versions of the same compiler. There is no way this is stable.
wly_cdgr 3y ago
Unfortunate direction. Header files are great and I wish modern languages would adopt them.
- tgv 3y ago
  I'm glad other languages solved that in another way. With C(++) headers, you pick the declarations at compile time, and the implementation at link time. That's a disaster waiting to happen, and unpleasant to unravel.
  - flohofwoe 3y ago
    It's almost impossible to get wrong though, because the same header will also be included in its own implementation. If there's a mismatch between the declaration and implementation you'll get at least compiler warnings.
    - tgv 3y ago
      It’s been a while, but IIRC there is no problem when you eg. change a struct, as long as the name is the same.
      - mathstuf 3y ago
        Indeed. Mismatching headers and libraries is not all that rare. I've seen headers found from Homebrew, libraries from the macOS SDK, and some related tool from `/usr`. Things…rarely work out well, but you only find out in your test suite (or linker if you're really lucky).
        flohofwoe 3y ago
        The only realword problem with headers I've seen so far is when there are multiple versions of the same header in the same project (bad idea anyway), and one isn't extremely careful about search paths. The module equivalent would be different versions of the same module under the same import name. Can the C++ module system actually deal with that?
evrimoztamur 3y ago
I think the funniest part is that you have to add ‘module;’ at the start, as if the extension doesn’t make that clear, or that the file exports the module explicitly with ‘export module mod1;’ You guys OK in C++ land?
- dagmx 3y ago
  C++ file extensions aren’t standardized unlike other languages. Hence the need for keywords to enable functionality. So you can’t really rely on extensions to be sufficient. Many build systems and compilers do assume specific extensions imply a file type by default, but those can often be configured to arbitrary extensions as well.
  - IshKebab 3y ago
    Existing C++ file extensions aren't standardised. There's no reason they couldn't mandate file extensions for this. And really even among the non-standard extensions it's like 95% .cpp, 4.9% .cc and 0.1% .c++. Ignoring the weirdos it's de facto .cpp or .cc. They could standardise that too if they really wanted and there was a benefit. This sounds like one of those things that often happens where people worry about technically possible but totally insignificant risks, and insist on handling them because they feel proud of having thought of them. Not enough YAGNI.
    - Conscat 3y ago
      What .ii for preprocessed files? And .ipp and .tpp for template implementation files?
    - groos 3y ago
      You forgot .cxx
    - dagmx 3y ago
      You’re missing cxx and mm, which are more common in codebases I deal with than most of your examples. Which is to say you can’t make sweeping statements. Anyway, I’d try and contextualize your argument more by adding that feature teams are restricted in the scope of the changes they can do to the features they’re working on. They can certainly suggest that the language specify specific extensions but that goes against decades of convention, and is a hill that would likely have taken more of their time and energy for little specific gain for the feature itself. I’m not arguing that there shouldn’t be standard extensions, just that I can understand why it didn’t happen. It’s like swimming upstream
mgaunard 3y ago
Modules is another of those features that were added to C++ due to peer pressure from other programming languages. The thing is that they're heavily implementation-specific things. Two of the biggest compiler vendors pushed the model that worked best for their implementation. The Microsoft model won the committee politics, which is why they're the only ones with an implementation. Too bad most C++ developers don't really use Microsoft software to begin with. They should have just stuck with a simple PCH-but-you-don't-have-to-include-it-first model. That would have been straightforward and immediately useful to everybody.
- humanrebar 3y ago
  MSVC seems to have the best funding model of the three major compilers. I estimate that has more to do with adoption velocity than head starts on implementation, especially given how long modules have been in progress.
  - dagmx 3y ago
    MSVC is also the least expansive of the three compilers. It has a much smaller set of languages it supports (No C, ObjC etc for example), and a much lower focus on performance [1] That’s not a criticism necessarily , but just to add that I think they likely have a more agile code base as a result. [1] https://reddit.com/r/cpp/s/BOj34UtKvu https://reddit.com/r/cpp/s/BOj34UtKvu
    - logicchains 3y ago
      Plus its stdlib is developed by STL, the most fittingly-named guy to be developing a C++ standard library.
  - mgaunard 3y ago
    As I said, the module system in C++20 is literally the Microsoft model. Clang had another model that was rejected, and most of the years in committee were actually about finding a compromise between Microsoft's and Clang's approach. In the end, Clang wasn't particularly happy about the end result either.
jll29 3y ago
Has the C++ standardization committee given a rationale why a module definition does not automatically also create a namespace with the same name? That would seem such a useful feature. Can mobiles be nested? (I'm thinking java.util.StringTokenizer.) If so, has a module hierarchy (as for Java) been defined for all STL standard functionality? Are their any compilers that implement C++20 in full in 2023?
- adjav 3y ago
  According to cppreference (https://en.cppreference.com/w/cpp/compiler_support/20 https://en.cppreference.com/w/cpp/compiler_support/20), MSVC has everything implemented. GCC is close but its modules support remains lacking.
- halflings 3y ago
  > Has the C++ standardization committee given a rationale why a module definition does not automatically also create a namespace with the same name? That would seem such a useful feature. I'm not sure what the official rationale is, but I feel that would cause great namespace pollution. Some companies use one namespace per "logical" unit of code (e.g. a subproject), not for each small piece of code (e.g. a small module implementing a couple of classes.
  - mathstuf 3y ago
    I'm speculating, but I think that if one enforced "module is a namespace", projects like Qt, standard libraries, and other libraries would be forced to ignore them for far longer than anyone wants due to ABI stability requirements. If `#include <vector>` got you a different `std::vector` than `import std;`, I don't know how you would get everything that passes a vector over an API boundary ported without doing it all-at-once. Not to mention that if it were the case, things like a "Boost incubator" library would be in a separate top-level namespace as it could not "inject" itself into `import boost;`.
- humanrebar 3y ago
  Modules adoption would be severely stunted if everyone had to adjust all calling code in order to convert a library to use modules. Ideally one could find/replace includes with analogous imports, a bit at a time. I guess we could instead mandate that module names have to match the namespaces inside (i.e., mandate what modules are named up front), but it's also common to have C++ namespaces be reopened for various reasons (like adding to overload sets in supported ways). Anyway, bottom line is that modules and namespaces need to have M:N relationships, though in practice, it should be Bad Practice to get too creative there. Nobody has added namespace/module matching rules to CppCoreGuidelines or clang-tidy yet, but right now is a good time for some ideas in that space.
swingingFlyFish 3y ago
I'm thinking from Python here so somebody clarify if I'm missing something. "As far as I know, header files came to be because when C was built disk space was at a prime and storing libraries with their full source code would be expensive. Header files included just the necessary to be able to call into the library while being considerably smaller." The modules/packages that were built from C/C++ for example in Python and maybe other scripting languages those were all compiled with the inclusion of the .h files. for exactly the reason that the OP states...for efficiency. Header files should really only include definitions, and when you compile it's tight code using only what you included. Wouldn't modules include everything? isn't that like saying: import * as opposed to say: Import module.package I'm just confused how this will help statically typed languages like C/C++ into maintaining and compiling to very efficient machine or executables. Someone enlighten me.
- dagmx 3y ago
  Modules don’t import everything. In Python, import is an act of bringing in everything into the namespace (under its own or otherwise). In compiled languages, it is the act of making it available within this compilation unit. Things that are unused are stripped out and only what you either use or re-export have an effect. That is of course for static binaries, for dynamic linkage none of it really matters. In fact modules can be more efficient than headers, because the way C/C++ work is that everything within the header file is verbatim copied into the file referencing it. This isn’t true with modules.
- knome 3y ago
  >"As far as I know, header files came to be because when C was built disk space was at a prime and storing libraries with their full source code would be expensive. Header files included just the necessary to be able to call into the library while being considerably smaller." This claim doesn't make sense to me. You don't want and can't include the implementation in multiple C compilation units because the symbols from them would conflict. I suppose if you include the existence of shared libraries as a means of saving disk space the assertion isn't wholly wrong, but I've never see anyone make such a claim before. Headers exist even within a C project so that the various parts of it can share the function prototypes and struct definitions while being independently compiled.
adjav 3y ago
Clang 18 supports `import std;` now, but you need to enable some build settings [1]. Also has `deducing this` support, which is nice - I am fully in favour of C++ continuing to poach the good parts of Rust. [1] https://libcxx.llvm.org/Modules.html https://libcxx.llvm.org/Modules.html
- Conscat 3y ago
  Deducing this is not a feature analogous to anything in Rust, the language with almost no type deduction at all. It bears a vague similarity to constrained generic traits, but you really have to squint to see it.
  - dureuill 3y ago
    Of course you're correct that Rust has no support for making the `Self` type itself generic, which is the core of the "deducing this" feature. However, "deducing this" has the side-effect of allowing to explicitly specify the "this" parameter in the argument list. The syntax looks the same as Rust, and the convention of calling this parameter "self" is taken from "other languages" (Python, Rust, ...)[0]. Similarly, it could be argued that the ability to pass the object by value in method is lifted from Rust. That's how I understand the GP comment anyway. [0]: https://devblogs.microsoft.com/cppblog/cpp23-deducing-this/ https://devblogs.microsoft.com/cppblog/cpp23-deducing-this/ "You don’t have to use the names Self and self, but I think they’re the clearest options, and this follows what several other programming languages do."
- 3836293648 3y ago
  The good part of rust isn't anything they added, it's everything that's gone
delta_p_delta_x 3y ago
> make No, please. If you're using C++20 modules, you also owe it to yourself to use a modern build system... Edit: it seemed like I fired right into a Makefile-shaped hornet's nest.
- kova12 3y ago
  There isn't one for c++ unfortunately
  - humanrebar 3y ago
    Don't know what qualifies as a Scotsman to you, but CMake is more modern than Makefile.
    - delta_p_delta_x 3y ago
      Yep, CMake. It has a weird string-typed syntax, and an even weirder function/function parameter syntax, but those are the most egregious issues. Target-based CMake is extremely straightforward to get started with, and CMake + Ninja is also frequently significantly faster than the alternatives (autoconf tools). It also easily hooks into package managers for C++ (Conan, vcpkg) and today, users can write C++ like they do JS or Rust: have a `vcpkg.json` file, top-level `CMakeLists.txt` file, and that's it. I've worked enough with Autotools to understand how ridiculously painful they are. Look at this monstrosity[1]. CMake may be hard to get started with, but it's easy to append, maintain, and refactor code that builds with CMake. [1]: https://en.wikipedia.org/wiki/GNU_Autotools#/media/File:Autoconf-automake-process.svg https://en.wikipedia.org/wiki/GNU_Autotools#/media/File:Auto...
    - duped 3y ago
      Don't use make! Use a makefile generator! (hope the sarcasm is obvious) CMake is pretty good, so is Meson, so is Conan. All of them have some unfortunate shortcomings that make the C++ ecosystem hard to work in, but the biggest problem is diversity of build systems for dependencies.
      - 3y ago
malkia 3y ago
We had this working: Turbo Pascal Units....
- CoastalCoder 3y ago
  I'm guessing that adding modules to a simple, clean language like Pascal is a lot easier that making almost any change to C++, the epitome of language complexity.
boris 3y ago
The Makefile in the article is actually incorrect and only "works" by accident: main.o must have a dependency on mod1.pcm which is one of the outputs produced when compiling mod1.ccm and is needed whenever compiling any transaction unit that imports mod1.
humanrebar 3y ago
Note that module files (.ccm files in the example) cannot be parsed in parallel like that. The build system has to know what order to parse module files in. I guess if all .ccm files only did #includes and no imports, the provided Makefile example would work, but that seems like a niche use case. Instead, some sort of dependency graph amongst the .ccm needs to be encoded in the Makefiles. There is an ISO paper describing how to query a compiler for this info (see https://wg21.link/p1689 https://wg21.link/p1689). Other papers sketch out how one might incorporate that into a dynamic Makefile graph, but so far nobody seems interested in going that far. Switching to a more actively maintained build system is probably the interesting alternative. If you want to learn more about how build systems can support C++ modules, see this blog post by the CMake maintainers. The principles translate to most (all?) nontrivial build systems. https://www.kitware.com/import-cmake-c20-modules https://www.kitware.com/import-cmake-c20-modules
up2isomorphism 3y ago
What’s the problem with header files? Also header files exists not just because of resource constraints.
- trealira 3y ago
  One good thing that comes from it is that templated code can now go into a module and be compiled only once. Before, you had to put template classes into headers, so you effectively had to recompile, for example, the class std::vector<T>, for each file you included the header <vector> in, because a new compiler process compiles each .cpp file from scratch, knowing nothing. If two separate files instantiated std::vector<int>, then it would be instantiated and defined separately at compile time, and at link time, one of the definitions would be discarded. The only way to speed it up before modules was to use pre-compiled headers, which are compiler-specific. Modules let the compiler share state across files, so it doesn't have to recompile template instantiations unnecessarily. I think this guy's blog explains it well: https://blog.ecosta.dev/en/tech/explaining-cpp20-modules https://blog.ecosta.dev/en/tech/explaining-cpp20-modules
  - maccard 3y ago
    Extern templates have existed since c++11 for that. > The only way to speed it up before modules was to use pre-compiled headers, which are compiler-specific. Modules are compiler specific too, unfortunately.
    - trealira 3y ago
      >Extern templates have existed since c++11 for that. True, but ar least you don't need to write extern template class SomeTemplateClass<SomeType> in the header, and then explicitly instantiate it in the source. You can just write "export template class SomeTemplateClass<typename T> { ... };" (i.e., "export" followed by the definition) in the module file. > Modules are compiler specific too, unfortunately. Ah, I didn't realize, my bad.
      - 3y ago
miki123211 3y ago
Is there any specific reason why we should prefer a module system as it exists in C++ over something like the SQLite's approach with Makeheaders[1]? As far as I understand it, they have a tool that can parse all their (C) files, figure out what each file declares and what declarations it depends on, and then autogenerate a header for every file with just the declarations it requires. It seems like a sensible approach, but nobody besides sQLite and Fossil (which comes from the same developers) seems to be using it, either in C or C++, which leads me to suspect that there are caveats I don't know about. Is there a reason why this is a bad idea? [1] https://fossil-scm.org/home/doc/trunk/tools/makeheaders.html https://fossil-scm.org/home/doc/trunk/tools/makeheaders.html
- mathstuf 3y ago
  Probably because things that don't get "used" can still affect things. There's a lot of detail in name lookup algorithms and SFINAE where computing what you end up using and giving each TU its own view is probably not much savings over using the headers as-is.
- david2ndaccount 3y ago
  This works for C, but not C++. And for C, it’s not that hard to do by hand so no one else bothers.
mathstuf 3y ago
FD: I'm a member of SG15, author of P1689[1], and implemented much of the support for compiling them in CMake. It is my opinion that hand-coded Makefiles are unlikely to handle modules well. They have the same required strategy as Fortran modules where the (documented!) approach by Intel was to run `make` in parallel until it works (basically using the `make` scheduler to get the TU compilation order eventually). Of course, there's no reliable way to know if you found a stale module or anything like that if there's a cycle or a module rename leaving old artifacts around. There is the `makedepf90` tool that could do the "dynamic dependencies" with a fixed-level recursive Makefile to order the compilations in the right order (though I think you may be able skip the recursive layer if you have no sources generated by tools built during the build). Anyways, this strategy fails if you want implementation-only modules, modules of the same name (that don't end up in the same process space; debug and release builds in a single tree are the more common example though), use modules from other projects (as you need to add rules to compile their module interfaces for your own build). [1] https://wg21.link/p1689 https://wg21.link/p1689
kazinator 3y ago
If you think your language has modules, but you are still building with make, there is something wrong. I programmed in Modula 2; that definitely didn't need make. The module definitions need to be parsed and semantically analyzed, so the compiler has to be the build tool.

90 points by vmsp 3y ago

I'm super disappointed with modules. It's 2023, we've pushed out a new c++ standard since modules were standardised, and they're not usable. They're not supported in cmake (unless you set CMAKE_EXPERIMENTAL_CXX_MODULE_CMAKE_API to 2182bf5c-ef0d-489a-91da-49dbc3090d2a if you're on the same version of cmake as I am). My experience is that no IDE's support them properly, so either you need to ifdef out for intellisense, or go back to no tooling support. The format breaks every update, so you can't distribute precompiled modules. Last but definitely not least, they're slower than headers. So, we've got a feature that we started talking about 11 years ago, standardised 3 years ago, not implemented fully, has no build system support, and is a downgrade over the existing solution from every real world use case I've seen. What a mess.

Davidbrcz 3y ago
That's the pinacle of C++ development !
shrimp_emoji 3y ago
It's still brand new, in C++ feature terms. I'm not surprised by, and would totally expect, all of the problems you cited.
- maccard 3y ago
  I think we've been talking about this and trying to implement it for long enough that "it's still new" stopped being an excuse 2 years ago.
vblanco 3y ago
They are not slower than headers. Ive been looking into it because modular STL is such a big win. On my little toy project i have .cpp files compiling in 0.05 seconds while doing import std. Downside is that at the moment you cant mix normal header STL with module STL in the same project (msvc), so its for cleanroom small projects only. I expect the second you can reliably use that almost everyone will switch overnight just from how fast of a speed boost it gives on the STL vs even precompiled headers.
- cwzwarich 3y ago
  The one way in which they are slower than headers is that they create longer dependency chains of translation units, whereas with headers you unlock more parallelism at the beginning of the build process, but much of it is duplicated work.
- klipt 3y ago
  I assume templates can only be partly preprocessed (parsed?) but not fully pre compiled, since final code depends on the template types?
  - vblanco 3y ago
    Yes, but template code is all on headers, so it gets parsed every single time its included on some compile unit. With modules this only happens once so its a huge speed upgrade in pretty much all cases.
    - moregrist 3y ago
      Whenever I’ve profiled compile times, parsing accounts for relatively little of the time, while the vast majority of the time is spent in the optimizer. So at least for my projects it’s a modest (maybe 10-20%) speed up, not the order of magnitude speed up I was hoping for. Thus C++ compile times will remain abysmal.
      - dagmx 3y ago
        For some template heavy code bases I’ve been in, going to PCH has cut my compile times to less than half. I assume modules will have a similar benefit in those particular repositories, but obviously YMMV
  - jcelerier 3y ago
    Depends on the compiler, clang is able to pre-instantiate templates and generate debug info as part of its pch system - (for instance most likely you have some std::vector<int> which can be instantiated somewhere in a transitively included header). In my projects enabling the relevant flags gave pretty nice speedups.
IAmLiterallyAB 3y ago
The progress is frustratingly slow. My understanding is GCC and Clang still haven't finished implementing them fully. Last I read they are still making significant changes, for example moving to the strong ownership model for symbols. Once they're done, hopefully the build system and IDE support will follow quickly. MSVC seems to be in much better shape.
- humanrebar 3y ago
  The progress is mostly unfunded. It's not that the work is especially slow (maybe it is a little?). Mostly it's that barely anyone is working on it. This is especially frustrating for organizations ostensibly paying vendors for high quality compilers.
  - Gibbon1 3y ago
    It's frustrating because consider how many hours are lost to compile times over the whole industry. I have no idea how many people are slopping C++ code. So lets just calculate it per 1000. Assume modules save an average of 10 minutes a day. 1000 X 250 days/year X 0.167 hours/day -> 41,666 hours a year per thousand coders. Or 21 man years. Yeah you'd think it'd be worth funding heavily.
    - kccqzy 3y ago
      When you use a good build system like Bazel modules do not save 10 minutes a day. The time saving is negligible. That's why large C++ shops do not care about modules and only volunteers are working on them.
      - maccard 3y ago
        Not everyone uses bazel. I would wager that a vanishingly small amount of people use bazel. If people using other things was enough to not let things be standardized we wouldn't have asio, ranges, fmt. Precompiled headers also exist, so why would we standardise modules?
aseipp 3y ago
Module support will be stabilized in CMake 3.28, actually: https://www.reddit.com/r/cpp/comments/16y9qv2/cmake_c_modules_support_in_328/ https://www.reddit.com/r/cpp/comments/16y9qv2/cmake_c_module... The other points (mainly integration points) are definitely valid though.
- maccard 3y ago
  That's really great to hear. Having proper build system support will finally let us use this in anger and report actual issues upstream.
  - mathstuf 3y ago
    Please also report relevant issues to CMake itself. We can't fix what we don't hear about :) . FD: CMake developer.
    - maccard 3y ago
      I absolutely will. Im usually not cutting edge enough to catch stuff, by the time I find it it's usually already reported.
humanrebar 3y ago
> I'm super disappointed with modules. It's 2023, we've pushed out a new c++ standard since modules were standardised, and they're not usable. Important question: Who are you disappointed in? Keep in mind that the ISO committee does not employ engineers and for international treaty reasons, it reasonably cannot. Point being, who should be doing this work? Are you talking to them about your expectations? Are you providing actual support to implementations that are public goods (i.e., open source compilers)?
- david2ndaccount 3y ago
  The ISO committee should not be in the business of inventing language features out of whole cloth. That is not how “standardization” works. They should only standardize existing practice.
  - deleted 3y ago
    [deleted]
  - rewmie 3y ago
    > The ISO committee should not be in the business of inventing language features out of whole cloth. That is not how “standardization” works. They should only standardize existing practice. How do you expect to standardize a feature that does not exist yet but the whole community is demanding? It sounds like the C++ standardization committee designed a feature following a process where all stakeholders had a say. Is this worse than being force-fed a vendor-specific solution?
    - david2ndaccount 3y ago
      I do not expect standardization of features that do not exist.
      - rewmie 3y ago
        > I do not expect standardization of features that do not exist. But that makes absolutely no sense, doesn't it? I mean, think about it for a moment. Specifying a standard behavior is not about picking a flawed winner from a list of ad-hoc implementation or settle with the least common denominator. Specifying a standard behavior is about providing the best possible solution that meets the design requirements following the input from the whole community.
IshKebab 3y ago
My feeling is that the desire for it has somewhat waned because so many people that care about the elegance of programming languages and use C++ have just moved to Rust. There are still plenty of people using C++ of course, but it certainly feels like more of a dead end than it did before Rust. Why bother putting a mountain of effort into maybe slightly improving it when in 10-20 years you won't be using it anyway?
- deterministic 3y ago
  The number of Rust developers is a drop in the ocean compared with C++ developers. There are more than 5 million C++ developers out there starting new C++ projects every single day. I am one of them.
  - steveklabnik 3y ago
    Not that I disagree that there are a large number of C++ programmers out there, but where did you get that number?
  - IshKebab 3y ago
    Yeah obviously that's true now because C++ has been around for literally decades and had basically 0 competition for all that time.
    - deterministic 3y ago
      And it will continue to be true for a very long time since the world runs on software written in C and C++ and that needs to be maintained and improved.
baybal2 3y ago
[dead]
matheusmoreira 3y ago
> The format breaks every update, so you can't distribute precompiled modules. When will they fix this? People keep using C to this day because these unstable binary interfaces make it so no one can reuse software. Can't write a library in almost any other language and expect it to just work. To this day, C is the lowest common denominator of software because of stuff like this.
- mathstuf 3y ago
  Unlikely to be fixed. The module files are basically AST dumps and no compiler keeps that stable over time. Even flag selections make incompatible BMIs. So if one TU uses C++20 and another uses C++23, they'll each need their own BMI of anything they import. FD: CMake developer that implemented the modules support
  - maccard 3y ago
    If module files are pretty much memory dumps that makes it all the more frustrating that it's taken this long, given thats what precompiled headers essentially are.
    - mathstuf 3y ago
      There are new rules about "reachability" and "module linkage". Not to mention things like methods defined within the class declaration are no longer as-if `inline` when within a module. It's not just "a faster preprocessor" mode; there are real rules associated with module boundaries.
- leni536 3y ago
  C++ has stable binary interfaces just like C.
  - matheusmoreira 3y ago
    I've seen compiled C++ code turn out to be incompatible with code produced by different versions of the same compiler. There is no way this is stable.

wly_cdgr 3y ago

Unfortunate direction. Header files are great and I wish modern languages would adopt them.

tgv 3y ago
I'm glad other languages solved that in another way. With C(++) headers, you pick the declarations at compile time, and the implementation at link time. That's a disaster waiting to happen, and unpleasant to unravel.
- flohofwoe 3y ago
  It's almost impossible to get wrong though, because the same header will also be included in its own implementation. If there's a mismatch between the declaration and implementation you'll get at least compiler warnings.
  - tgv 3y ago
    It’s been a while, but IIRC there is no problem when you eg. change a struct, as long as the name is the same.
    - mathstuf 3y ago
      Indeed. Mismatching headers and libraries is not all that rare. I've seen headers found from Homebrew, libraries from the macOS SDK, and some related tool from `/usr`. Things…rarely work out well, but you only find out in your test suite (or linker if you're really lucky).
      - flohofwoe 3y ago
        The only realword problem with headers I've seen so far is when there are multiple versions of the same header in the same project (bad idea anyway), and one isn't extremely careful about search paths. The module equivalent would be different versions of the same module under the same import name. Can the C++ module system actually deal with that?

evrimoztamur 3y ago

I think the funniest part is that you have to add ‘module;’ at the start, as if the extension doesn’t make that clear, or that the file exports the module explicitly with ‘export module mod1;’ You guys OK in C++ land?

dagmx 3y ago
C++ file extensions aren’t standardized unlike other languages. Hence the need for keywords to enable functionality. So you can’t really rely on extensions to be sufficient. Many build systems and compilers do assume specific extensions imply a file type by default, but those can often be configured to arbitrary extensions as well.
- IshKebab 3y ago
  Existing C++ file extensions aren't standardised. There's no reason they couldn't mandate file extensions for this. And really even among the non-standard extensions it's like 95% .cpp, 4.9% .cc and 0.1% .c++. Ignoring the weirdos it's de facto .cpp or .cc. They could standardise that too if they really wanted and there was a benefit. This sounds like one of those things that often happens where people worry about technically possible but totally insignificant risks, and insist on handling them because they feel proud of having thought of them. Not enough YAGNI.
  - Conscat 3y ago
    What .ii for preprocessed files? And .ipp and .tpp for template implementation files?
  - groos 3y ago
    You forgot .cxx
  - dagmx 3y ago
    You’re missing cxx and mm, which are more common in codebases I deal with than most of your examples. Which is to say you can’t make sweeping statements. Anyway, I’d try and contextualize your argument more by adding that feature teams are restricted in the scope of the changes they can do to the features they’re working on. They can certainly suggest that the language specify specific extensions but that goes against decades of convention, and is a hill that would likely have taken more of their time and energy for little specific gain for the feature itself. I’m not arguing that there shouldn’t be standard extensions, just that I can understand why it didn’t happen. It’s like swimming upstream

mgaunard 3y ago

Modules is another of those features that were added to C++ due to peer pressure from other programming languages. The thing is that they're heavily implementation-specific things. Two of the biggest compiler vendors pushed the model that worked best for their implementation. The Microsoft model won the committee politics, which is why they're the only ones with an implementation. Too bad most C++ developers don't really use Microsoft software to begin with. They should have just stuck with a simple PCH-but-you-don't-have-to-include-it-first model. That would have been straightforward and immediately useful to everybody.

humanrebar 3y ago
MSVC seems to have the best funding model of the three major compilers. I estimate that has more to do with adoption velocity than head starts on implementation, especially given how long modules have been in progress.
- dagmx 3y ago
  MSVC is also the least expansive of the three compilers. It has a much smaller set of languages it supports (No C, ObjC etc for example), and a much lower focus on performance [1] That’s not a criticism necessarily , but just to add that I think they likely have a more agile code base as a result. [1] https://reddit.com/r/cpp/s/BOj34UtKvu https://reddit.com/r/cpp/s/BOj34UtKvu
  - logicchains 3y ago
    Plus its stdlib is developed by STL, the most fittingly-named guy to be developing a C++ standard library.
- mgaunard 3y ago
  As I said, the module system in C++20 is literally the Microsoft model. Clang had another model that was rejected, and most of the years in committee were actually about finding a compromise between Microsoft's and Clang's approach. In the end, Clang wasn't particularly happy about the end result either.

jll29 3y ago

Has the C++ standardization committee given a rationale why a module definition does not automatically also create a namespace with the same name? That would seem such a useful feature. Can mobiles be nested? (I'm thinking java.util.StringTokenizer.) If so, has a module hierarchy (as for Java) been defined for all STL standard functionality? Are their any compilers that implement C++20 in full in 2023?

adjav 3y ago
According to cppreference (https://en.cppreference.com/w/cpp/compiler_support/20 https://en.cppreference.com/w/cpp/compiler_support/20), MSVC has everything implemented. GCC is close but its modules support remains lacking.
halflings 3y ago
> Has the C++ standardization committee given a rationale why a module definition does not automatically also create a namespace with the same name? That would seem such a useful feature. I'm not sure what the official rationale is, but I feel that would cause great namespace pollution. Some companies use one namespace per "logical" unit of code (e.g. a subproject), not for each small piece of code (e.g. a small module implementing a couple of classes.
- mathstuf 3y ago
  I'm speculating, but I think that if one enforced "module is a namespace", projects like Qt, standard libraries, and other libraries would be forced to ignore them for far longer than anyone wants due to ABI stability requirements. If `#include <vector>` got you a different `std::vector` than `import std;`, I don't know how you would get everything that passes a vector over an API boundary ported without doing it all-at-once. Not to mention that if it were the case, things like a "Boost incubator" library would be in a separate top-level namespace as it could not "inject" itself into `import boost;`.
humanrebar 3y ago
Modules adoption would be severely stunted if everyone had to adjust all calling code in order to convert a library to use modules. Ideally one could find/replace includes with analogous imports, a bit at a time. I guess we could instead mandate that module names have to match the namespaces inside (i.e., mandate what modules are named up front), but it's also common to have C++ namespaces be reopened for various reasons (like adding to overload sets in supported ways). Anyway, bottom line is that modules and namespaces need to have M:N relationships, though in practice, it should be Bad Practice to get too creative there. Nobody has added namespace/module matching rules to CppCoreGuidelines or clang-tidy yet, but right now is a good time for some ideas in that space.

swingingFlyFish 3y ago

I'm thinking from Python here so somebody clarify if I'm missing something. "As far as I know, header files came to be because when C was built disk space was at a prime and storing libraries with their full source code would be expensive. Header files included just the necessary to be able to call into the library while being considerably smaller." The modules/packages that were built from C/C++ for example in Python and maybe other scripting languages those were all compiled with the inclusion of the .h files. for exactly the reason that the OP states...for efficiency. Header files should really only include definitions, and when you compile it's tight code using only what you included. Wouldn't modules include everything? isn't that like saying: import * as opposed to say: Import module.package I'm just confused how this will help statically typed languages like C/C++ into maintaining and compiling to very efficient machine or executables. Someone enlighten me.

dagmx 3y ago
Modules don’t import everything. In Python, import is an act of bringing in everything into the namespace (under its own or otherwise). In compiled languages, it is the act of making it available within this compilation unit. Things that are unused are stripped out and only what you either use or re-export have an effect. That is of course for static binaries, for dynamic linkage none of it really matters. In fact modules can be more efficient than headers, because the way C/C++ work is that everything within the header file is verbatim copied into the file referencing it. This isn’t true with modules.
knome 3y ago
>"As far as I know, header files came to be because when C was built disk space was at a prime and storing libraries with their full source code would be expensive. Header files included just the necessary to be able to call into the library while being considerably smaller." This claim doesn't make sense to me. You don't want and can't include the implementation in multiple C compilation units because the symbols from them would conflict. I suppose if you include the existence of shared libraries as a means of saving disk space the assertion isn't wholly wrong, but I've never see anyone make such a claim before. Headers exist even within a C project so that the various parts of it can share the function prototypes and struct definitions while being independently compiled.

adjav 3y ago

Clang 18 supports `import std;` now, but you need to enable some build settings [1]. Also has `deducing this` support, which is nice - I am fully in favour of C++ continuing to poach the good parts of Rust. [1] https://libcxx.llvm.org/Modules.html https://libcxx.llvm.org/Modules.html

Conscat 3y ago
Deducing this is not a feature analogous to anything in Rust, the language with almost no type deduction at all. It bears a vague similarity to constrained generic traits, but you really have to squint to see it.
- dureuill 3y ago
  Of course you're correct that Rust has no support for making the `Self` type itself generic, which is the core of the "deducing this" feature. However, "deducing this" has the side-effect of allowing to explicitly specify the "this" parameter in the argument list. The syntax looks the same as Rust, and the convention of calling this parameter "self" is taken from "other languages" (Python, Rust, ...)[0]. Similarly, it could be argued that the ability to pass the object by value in method is lifted from Rust. That's how I understand the GP comment anyway. [0]: https://devblogs.microsoft.com/cppblog/cpp23-deducing-this/ https://devblogs.microsoft.com/cppblog/cpp23-deducing-this/ "You don’t have to use the names Self and self, but I think they’re the clearest options, and this follows what several other programming languages do."
3836293648 3y ago
The good part of rust isn't anything they added, it's everything that's gone

delta_p_delta_x 3y ago

> make No, please. If you're using C++20 modules, you also owe it to yourself to use a modern build system... Edit: it seemed like I fired right into a Makefile-shaped hornet's nest.

kova12 3y ago
There isn't one for c++ unfortunately
- humanrebar 3y ago
  Don't know what qualifies as a Scotsman to you, but CMake is more modern than Makefile.
  - delta_p_delta_x 3y ago
    Yep, CMake. It has a weird string-typed syntax, and an even weirder function/function parameter syntax, but those are the most egregious issues. Target-based CMake is extremely straightforward to get started with, and CMake + Ninja is also frequently significantly faster than the alternatives (autoconf tools). It also easily hooks into package managers for C++ (Conan, vcpkg) and today, users can write C++ like they do JS or Rust: have a `vcpkg.json` file, top-level `CMakeLists.txt` file, and that's it. I've worked enough with Autotools to understand how ridiculously painful they are. Look at this monstrosity[1]. CMake may be hard to get started with, but it's easy to append, maintain, and refactor code that builds with CMake. [1]: https://en.wikipedia.org/wiki/GNU_Autotools#/media/File:Autoconf-automake-process.svg https://en.wikipedia.org/wiki/GNU_Autotools#/media/File:Auto...
  - duped 3y ago
    Don't use make! Use a makefile generator! (hope the sarcasm is obvious) CMake is pretty good, so is Meson, so is Conan. All of them have some unfortunate shortcomings that make the C++ ecosystem hard to work in, but the biggest problem is diversity of build systems for dependencies.
    - 3y ago

malkia 3y ago

We had this working: Turbo Pascal Units....

CoastalCoder 3y ago
I'm guessing that adding modules to a simple, clean language like Pascal is a lot easier that making almost any change to C++, the epitome of language complexity.

boris 3y ago

The Makefile in the article is actually incorrect and only "works" by accident: main.o must have a dependency on mod1.pcm which is one of the outputs produced when compiling mod1.ccm and is needed whenever compiling any transaction unit that imports mod1.

humanrebar 3y ago

Note that module files (.ccm files in the example) cannot be parsed in parallel like that. The build system has to know what order to parse module files in. I guess if all .ccm files only did #includes and no imports, the provided Makefile example would work, but that seems like a niche use case. Instead, some sort of dependency graph amongst the .ccm needs to be encoded in the Makefiles. There is an ISO paper describing how to query a compiler for this info (see https://wg21.link/p1689 https://wg21.link/p1689). Other papers sketch out how one might incorporate that into a dynamic Makefile graph, but so far nobody seems interested in going that far. Switching to a more actively maintained build system is probably the interesting alternative. If you want to learn more about how build systems can support C++ modules, see this blog post by the CMake maintainers. The principles translate to most (all?) nontrivial build systems. https://www.kitware.com/import-cmake-c20-modules https://www.kitware.com/import-cmake-c20-modules

up2isomorphism 3y ago

What’s the problem with header files? Also header files exists not just because of resource constraints.

trealira 3y ago
One good thing that comes from it is that templated code can now go into a module and be compiled only once. Before, you had to put template classes into headers, so you effectively had to recompile, for example, the class std::vector<T>, for each file you included the header <vector> in, because a new compiler process compiles each .cpp file from scratch, knowing nothing. If two separate files instantiated std::vector<int>, then it would be instantiated and defined separately at compile time, and at link time, one of the definitions would be discarded. The only way to speed it up before modules was to use pre-compiled headers, which are compiler-specific. Modules let the compiler share state across files, so it doesn't have to recompile template instantiations unnecessarily. I think this guy's blog explains it well: https://blog.ecosta.dev/en/tech/explaining-cpp20-modules https://blog.ecosta.dev/en/tech/explaining-cpp20-modules
- maccard 3y ago
  Extern templates have existed since c++11 for that. > The only way to speed it up before modules was to use pre-compiled headers, which are compiler-specific. Modules are compiler specific too, unfortunately.
  - trealira 3y ago
    >Extern templates have existed since c++11 for that. True, but ar least you don't need to write extern template class SomeTemplateClass<SomeType> in the header, and then explicitly instantiate it in the source. You can just write "export template class SomeTemplateClass<typename T> { ... };" (i.e., "export" followed by the definition) in the module file. > Modules are compiler specific too, unfortunately. Ah, I didn't realize, my bad.
    - 3y ago

miki123211 3y ago

Is there any specific reason why we should prefer a module system as it exists in C++ over something like the SQLite's approach with Makeheaders[1]? As far as I understand it, they have a tool that can parse all their (C) files, figure out what each file declares and what declarations it depends on, and then autogenerate a header for every file with just the declarations it requires. It seems like a sensible approach, but nobody besides sQLite and Fossil (which comes from the same developers) seems to be using it, either in C or C++, which leads me to suspect that there are caveats I don't know about. Is there a reason why this is a bad idea? [1] https://fossil-scm.org/home/doc/trunk/tools/makeheaders.html https://fossil-scm.org/home/doc/trunk/tools/makeheaders.html

mathstuf 3y ago
Probably because things that don't get "used" can still affect things. There's a lot of detail in name lookup algorithms and SFINAE where computing what you end up using and giving each TU its own view is probably not much savings over using the headers as-is.
david2ndaccount 3y ago
This works for C, but not C++. And for C, it’s not that hard to do by hand so no one else bothers.

mathstuf 3y ago

FD: I'm a member of SG15, author of P1689[1], and implemented much of the support for compiling them in CMake. It is my opinion that hand-coded Makefiles are unlikely to handle modules well. They have the same required strategy as Fortran modules where the (documented!) approach by Intel was to run `make` in parallel until it works (basically using the `make` scheduler to get the TU compilation order eventually). Of course, there's no reliable way to know if you found a stale module or anything like that if there's a cycle or a module rename leaving old artifacts around. There is the `makedepf90` tool that could do the "dynamic dependencies" with a fixed-level recursive Makefile to order the compilations in the right order (though I think you may be able skip the recursive layer if you have no sources generated by tools built during the build). Anyways, this strategy fails if you want implementation-only modules, modules of the same name (that don't end up in the same process space; debug and release builds in a single tree are the more common example though), use modules from other projects (as you need to add rules to compile their module interfaces for your own build). [1] https://wg21.link/p1689 https://wg21.link/p1689

kazinator 3y ago

If you think your language has modules, but you are still building with make, there is something wrong. I programmed in Modula 2; that definitely didn't need make. The module definitions need to be parsed and semantically analyzed, so the compiler has to be the build tool.