30 ms·

> Why would modules make it harder to interface with c++ from python ? Is it because the header files are a direct interface to the functions? Well, they're si

> Why would modules make it harder to interface with c++ from python ? Is it because the header files are a direct interface to the functions? Well, they're simpler, they're text, and the tooling already exists (I use swig all the time to automatically make C functions and symbols available to Java and Python. Some Lisps as well). The reason the tooling exists, is because they are simple, and they are text. Looking at the C++ module definition, it's unclear to me[1] that similar tooling for C++ modules will ever be possible. As long as the most popular languages (Python, Java, etc) rely on a C header file for inter-call interoperability, header files aren't going away. No one wants to manually write the interface into a C++ library by hand. [1] I'm no C++ expert though - the problems I see with modules includes things like lack of a standard to grab the interface from the compiled module, complexity of C++ to determine the interface from the actual source code, etc.

nicoburns 3y ago
> Well, they're simpler, they're text, and the tooling already exists (I use swig all the time to automatically make C functions and symbols available to Java and Python. Some Lisps as well). > The reason the tooling exists, is because they are simple, and they are text. Looking at the C++ module definition, it's unclear to me[1] that similar tooling for C++ modules will ever be possible. Perhaps there's something specific about C++'s module design that makes this difficult, but Rust has modules and has similar tooling to enable you to automatically generate bindings to other languages (including Python and Java).
Rusky 3y ago
Really there's nothing stopping tooling like swig (or a C++ compiler for that matter) from processing module interfaces as text, just like headers. The language spec for modules doesn't require (or even really talk about) compiling them to any kind of binary or intermediate format. Rather, the language spec aspect of modules is about symbol visibility and collisions. In terms that interop cares about, moving a declaration into a module just adjusts its name mangling, without changing much else about e.g. its presence in the object file.
- lelanthran 3y ago
  > Really there's nothing stopping tooling like swig (or a C++ compiler for that matter) from processing module interfaces as text, just like headers. Well the one hurdle is that C++ is a lot more complex to process than C. You'll note in my original comment upthread, I pointed out that the C++ libraries provide a C interface in their headers, and that is what tools like swig work with. They don't (and can't, because of symbol mangling) read C++-only headers. Since they can't do it for headers (because the resulting symbols differ from compiler to compiler), why would they be able to do it for modules? I'm of the understanding that C++ modules are unable to provide a C interface (`extern "C"` or similar), so it's kinda pointless reading them and generating a list of symbols, because then the generated stubs only work when the library is compiled with that specific compiler. I think that it would be nice if the compiler can be instructed to output header files for the modules it reads. You'll still have the problem that the header now has mangled symbols in it (because modules cannot have `extern "C"`), but at least tools can generate stubs for that specific instance of the compiled library. (PS. I'm happy to be corrected about the `extern "C"` thing - if I am wrong about that then, sure, tooling that scans C++ modules for `extern "C"` can in fact be made quite easily).
  - Rusky 3y ago
    You can write `extern "C"` declarations in modules, actually. And either way the complexity of C++ and its name mangling is independent of headers vs modules, no? Further, swig does seem to support quite a bit of C++. IIUC it even avoids dealing with name mangling by generating wrappers that provide a C interface.
    - lelanthran 3y ago
      > You can write `extern "C"` declarations in modules, actually. See, I did not know that :-) > And either way the complexity of C++ and its name mangling is independent of headers vs modules, no? You got me self-doubting here: C++ implementations are necessarily more complex than their interfaces. The full complexity of C++ (not restricted to only name-mangling) is not usually found in headers intended for consumption by a C-interface compatible consumer. The intention of headers is to serve as an interface alone, but absolutely nothing enforces that - nothing stops the developer from dumping the entire implementation into the header (and this is a popular route for some libraries that are header-only). Modules are not intended to serve as an interface alone, so it is more likely that devs (myself included) will simply throw the entire implementation into a module, because it seems like a better idea to do so with modules. IOW, common practice for headers is to only have the interface necessary to use the implementation, but I think that common practice for modules will be to have the implementation and interface included in a single module. > Further, swig does seem to support quite a bit of C++. IIUC it even avoids dealing with name mangling by generating wrappers that provide a C interface. Swig supports much of C++ in a smart way, by default. For things like demangling swig generates `extern "C"` wrappers (for functions, definitely - not so sure about class typenames and enum typenames). But, to generate those wrappers from modules requires the original source code for those modules, and to use generated C++ wrappers, it needs to be compiled with the same compiler: Not a large hurdle, but it is definitely an additional concern compared to using the headers. It will be interesting to see how this plays out: how long it takes until swig support for modules, or swig-like tooling for modules, comes along. It's still early days for modules, after all.

nicoburns 3y ago

> Well, they're simpler, they're text, and the tooling already exists (I use swig all the time to automatically make C functions and symbols available to Java and Python. Some Lisps as well). > The reason the tooling exists, is because they are simple, and they are text. Looking at the C++ module definition, it's unclear to me[1] that similar tooling for C++ modules will ever be possible. Perhaps there's something specific about C++'s module design that makes this difficult, but Rust has modules and has similar tooling to enable you to automatically generate bindings to other languages (including Python and Java).

Rusky 3y ago

Really there's nothing stopping tooling like swig (or a C++ compiler for that matter) from processing module interfaces as text, just like headers. The language spec for modules doesn't require (or even really talk about) compiling them to any kind of binary or intermediate format. Rather, the language spec aspect of modules is about symbol visibility and collisions. In terms that interop cares about, moving a declaration into a module just adjusts its name mangling, without changing much else about e.g. its presence in the object file.

lelanthran 3y ago
> Really there's nothing stopping tooling like swig (or a C++ compiler for that matter) from processing module interfaces as text, just like headers. Well the one hurdle is that C++ is a lot more complex to process than C. You'll note in my original comment upthread, I pointed out that the C++ libraries provide a C interface in their headers, and that is what tools like swig work with. They don't (and can't, because of symbol mangling) read C++-only headers. Since they can't do it for headers (because the resulting symbols differ from compiler to compiler), why would they be able to do it for modules? I'm of the understanding that C++ modules are unable to provide a C interface (`extern "C"` or similar), so it's kinda pointless reading them and generating a list of symbols, because then the generated stubs only work when the library is compiled with that specific compiler. I think that it would be nice if the compiler can be instructed to output header files for the modules it reads. You'll still have the problem that the header now has mangled symbols in it (because modules cannot have `extern "C"`), but at least tools can generate stubs for that specific instance of the compiled library. (PS. I'm happy to be corrected about the `extern "C"` thing - if I am wrong about that then, sure, tooling that scans C++ modules for `extern "C"` can in fact be made quite easily).
- Rusky 3y ago
  You can write `extern "C"` declarations in modules, actually. And either way the complexity of C++ and its name mangling is independent of headers vs modules, no? Further, swig does seem to support quite a bit of C++. IIUC it even avoids dealing with name mangling by generating wrappers that provide a C interface.
  - lelanthran 3y ago
    > You can write `extern "C"` declarations in modules, actually. See, I did not know that :-) > And either way the complexity of C++ and its name mangling is independent of headers vs modules, no? You got me self-doubting here: C++ implementations are necessarily more complex than their interfaces. The full complexity of C++ (not restricted to only name-mangling) is not usually found in headers intended for consumption by a C-interface compatible consumer. The intention of headers is to serve as an interface alone, but absolutely nothing enforces that - nothing stops the developer from dumping the entire implementation into the header (and this is a popular route for some libraries that are header-only). Modules are not intended to serve as an interface alone, so it is more likely that devs (myself included) will simply throw the entire implementation into a module, because it seems like a better idea to do so with modules. IOW, common practice for headers is to only have the interface necessary to use the implementation, but I think that common practice for modules will be to have the implementation and interface included in a single module. > Further, swig does seem to support quite a bit of C++. IIUC it even avoids dealing with name mangling by generating wrappers that provide a C interface. Swig supports much of C++ in a smart way, by default. For things like demangling swig generates `extern "C"` wrappers (for functions, definitely - not so sure about class typenames and enum typenames). But, to generate those wrappers from modules requires the original source code for those modules, and to use generated C++ wrappers, it needs to be compiled with the same compiler: Not a large hurdle, but it is definitely an additional concern compared to using the headers. It will be interesting to see how this plays out: how long it takes until swig support for modules, or swig-like tooling for modules, comes along. It's still early days for modules, after all.