5 ms·

> What is great about header files? One great thing - it's nice to have an interface[1] which already has tooling to let other languages use your private, hidd

> What is great about header files? One great thing - it's nice to have an interface[1] which already has tooling to let other languages use your private, hidden and compiled implementation. If those C++ libraries that almost all of Python scientific work is built on had used modules[2] (or something similar) instead of headers, interfacing Python to those libraries would have been an immense effort. Even with modules existing now, if you're writing something core and foundational in C++, it's not getting adopted like Numpy, BLAS, etc unless you also provide header files. And that's just Python. Think of all the other languages that rely on performance critical code written in C++, and used with a C-only interface. [1] Of course, your interface can't be a C++ interface either, so there's that. [2] If modules were around at the time.

mardifoufs 3y ago
That's interesting! Why would modules make it harder to interface with c++ from python ? Is it because the header files are a direct interface to the functions? How hard would it be to do the same from modules, would it just require tooling to support modules rather than directly provided header files? (I just realized that I don't completely understand how exactly the new modules work, so I have some further reading to do! )
- lelanthran 3y ago
  > Why would modules make it harder to interface with c++ from python ? Is it because the header files are a direct interface to the functions? Well, they're simpler, they're text, and the tooling already exists (I use swig all the time to automatically make C functions and symbols available to Java and Python. Some Lisps as well). The reason the tooling exists, is because they are simple, and they are text. Looking at the C++ module definition, it's unclear to me[1] that similar tooling for C++ modules will ever be possible. As long as the most popular languages (Python, Java, etc) rely on a C header file for inter-call interoperability, header files aren't going away. No one wants to manually write the interface into a C++ library by hand. [1] I'm no C++ expert though - the problems I see with modules includes things like lack of a standard to grab the interface from the compiled module, complexity of C++ to determine the interface from the actual source code, etc.
  - nicoburns 3y ago
    > Well, they're simpler, they're text, and the tooling already exists (I use swig all the time to automatically make C functions and symbols available to Java and Python. Some Lisps as well). > The reason the tooling exists, is because they are simple, and they are text. Looking at the C++ module definition, it's unclear to me[1] that similar tooling for C++ modules will ever be possible. Perhaps there's something specific about C++'s module design that makes this difficult, but Rust has modules and has similar tooling to enable you to automatically generate bindings to other languages (including Python and Java).
  - Rusky 3y ago
    Really there's nothing stopping tooling like swig (or a C++ compiler for that matter) from processing module interfaces as text, just like headers. The language spec for modules doesn't require (or even really talk about) compiling them to any kind of binary or intermediate format. Rather, the language spec aspect of modules is about symbol visibility and collisions. In terms that interop cares about, moving a declaration into a module just adjusts its name mangling, without changing much else about e.g. its presence in the object file.
    - lelanthran 3y ago
      > Really there's nothing stopping tooling like swig (or a C++ compiler for that matter) from processing module interfaces as text, just like headers. Well the one hurdle is that C++ is a lot more complex to process than C. You'll note in my original comment upthread, I pointed out that the C++ libraries provide a C interface in their headers, and that is what tools like swig work with. They don't (and can't, because of symbol mangling) read C++-only headers. Since they can't do it for headers (because the resulting symbols differ from compiler to compiler), why would they be able to do it for modules? I'm of the understanding that C++ modules are unable to provide a C interface (`extern "C"` or similar), so it's kinda pointless reading them and generating a list of symbols, because then the generated stubs only work when the library is compiled with that specific compiler. I think that it would be nice if the compiler can be instructed to output header files for the modules it reads. You'll still have the problem that the header now has mangled symbols in it (because modules cannot have `extern "C"`), but at least tools can generate stubs for that specific instance of the compiled library. (PS. I'm happy to be corrected about the `extern "C"` thing - if I am wrong about that then, sure, tooling that scans C++ modules for `extern "C"` can in fact be made quite easily).
dagmx 3y ago
Headers are a really bad interface IMHO. Yes there’s tooling, but they require your headers be very very specific. Any deviance can break that tooling, so you’re either lucky in what you can bind or are writing to a very strict spec. Modules on the other hand are more strict and would likely have made the job of writing language bindings easier and more predictable. I’m not sure I agree with your logic about NumPy etc. You seem to be conflating technology limitations with policy ones. They could most certainly use modules too , since they exist in the compiled space anyway. They won’t because they make certain compiler compatibility guarantees but that’s a conflict of interests not of technical quality. In general your argument feels a bit “older is better due to legacy support”. That would preclude almost any advancement in the space as well, because any new language feature (even just a keyword) would break all your examples above equally too.
- lelanthran 3y ago
  > Modules on the other hand are more strict and would likely have made the job of writing language bindings easier and more predictable. There's two ways for tooling like swig to work with modules: 1. Read the compiled module and generate the stubs to call into it. 2. Read the source code for the module and generate the stubs for it. Which of those, do you think, is easier than reading C (not C++) header files?
  - dagmx 3y ago
    Reading the module information would be significantly easier. (which is not something swig can do today since swig hasn’t been updated in a while ) Parsing code to define an API is horribly fragile. A single macro or unknown keyword can break it. Hence why you had to specify C not C++ headers. Having a statically defined metadata like C++ modules do would have greatly simplified binding generation

mardifoufs 3y ago

That's interesting! Why would modules make it harder to interface with c++ from python ? Is it because the header files are a direct interface to the functions? How hard would it be to do the same from modules, would it just require tooling to support modules rather than directly provided header files? (I just realized that I don't completely understand how exactly the new modules work, so I have some further reading to do! )

lelanthran 3y ago
> Why would modules make it harder to interface with c++ from python ? Is it because the header files are a direct interface to the functions? Well, they're simpler, they're text, and the tooling already exists (I use swig all the time to automatically make C functions and symbols available to Java and Python. Some Lisps as well). The reason the tooling exists, is because they are simple, and they are text. Looking at the C++ module definition, it's unclear to me[1] that similar tooling for C++ modules will ever be possible. As long as the most popular languages (Python, Java, etc) rely on a C header file for inter-call interoperability, header files aren't going away. No one wants to manually write the interface into a C++ library by hand. [1] I'm no C++ expert though - the problems I see with modules includes things like lack of a standard to grab the interface from the compiled module, complexity of C++ to determine the interface from the actual source code, etc.
- nicoburns 3y ago
  > Well, they're simpler, they're text, and the tooling already exists (I use swig all the time to automatically make C functions and symbols available to Java and Python. Some Lisps as well). > The reason the tooling exists, is because they are simple, and they are text. Looking at the C++ module definition, it's unclear to me[1] that similar tooling for C++ modules will ever be possible. Perhaps there's something specific about C++'s module design that makes this difficult, but Rust has modules and has similar tooling to enable you to automatically generate bindings to other languages (including Python and Java).
- Rusky 3y ago
  Really there's nothing stopping tooling like swig (or a C++ compiler for that matter) from processing module interfaces as text, just like headers. The language spec for modules doesn't require (or even really talk about) compiling them to any kind of binary or intermediate format. Rather, the language spec aspect of modules is about symbol visibility and collisions. In terms that interop cares about, moving a declaration into a module just adjusts its name mangling, without changing much else about e.g. its presence in the object file.
  - lelanthran 3y ago
    > Really there's nothing stopping tooling like swig (or a C++ compiler for that matter) from processing module interfaces as text, just like headers. Well the one hurdle is that C++ is a lot more complex to process than C. You'll note in my original comment upthread, I pointed out that the C++ libraries provide a C interface in their headers, and that is what tools like swig work with. They don't (and can't, because of symbol mangling) read C++-only headers. Since they can't do it for headers (because the resulting symbols differ from compiler to compiler), why would they be able to do it for modules? I'm of the understanding that C++ modules are unable to provide a C interface (`extern "C"` or similar), so it's kinda pointless reading them and generating a list of symbols, because then the generated stubs only work when the library is compiled with that specific compiler. I think that it would be nice if the compiler can be instructed to output header files for the modules it reads. You'll still have the problem that the header now has mangled symbols in it (because modules cannot have `extern "C"`), but at least tools can generate stubs for that specific instance of the compiled library. (PS. I'm happy to be corrected about the `extern "C"` thing - if I am wrong about that then, sure, tooling that scans C++ modules for `extern "C"` can in fact be made quite easily).

dagmx 3y ago

Headers are a really bad interface IMHO. Yes there’s tooling, but they require your headers be very very specific. Any deviance can break that tooling, so you’re either lucky in what you can bind or are writing to a very strict spec. Modules on the other hand are more strict and would likely have made the job of writing language bindings easier and more predictable. I’m not sure I agree with your logic about NumPy etc. You seem to be conflating technology limitations with policy ones. They could most certainly use modules too , since they exist in the compiled space anyway. They won’t because they make certain compiler compatibility guarantees but that’s a conflict of interests not of technical quality. In general your argument feels a bit “older is better due to legacy support”. That would preclude almost any advancement in the space as well, because any new language feature (even just a keyword) would break all your examples above equally too.

lelanthran 3y ago
> Modules on the other hand are more strict and would likely have made the job of writing language bindings easier and more predictable. There's two ways for tooling like swig to work with modules: 1. Read the compiled module and generate the stubs to call into it. 2. Read the source code for the module and generate the stubs for it. Which of those, do you think, is easier than reading C (not C++) header files?
- dagmx 3y ago
  Reading the module information would be significantly easier. (which is not something swig can do today since swig hasn’t been updated in a while ) Parsing code to define an API is horribly fragile. A single macro or unknown keyword can break it. Hence why you had to specify C not C++ headers. Having a statically defined metadata like C++ modules do would have greatly simplified binding generation