4 ms·

Having used both SWIG and Boost.Python professionally, I find manual work required by the latter to be 100% worthwhile and usually necessary. C++ and Python ar

Having used both SWIG and Boost.Python professionally, I find manual work required by the latter to be 100% worthwhile and usually necessary. C++ and Python are not very similar, and the result of auto-binding them typically gives up a combination of fluency and performance. A simple example of the former is a C++ algorithm which writes to an output iterator. In Python this might be expressed as a generator. But none of the auto-binding tools can do this transformation. As for the latter--performance--I have seen 100x performance penalties in practice when the blind lead the bind. A good example is that in Python we have memoryview and the buffer protocol, but auto-binding tools take these no further than std::vector (if that). A C++ API which produces a large stream of numeric data just begs to be bound using the buffer protocol or perhaps even NumPy directly. But if a C++ API produces small values really quickly, an auto-bound one will produce the same small values really slowly. Some people see the writing of bindings as manual labor to be avoided. I see it as an optimization opportunity. There are huge gains available.

gpshead 9y ago
There are always justifiable use cases for manually written Python bindings. But a "there be dragons" caveat applies as getting the CPython API correct is complicated. Reference counting and error checking bugs are common in hand written CPython C API code. For most cross language bindings the primary goal is to "just work reliably". Optimization can happen later after you have profiles to figure out where it is worth doing and in what way.
stinos 9y ago
> I find manual work required by the latter to be 100% worthwhile and usually necessary Roughly the same experience. Automatic generation for anything more than trivial examples always seems to lead to the need for more and more configuration to keep everything in line, to the point where just writing everything by hand becomes less work. This is likely due to the type of software we use it for, but still, the amount of time I lost on SWIG really seems wasted instead of leaving the feeling of having learnt at least something. It was all not very pleasant. The Python side of our software gets exposed to less tech-savvy users and is meant to be a more friendly layer over C++ functions and classes. But the C++ side isn't really an API in the sense there is no hard distinction between the 'internal' layer and the API layer. Some class which is only used internally might the next day also be wrapped to be available in Python. So there's no single directory or so one can point to and say 'everything in there is the API'. Also given a C++ class, sometimes only a couple of methods have to get exposed to the users, sometimes with a different name even, or some arguments defaulted etc. Preferrably without having to write a wrapper just for the sake of exposing it. All this turned out to be a nightmare in SWIG. Just manually writing one or more lines for registering the function manually (not Boost.Python but similar) is usually a one-time-almost-never-look-back thing which, for us, is way less work in the end. Even with the hundreds of functions we have already. And of course performance is the other advantage.
- mountain_lion 9y ago
  CLIF is used inside Google for some very non-trivial projects.
  - stinos 9y ago
    Well I'm always open to new ways of doing things. Any estimate of how hard it would be to generate MicroPython [1] wrappers for C++ code (in a rather configurable way as described earlier)? [1] https://github.com/micropython/micropython https://github.com/micropython/micropython
    - gpshead 9y ago
      It seems entirely reasonable for someone to create a MicroPython wrapper generator. It appears to have a C/C++ API so it should be similar to the existing CPython Generator and Runtime.
zeptomu 9y ago
Assuming one wants to call C++ code from Python (the common case) I would even start with simple pipe-based IPC and just build an executable in C++ that acts as a bridge between Python (or any other language in this case) and C++. E.g. using JSON serialization it would look like this: $LANG (e.g. Python) <-> JSON <-> Pipe <-> JSON <-> C++ If this is too slow I still wouldn't use a binding generator (SWIG, etc.), but build a C API using opaque pointers that offer the required functions, i.e. a bridge between C and C++. Finally one can simply call into this API with Python's ctypes which are part of its standard library. Python <-> ctypes <-> C-API <-> C++ I have worked previously with SWIG, but my experience has been very bad, in particular the problem with binding generators is similar to the problem parser generator face - they make the simple stuff simple and the hard stuff hard. Furthermore the bindings that are auto-generated are mostly too low-level, so one normally writers wrappers around them to make them more "pythonic" which raises the questions why not write the bindings from scratch anyway?

gpshead 9y ago

There are always justifiable use cases for manually written Python bindings. But a "there be dragons" caveat applies as getting the CPython API correct is complicated. Reference counting and error checking bugs are common in hand written CPython C API code. For most cross language bindings the primary goal is to "just work reliably". Optimization can happen later after you have profiles to figure out where it is worth doing and in what way.

stinos 9y ago

> I find manual work required by the latter to be 100% worthwhile and usually necessary Roughly the same experience. Automatic generation for anything more than trivial examples always seems to lead to the need for more and more configuration to keep everything in line, to the point where just writing everything by hand becomes less work. This is likely due to the type of software we use it for, but still, the amount of time I lost on SWIG really seems wasted instead of leaving the feeling of having learnt at least something. It was all not very pleasant. The Python side of our software gets exposed to less tech-savvy users and is meant to be a more friendly layer over C++ functions and classes. But the C++ side isn't really an API in the sense there is no hard distinction between the 'internal' layer and the API layer. Some class which is only used internally might the next day also be wrapped to be available in Python. So there's no single directory or so one can point to and say 'everything in there is the API'. Also given a C++ class, sometimes only a couple of methods have to get exposed to the users, sometimes with a different name even, or some arguments defaulted etc. Preferrably without having to write a wrapper just for the sake of exposing it. All this turned out to be a nightmare in SWIG. Just manually writing one or more lines for registering the function manually (not Boost.Python but similar) is usually a one-time-almost-never-look-back thing which, for us, is way less work in the end. Even with the hundreds of functions we have already. And of course performance is the other advantage.

mountain_lion 9y ago
CLIF is used inside Google for some very non-trivial projects.
- stinos 9y ago
  Well I'm always open to new ways of doing things. Any estimate of how hard it would be to generate MicroPython [1] wrappers for C++ code (in a rather configurable way as described earlier)? [1] https://github.com/micropython/micropython https://github.com/micropython/micropython
  - gpshead 9y ago
    It seems entirely reasonable for someone to create a MicroPython wrapper generator. It appears to have a C/C++ API so it should be similar to the existing CPython Generator and Runtime.

zeptomu 9y ago

Assuming one wants to call C++ code from Python (the common case) I would even start with simple pipe-based IPC and just build an executable in C++ that acts as a bridge between Python (or any other language in this case) and C++. E.g. using JSON serialization it would look like this: $LANG (e.g. Python) <-> JSON <-> Pipe <-> JSON <-> C++ If this is too slow I still wouldn't use a binding generator (SWIG, etc.), but build a C API using opaque pointers that offer the required functions, i.e. a bridge between C and C++. Finally one can simply call into this API with Python's ctypes which are part of its standard library. Python <-> ctypes <-> C-API <-> C++ I have worked previously with SWIG, but my experience has been very bad, in particular the problem with binding generators is similar to the problem parser generator face - they make the simple stuff simple and the hard stuff hard. Furthermore the bindings that are auto-generated are mostly too low-level, so one normally writers wrappers around them to make them more "pythonic" which raises the questions why not write the bindings from scratch anyway?