4 ms·

I built bindings for various C++ libraries in SWIG (not only for Python but also Lua and Ruby) and I was always very impressed with the result. What's amazing a

I built bindings for various C++ libraries in SWIG (not only for Python but also Lua and Ruby) and I was always very impressed with the result. What's amazing about SWIG is its ability to easily port very complex and rich code. I successfully exported very complex class hierarchies and code relying heavily on pointers and STL to Python and/or Lua. The process consists of writing several header-like configuration files that SWIG ingests and uses to generate bindings, which is often pretty straightforward. Often you can simply import the normal C/C++ header files in those configs and add some glue code / extra hints to help SWIG in cases where it can't figure out what to do with a particular type. In my experience, the most difficult aspect of binding generation is not writing the glue code, but doing so in a scalable way. Manually writing bindings for any real-world C++ codebase would therefore be extremely tedious (IMHO), so having an automated system that does this work for you is a huge time saver. Some larger libraries / frameworks have their own wrapping generators Btw, PyQt (which provides extremely good bindings to QT) for example has SIP which is worth looking at (it's open source).

piokuc 9y ago
What's also important is to have a simple, neat API, ideally C API. If you do have it, then writing wrappers for any language, even manually, without SWIG (which is great, but does add complexity to your project's build process), is simple.
- ThePhysicist 9y ago
  Yes that's a valid point, manually writing bindings for C++ is very hard though (IMHO), as there are many intricacies that you have to keep in mind, and SWIG is really good at automating all of these away. For C-code the story is a bit different because usually the code is much less complex.
brational 9y ago
Have you used pybind? I've used SWIG extensively so I'm familiar with that - you talked about SWIG but not how it compares to this new one. Reading through some docs, it looks like pybind suits better for a python codebase that occasionally needs C++ pieces added in as you have to setup the C++ code to handle the py needs. SWIG I was always impressed with exactly your point. You could sometimes just make 1 big header file in a facade pattern to expose some "start" buttons needed to run a huge c++ application and nothing else needed to be setup. No going back and forth. Example usage here was a single heavy duty image processing application but using python has the distributor and work manager. C++ code was a "worker" and python and zeroMQ did everything else. So the only "connection" we needed was a start button and some metadata to the c++. Maybe 1 header file with 100-200 lines of code to make the bridge for SWIG. (to anyone else reading) does that sound trivially simple to the do the same with pybind?
- ThePhysicist 9y ago
  I haven't used pybind yet, so I can't compare the two unfortunately. From how I understand the docs it seems you would have to write the interface definitions in C++ and the compiler would generate the bindings for you, which sounds like a nice approach that could be a bit more cumbersome though as (again, in my understanding) forces you to write and adapt all bindings by hand.
  - brational 9y ago
    >could be a bit more cumbersome though as (again, in my understanding) forces you to write and adapt all bindings by hand. That was my interpretation as well.
- sqeaky 9y ago
  SWIG is amazing. I have usually taken a different approach so that I could get the scaling you talk about. I point SWIG at the real headers in my projects and use ifdefs to block out things don't want SWIG to see. There are also a few rules that should be followed when doing this. Things like: don't let SWIG see overloads if the other language doesn't support them; don't return raw pointers that need to be deleted; don't let swig see std headers, etc... Then I make SWIG (optionally) run as part of the build. Any warning it makes need to be resolved somehow because CI will reject commits causing warnings. It cost just a little more dev time, but prevents eons of debugging time.
BeetleB 9y ago
Swig is great when it works, a nightmare when it doesn't. A coworker in a former job used Swig to enable a C library to be used in Python. Initially, all was great. But then we discovered a memory leak. He tried for weeks to identify it, but never could. Cython would be my first choice.
- sqeaky 9y ago
  I have used SWIG a great deal. Any time I have had leaks near it they were in my code or not really leaks and just quirks of the garbage collector in the Non-C language. I have used swig with Ruby, Lua and Java and it the code it makes doesn't leak unless the C/C++ code is bad, its just too simple.
- ThePhysicist 9y ago
  Ownership can be a tricky question, but usually SWIG is pretty explicit about which memory/objects it owns and which it doesn't. Like the other poster I also often thought that my program was leaking memory as the Python interpreter does not immediately release unused memory, so even if everything is fine it can seem like your program is consuming more and more memory (explicitly collecting garbage using `gc.collect()` usually helps here). In general, you should be really clear about who owns a given object (either the C/C++-side of your code or the Python side) and avoid passing ownership back and forth between the two. I already mapped some pretty intricate code using SWIG (including a parser generator that created lots of object instances in C++ and passed them to Lua for manipulation, and where the Lua side could also create objects and link them to the C++-created objects) and so far didn't have any problems with memory leaks due to the binding code.

piokuc 9y ago

What's also important is to have a simple, neat API, ideally C API. If you do have it, then writing wrappers for any language, even manually, without SWIG (which is great, but does add complexity to your project's build process), is simple.

ThePhysicist 9y ago
Yes that's a valid point, manually writing bindings for C++ is very hard though (IMHO), as there are many intricacies that you have to keep in mind, and SWIG is really good at automating all of these away. For C-code the story is a bit different because usually the code is much less complex.

brational 9y ago

Have you used pybind? I've used SWIG extensively so I'm familiar with that - you talked about SWIG but not how it compares to this new one. Reading through some docs, it looks like pybind suits better for a python codebase that occasionally needs C++ pieces added in as you have to setup the C++ code to handle the py needs. SWIG I was always impressed with exactly your point. You could sometimes just make 1 big header file in a facade pattern to expose some "start" buttons needed to run a huge c++ application and nothing else needed to be setup. No going back and forth. Example usage here was a single heavy duty image processing application but using python has the distributor and work manager. C++ code was a "worker" and python and zeroMQ did everything else. So the only "connection" we needed was a start button and some metadata to the c++. Maybe 1 header file with 100-200 lines of code to make the bridge for SWIG. (to anyone else reading) does that sound trivially simple to the do the same with pybind?

ThePhysicist 9y ago
I haven't used pybind yet, so I can't compare the two unfortunately. From how I understand the docs it seems you would have to write the interface definitions in C++ and the compiler would generate the bindings for you, which sounds like a nice approach that could be a bit more cumbersome though as (again, in my understanding) forces you to write and adapt all bindings by hand.
- brational 9y ago
  >could be a bit more cumbersome though as (again, in my understanding) forces you to write and adapt all bindings by hand. That was my interpretation as well.
sqeaky 9y ago
SWIG is amazing. I have usually taken a different approach so that I could get the scaling you talk about. I point SWIG at the real headers in my projects and use ifdefs to block out things don't want SWIG to see. There are also a few rules that should be followed when doing this. Things like: don't let SWIG see overloads if the other language doesn't support them; don't return raw pointers that need to be deleted; don't let swig see std headers, etc... Then I make SWIG (optionally) run as part of the build. Any warning it makes need to be resolved somehow because CI will reject commits causing warnings. It cost just a little more dev time, but prevents eons of debugging time.

BeetleB 9y ago

Swig is great when it works, a nightmare when it doesn't. A coworker in a former job used Swig to enable a C library to be used in Python. Initially, all was great. But then we discovered a memory leak. He tried for weeks to identify it, but never could. Cython would be my first choice.

sqeaky 9y ago
I have used SWIG a great deal. Any time I have had leaks near it they were in my code or not really leaks and just quirks of the garbage collector in the Non-C language. I have used swig with Ruby, Lua and Java and it the code it makes doesn't leak unless the C/C++ code is bad, its just too simple.
ThePhysicist 9y ago
Ownership can be a tricky question, but usually SWIG is pretty explicit about which memory/objects it owns and which it doesn't. Like the other poster I also often thought that my program was leaking memory as the Python interpreter does not immediately release unused memory, so even if everything is fine it can seem like your program is consuming more and more memory (explicitly collecting garbage using `gc.collect()` usually helps here). In general, you should be really clear about who owns a given object (either the C/C++-side of your code or the Python side) and avoid passing ownership back and forth between the two. I already mapped some pretty intricate code using SWIG (including a parser generator that created lots of object instances in C++ and passed them to Lua for manipulation, and where the Lua side could also create objects and link them to the C++-created objects) and so far didn't have any problems with memory leaks due to the binding code.