11 ms·
C++ Language Interface Foundation (CLIF)
- deleted 10y ago[deleted]
- gumby 10y agoFinally! I have been suffering under SWIG and have been hoping for some time that someone would get the compiler to do this.+ Not to say SWIG isn't an amazing effort, but it's a whole C++ compiler maintained by a very small team. + I already put my time in at the gcc salt mines so no, I didn't try doing this myself.
- m-j-fox 10y agoI'm cautiously excited. The problem with C++ is there's just so many good frameworks but you can't use them without creating more C++. SWIG should be called WIG. And anyway, I can't see it surviving modern C++. As long as I'm ranting, what is modern C++ but a bunch of new languages that are incompatible with C++, each other, and everything else? Forgive me if I'm wrong but this seems like a terrible idea.
- sriram_sun 10y ago> SWIG should be called WIG. Depends on your perspective. The other alternative is (was?) to start typing up those wrapper functions manually.
- pjmlp 10y agoModern C++ means picking up the ideas from Alexandrescu, avoiding unsafe C style programming unless profiler tells otherwise and using the higher level features from C++ for writing nice, usable, safe libraries. Many of the idioms actually already possible back in the C++ARM days, before C++98 was a thing, but spoiled by C refugees.
- m-j-fox 10y agoMaybe I got my terminology wrong. I'm talking about the wave of new standards. I feel like they're a bunch of new languages which are improvements over C++ but not backwards compatible with C++. Every problem you have interoperating two different languages you have between these different C++es but worse because there aren't tools like SWIG to help you.
- pjmlp 10y agoYou can say the same about any programming language that enjoys wide market adoption, except maybe for C that still thinks computers are like PDP-11's, with C99 and C11 being very tiny evolutions with little regard to improve the overall productivity. Even Fortran and Cobol(!) have evolved more than C.
- mountain_lion 10y agoPerhaps the best thing about CLIF is that it never, ever, parses any C++ by hand [0]. It always uses a state-of-the-art, fully-industrial-strength, well-supported compiler: Clang. And it can be updated in lockstep with Clang near trivially. [0] Actually, there is one tiny exception: CLIF occasionally separates a C++ qualified name on the scope-resolution operator "::".
- cjhanks 10y agoI like this idea, I will try this soon. Is there any plan for somewhat transparently supporting NumPy?
- mish33 10y agoNumPy support is a borderline issue for CLIF. Typical usage I saw is the following: C++ has some class(es) that best represented as NumPy arrays. So because those classes are specific to the project (not generic like std::) they don't fit into CLIF runtime, but the project supply a C++ library with custom conversion functions as described in ext.md that use NumPy C API tells CLIF that those classes are convertible to Python objects (that NumPy objects are). That all NumPy integration CLIF needs and it has to come from the user.
- waynecochran 10y agoHaving trouble groking this. Can someone give a simple use case / example? If the Parser generates language agnostic data, how can this data be passed to the Matcher which parses C++ headers (i.e., C++ headers are not language agnostic). A good README should give a "high level" description of what the things is and what its used for.
- krona 10y agoThe repository mentions "other languages" but I could only find Python examples? While I think wrapper generators are a noble idea, I doubt I would choose this over the convenience of IDL-like metaprogramming approaches (e.g. pybind11, or Boost.Python)
- Varriount 10y agoDoesn't the IDL approach require that the declarations/intermediate language be written manually though? Most C++ libraries out there don't have any IDL files (looks at QT)
- krona 10y agoI mean IDL-like, as in valid c++ that is almost as easy to read as an IDL (to me at least) but without losing the ability to dive in to the binding if necessary. If you're trying to produce a binding that feels idiomatic in the target language, I'd say this is important, and this is where Swig-like approaches show their weakness. I think the author of Swig shares this sentiment: (http://code.activestate.com/lists/python-dev/109281/ http://code.activestate.com/lists/python-dev/109281/).
- mountain_lion 10y agoCLIF is designed to support arbitrary front-end languages--and other languages are in the exploratory phases. But there is no full support for other languages at this time.
- gpshead 10y agoMetaprogramming such as pybind11 does is neat. Thanks for the pointer to the project. It still looks like manually written C extension modules to me based off of what I see in https://github.com/pybind/pybind11/blob/master/docs/basics.rst https://github.com/pybind/pybind11/blob/master/docs/basics.r.... Just much shorter code with a lot of the hard to get right details taken care of you. Good! Better than the status quo. But it doesn't abstract the problem away very much. (no doubt some will consider that a feature) What if I also wanted my wrapper available on a non CPython VM? The proper way to use PyPy is not via its fake CPython API support (slow and memory hungry). Imagine if a PyPy Generator were added to CLIF. It'd use the same interface definition Parser but generate an entirely different set of code. In PyPy's case that would probably be a C library wrapping the C++ for generated cffi based Python code to interact with. Admittedly all hypothetical here until other Python VMs have CLIF Generator implementations.
- rayiner 10y agoC++ is an extremely challenging language to write a wrapper generator for. I did a C++ to Common Lisp generator about a decade ago for my GSoC project, based on GCC-XML (with the goal of being able to wrap QT). The two challenges you run into are that the GCC C+ ABI is extremely complicated and requires a runtime on top of whatever low-level C FFI you're working with. The second is that many semantics just don't map very easily. I imagine the latter problem has only gotten worse with heavier use of templated types in APIs. It looks like CLIF doesn't expose the template per se (at least in the Python examples) but requires explicit instantiation of each specialization as a distinct Python class. Templates mess with the very concept of an FFI generator, which tends to assume a reasonably clear separation between compile time and run time (so you don't need to run a C++ compiler as a pre-pass on any code that uses a binding). All that is a roundabout way of saying: for god's sake write your system APIs in C.
- mountain_lion 10y agoIt makes sense to write system APIs in C, but there are many more things than system APIs, and many of them find it useful to use C++ constructs. (edit) Clang's internals are also several orders of magnitude easier to work with than GCC's internals. In fact, clang is designed to be used as a library, and CLIF leverages that very, very highly. See, for example, https://clang.llvm.org/docs/ClangTools.html https://clang.llvm.org/docs/ClangTools.html . Clang tools are very powerful, and much of CLIF is based on a Clang tool, not a GCC internals hack.
- chubot 10y agoHow stable are Clang's internals? Are you supposed to be able to write your own Clang tools and link them dynamically, or are they mostly distributed and built together with the Clang source code? (i.e. like Linux drivers where there is no stable interface.) I know that LLVM is notable for not maintaining API stability -- not sure about Clang.
- mountain_lion 10y agoThere are (roughly) two ways of using Clang as a library. The first is a via a C-based API into a libclang.so. This is an API designed to be extremely stable, is fairly powerful, and works relatively well. That API doesn't expose absolutely everything though, which is one reason it can be so stable. Tools that need more are typically shipped and built along side the clang source code. CLIF is the latter. You can read about both of them here: http://clang.llvm.org/docs/Tooling.html http://clang.llvm.org/docs/Tooling.html
- nerdponx 10y agoHow much C++ do I need to know in order to wrap some else's C++ library?
- mhh__ 10y agoQuite a lot, or at least by proxy given that you would have to talk to library and compilers that are written in C++. However: Some libraries/packages do expose a C api, which you could use in a language assuming it had a decent FFI. A lot of major C++ projects use exceptions, which you will have to catch and handle inside of the C++ because even if C could catch exceptions, there is no guarantee of ABI compatibility. Some languages, I think D can do it, can catch an exception thrown in C++ code. You might have better luck integrating a standard data interface between the C++ and any other language: However, that could be a little slow.
- jzwinck 10y agoHaving used both SWIG and Boost.Python professionally, I find manual work required by the latter to be 100% worthwhile and usually necessary. C++ and Python are not very similar, and the result of auto-binding them typically gives up a combination of fluency and performance. A simple example of the former is a C++ algorithm which writes to an output iterator. In Python this might be expressed as a generator. But none of the auto-binding tools can do this transformation. As for the latter--performance--I have seen 100x performance penalties in practice when the blind lead the bind. A good example is that in Python we have memoryview and the buffer protocol, but auto-binding tools take these no further than std::vector (if that). A C++ API which produces a large stream of numeric data just begs to be bound using the buffer protocol or perhaps even NumPy directly. But if a C++ API produces small values really quickly, an auto-bound one will produce the same small values really slowly. Some people see the writing of bindings as manual labor to be avoided. I see it as an optimization opportunity. There are huge gains available.
- gpshead 10y agoThere are always justifiable use cases for manually written Python bindings. But a "there be dragons" caveat applies as getting the CPython API correct is complicated. Reference counting and error checking bugs are common in hand written CPython C API code. For most cross language bindings the primary goal is to "just work reliably". Optimization can happen later after you have profiles to figure out where it is worth doing and in what way.
- stinos 10y ago> I find manual work required by the latter to be 100% worthwhile and usually necessary Roughly the same experience. Automatic generation for anything more than trivial examples always seems to lead to the need for more and more configuration to keep everything in line, to the point where just writing everything by hand becomes less work. This is likely due to the type of software we use it for, but still, the amount of time I lost on SWIG really seems wasted instead of leaving the feeling of having learnt at least something. It was all not very pleasant. The Python side of our software gets exposed to less tech-savvy users and is meant to be a more friendly layer over C++ functions and classes. But the C++ side isn't really an API in the sense there is no hard distinction between the 'internal' layer and the API layer. Some class which is only used internally might the next day also be wrapped to be available in Python. So there's no single directory or so one can point to and say 'everything in there is the API'. Also given a C++ class, sometimes only a couple of methods have to get exposed to the users, sometimes with a different name even, or some arguments defaulted etc. Preferrably without having to write a wrapper just for the sake of exposing it. All this turned out to be a nightmare in SWIG. Just manually writing one or more lines for registering the function manually (not Boost.Python but similar) is usually a one-time-almost-never-look-back thing which, for us, is way less work in the end. Even with the hundreds of functions we have already. And of course performance is the other advantage.
- santaclaus 10y agoCan this wrap Qt (or barring Qt, wxWidgets)? There are great Qt bindings in Python, but in other languages, not so much.
- gpshead 10y agoI wondered how long it would take before someone posted CLIF here since we haven't put up a blog post about it yet... :) Interesting stat: Around 80% of our the Python C++ extension module wrappings being added to our code base are now being done using CLIF instead of SWIG. We are actively working on forbidding new Python SWIG wrappers from being added and migrating important legacy wrappings off of it onto CLIF. Who am I? I am TL of the team that create CLIF (design and code reviewer, _not_ a primary author; they can identify themselves as they see fit). -gps@
- obstinate 10y agoWhen are y'all doing Go and Java, though? The fact that this exists just compounds the pain of writing SWIG for the other two languages. The grass really is greener.
- inetknght 10y agoSo... someone with basically zero Python experience and a ton of C++ experience could basically "hand that experience down" to his newly minted Python-self?
- gumby 10y agoSounds like a worthy goal. There will always be infelicities (as with human language translation as well) but why not?
- malkia 10y agoThere are several issues that I usually hit with bindings to C/C++ and other languages: - Calling a native function that itself takes callback, and that callback might be your non-native code. (trampolines?) - Dealing with memory allocation. Who owns what. - Exceptions or long_jmps - Green threads, fibers, etc. - Other?
- solipsism 10y agoCallbacks are well supported in SWIG for most languages.
- malkia 10y agoAlso: - int64 -> uint64 conversion - int64/uint64 -> double conversion
- dekhn 10y agocallbacks in swig are straightforward. you have a C/C++ reciver function that gets registered along with a pointer to the scripting language target function. I've used this design for over a decade.
- malkia 10y agoI was mentioning FFI in general, like Common Lisp's FFI, LuaJIT, others. For example for callback back, you need to have a trampoline, where the "C/C++" trampoline would call back your language runtime "engine". But this introduces gap in the "stack", and might not work with all VM's, or you may run out of trampoline slots, and not being able to allocate new, since this would require dynamic code generation, and say on iOS and certain other game consoles that's not allowed.
- dekhn 10y agothere are no trampoline slots required. it's a function pointer that's provided as an argument (user data). Can you point to an FFI that doesn't support callbacks in this way? This would be a major problen in the FFI implementation.
- ddobrev 10y agoFellows, I would suggest you check https://github.com/mono/CppSharp https://github.com/mono/CppSharp. It's Clang-based as well and despite the name, it's not bound to C# or .NET, generators for any languages can be added. It's feature complete with the exception of templates which are being worked on as we speak. It's also fully automated, manual intervention is only required if the user wants binding-specific customisation.
- ddobrev 10y agoFellows, I would suggest you check https://github.com/mono/CppSharp https://github.com/mono/CppSharp. It's Clang-based as well and despite the name, it's not bound to C# or .NET, generators for any languages can be added. It's feature complete with the exception of templates which are being worked on as we speak. It's also fully automated, manual intervention is only required if the user wants binding-specific customisation.