28 ms·
Pybind11 — Seamless operability between C++11 and Python
- fulafel 9y agoI like how the first basic example invokes undefined behaviour without mention: int add(int i, int j) { return i + j; }
- deleted 9y ago[deleted]
- devrandomguy 9y agoLooks fine to me. Are you supposed to check for integer overflow before doing the addition, or something?
- nikbackm 9y agoProbably. Signed integer overflow is undefined behaviour in C and C++.
- first_amendment 9y agoSo is sqrt(-1) but that doesn't mean sqrt() shouldn't exist. Just don't call add() with arguments that will overflow.
- devrandomguy 9y agoSo, sanitize your `/add?a=1&b=2` endpoint inputs by making sure that they don't add up to something greater than... wait a sec. Actually, let's just return an unsigned int. No wait! We might get a negative number as input. Hmm, can I bitbang my way out of this? Can I be sure that the signed ints are two's complement? Man, these languages are seriously !!fun!!
- first_amendment 9y agoThere will always be syntactically valid yet semantically invalid statements in any grammar. Undefined behavior isn't bad, it's a unavoidable consequence of all languages, programming or otherwise. Grammatical constructions that have no meaning will always exist. If you want to make sure your callers are obeying the interface contract, use assert() to not incur runtime overhead in production and guard against programming error: int add(int a, int b) { assert(!__builtin_sadd_overflow(a, b, NULL)); return a + b; }
- icebraining 9y agoThere will always be syntactically valid yet semantically invalid statements in any grammar. Really? Why? And can you give an example of that in, say, Python?
- first_amendment 9y ago(-1) ** 0.5 Why define something that has no meaning? Just a waste of cycles, CPU and brain.
- pavanky 9y agoI am assuming this is python. This fails in python2 but works (returns complex number) in python3.
- geofft 9y agoThis is well-defined: >>> (-1) ** 0.5 Traceback (most recent call last): File "<stdin>", line 1, in <module> ValueError: negative number cannot be raised to a fractional power Any conforming implementation of Python 2.7 must raise ValueError. Why define it? So that a programmer can catch ValueError.
- first_amendment 9y agoAs I said in another thread, trusting a buggy program to debug itself is futile. Effectively handling unintentional ValueError (or IndexError, TypeError) is not practically possible. For one, there's a good chance the program's persistent shared state is in an inconsistent state when those unintentional exceptions happen. Better to just abort().
- deleted 9y ago[deleted]
- junke 9y agoCL-USER> (sqrt -1) #C(0.0 1.0)
- pjmlp 9y agoYep, add(INT64_MAX, 1) Is undefined. If using Ada, similar kind of code would produce "raised CONSTRAINT_ERROR : program.adb:7 overflow check failed" To be fair, Ada's behavior is easy to reproduce with a checked integer class like https://accu.org/index.php/journals/324 https://accu.org/index.php/journals/324
- coldtea 9y agoSo? There millions of lines of code, from the best C/C++ programmers, even the kernel, that add two ints in all kinds of programs. Why is this suddenly a valid concern, especially for a code example, not NASA's missile code or Tesla's self-driving libs?
- pjmlp 9y agoIt is not a sudden concern, it has been ignored since C and C++ exist, and only became worse with code exposed to the world via the Internet. "Many years later we asked our customers whether they wished us to provide an option to switch off these checks in the interests of efficiency on production runs. Unanimously, they urged us not to--they already knew how frequently subscript errors occur on production runs where failure to detect them could be disastrous. I note with fear and horror that even in 1980, language designers and users have not learned this lesson. In any respectable branch of engineering, failure to observe such elementary precautions would have long been against the law." -- Tony Hoare, "The 1980 ACM Turing Award Lecture" Millions of people also drive without security belt or helmet, apparently they are an useless extra.
- the_grouch 9y agoOne can do an overflow analysis of a whole program. This however takes time and would prevent frequent Hacker News comments. Do people who write such programs comment here? I don't think so.
- zurn 9y agoGoogle for "site:cvedetails.com integer overflow linux" Yeah, there are places in code like that, and many of those lines have been security vulnerabilities. For anyone who is surprised by this and is handling untrusted data in C/C++, it's a good idea to read up on the subject.
- coldtea 9y agoI like it how contrived complaints can get...
- aoloe 9y agowhat about opening a ticket / doing a pull request with a better trivial example that does not "teach" bad habits?
- geofft 9y agoA simple fix is using unsigned integers, as unsigned overflow is well-defined. (There isn't a way in C/C++ to request an exception or fault on integer overflow, which is what you'd want here, since you're running inside Python. A better conceptual fix is just doing the addition in Python, which is a more robust language than C/C++ for signed integer addition, but that kind of defeats the point of the example.)
- first_amendment 9y ago> There isn't a way in C/C++ to request an exception or fault on integer overflow This is incorrect. See compiler options: -ftrapv, -fsanitize=undefined. See compiler builtins: __builtin_add_overflow and friends.
- fulafel 9y agoBut no way in standard C/C++.
- deleted 9y ago[deleted]
- first_amendment 9y agoGCC/Clang/VC++ represent 99% of the compiler market. In the exceedingly rare chance your vendor doesn't implement those extensions, you're probably a big enough customer of theirs that you can request it.
- geofft 9y agoI mean, yes, I'm sympathetic to the argument that most people should use a variant of C that isn't actually C and is much better about undefined behavior (not just integer overflow but other things too). https://blog.regehr.org/archives/1180 https://blog.regehr.org/archives/1180 has some thoughts along these lines. I'm not sure that pybind11 wants to have its docs using a nonstandard variant of C, though, for lots of reasons, including that I expect the major use case of pybind11 is binding to existing C or C++ projects which are generally written in standard C or C++. (But not always! Postgres, for instance, is written in a variant of C with -fwrapv - not even -ftrapv, but a definition of signed overflow - instead of standard C.)
- louiz 9y agoJust like “operator+(int, int)” invokes undefined behaviour without mention: you can write “a + b” and it will break if the addition overflows! Or std::vector<T>::operator[] invokes undefined behaviour if the argument is invalid (out of band). Or printf() invokes undefined behaviour if an argument is of the wrong type (for example "%s" with a corresponding int). Or strcpy invokes undefined behaviour if you give it invalid pointers (or non-null terminated strings, etc). So, yeah, many functions can invoke undefined behaviour when they are called with bad arguments, that doesn’t make them bad functions.
- fulafel 9y agoThe difference is that this was a function exposed to Python. The expectation of Python programmers is that calling into the module won't explode. The Python vs C++ joke here is that the minimal a + b example turns out to be unsafe to call with random numbers.
- kazinator 9y agoA certain loss of safety is common when you're using FFI. Take any language's FFI and bind to the C library atoi function. Poof, you have instant undefined behavior for bad string to int conversions. If you want a safe, robust module that is based on FFI, you have to write some padding in the higher level language that avoids misusing the foreign API in any way. Directly exposed FFI stuff is not safe and cannot reasonably be safe; it makes no sense to expect that.
- fulafel 9y agoI disagree that this is a FFI. There's are FFIs for Python, the builtin one is called ctypes, and users know it's an exception to Python's safety. Modules implemented in C++ and presenting functionality to Python programs in a native way are expected to be safe.
- kazinator 9y agoUnless I'm mistaken, the topic is Pybind11, which is a foreign binding mechanism. Being criticized is the example in its documentation: http://pybind11.readthedocs.io/en/master/basics.html http://pybind11.readthedocs.io/en/master/basics.html under the heading "Creating bindings for a simple function". Looks like FFI to me.
- e12e 9y agoI for one am grateful for this comment, I program rarely enough in C/c++ that I wouldn't have assumed signed overflow was undefined. As the example below with "int a; a+5>a" illustrates, that's potentially pretty serious. I wonder if the "best" fix here, would be to use assembler intrinsics to maintain performance, possibly with a fallback to a type cast/promotion to a bigger int for the sum, and some form of defined behaviour for overflow on the python side? (reinventing the python number tower with full automatic promotion seems counter-productive here, as presumably the idea is to work with fast/simple/native (signed) integer math...) Or just redefine the function to use unsigned ints.
- ivan_ah 9y agoother projects in that space are: cython (need to manually do wrapping), xdress, SWIG, SIP, clif. Does anyone have experience with using these and able to compare/contrast? I'm afraid of C++, but if I can wrap it and learn how to use from a python REPL, I think I can handle... any recommendations/tuotirals/howtos would be much appreciated. (The library I want to wrap is https://github.com/openzim/libzim https://github.com/openzim/libzim )
- tehsauce 9y agoI used pybind11 for my open source asteroid detection code. It's very slick, especially it's ability to seamlessly convert between stl and python lists- for example a vector of strings converts directly to a python list of strings. Also the interoperability with numpy. A class containing an stl of primitive types and the dimensions can easily be turned into a numpy array. Very neat
- tehsauce 9y agohttps://github.com/DiracInstitute/kbmod https://github.com/DiracInstitute/kbmod The project I was referring to :)
- first_amendment 9y agoI've used cython, swig, and boost.python. I prefer pybind11 to all of them. It requires the least extra tooling and allows you to seamlessly transition between C++ and Python using the idioms native to each language.
- ThePhysicist 9y agoI built bindings for various C++ libraries in SWIG (not only for Python but also Lua and Ruby) and I was always very impressed with the result. What's amazing about SWIG is its ability to easily port very complex and rich code. I successfully exported very complex class hierarchies and code relying heavily on pointers and STL to Python and/or Lua. The process consists of writing several header-like configuration files that SWIG ingests and uses to generate bindings, which is often pretty straightforward. Often you can simply import the normal C/C++ header files in those configs and add some glue code / extra hints to help SWIG in cases where it can't figure out what to do with a particular type. In my experience, the most difficult aspect of binding generation is not writing the glue code, but doing so in a scalable way. Manually writing bindings for any real-world C++ codebase would therefore be extremely tedious (IMHO), so having an automated system that does this work for you is a huge time saver. Some larger libraries / frameworks have their own wrapping generators Btw, PyQt (which provides extremely good bindings to QT) for example has SIP which is worth looking at (it's open source).
- teajunky 9y agoHas anyone looked at https://pypi.python.org/pypi/cppyy https://pypi.python.org/pypi/cppyy ? This seems to be even easier than pybind11
- aldanor 9y agoSome may find it useful: I've written a Jupyter notebook extension for pybind11 a while ago, so you can write C++ in the notebook and it gets automagically compiled and imported into the kernel (plus you get C++ syntax highlighting and a few other goodies): https://github.com/aldanor/ipybind https://github.com/aldanor/ipybind (still not on pypi but the plan is to release it soon)
- nmalaguti 9y agoAt what point should I start thinking about rewriting computationally expensive parts of my Python application in C++ and making bindings? Does anyone have experience taking one expensive method and replacing it with C/C++? Were the trade offs worth it?
- aldanor 9y agoA typical use case is pushing loops in the hot path of your code to the C++ space. A simple loop that does nothing like `for i in range(int(1e9)): pass` takes about 20 seconds to execute in Python on my machine, whereas in C++ the overhead would be thousands times smaller. Because there's also overhead for transferring objects to/from pybind11 (it has to keep track of object lifetimes, figure out the conversions, etc), it's generally more beneficial to wrap big chunks of logic in C++ rather than every single method.
- aldanor 9y agoFor the sake of illustration, there's a simple but realistic example at the very end of my talk at this year's EuroPython: https://speakerdeck.com/aldanor/pybind11-seamless-operability-between-c-plus-plus-11-and-python?slide=50 https://speakerdeck.com/aldanor/pybind11-seamless-operabilit... (reimplementing rolling_stats from pandas that ends up being faster)
- chombier 9y agoThe easiest would be to first give cython a try, as it can provide a nice performance boost at a minimum effort in some cases. If you just need to bind simple number crunching functions, you may also write a small library exposing them with c linkage, then access them using ctypes. For more advanced usage, you will probably benefit from using a binding framework instead of using the python c api.
- sametmax 9y agoFirst, try to optimize your Python. It's unexpected what you can do with it. E.G: slicing assignment is crazy fast, removing calls and lookups goes a long way, using built-ins + generators + @lru_cache + slicing wins a lot, etc. Also Python 3.6 is faster, so upgrading is nice. Then, you try pypy. It may very well run your code 10 times faster with no work on your part. If you can't or the result is not as good as expected, you can start using EXISTING compiled extensions. numpy, uvloop, ujson, etc. After that, and only now, should you think about a rewrite. Numba for number related code, cython for classic hot paths or nuikta for the entire app: they can turn Python code into compiled code, and will prevent a rewrite in a low level tech. If all of that failed, congratulation, you are part of the 0.00001%. So rewriting bottlenecks in a compiled language is a good option. C or C++ will do. But remember you can do it in Rust too!
- KKKKkkkk1 9y agoI've been using boost::python, and the experience is what I'd call cargo-cult programming. You copy and paste stuff off the tutorial, and when you get a compiler error, you just try other stuff at random, because the inner workings of the library are inscrutable. I vastly prefer cython to this. Is Pybind11 any better?
- autopoiesis 9y agoPybind is derived from boost::python, so it's not much different. I wrote a large Python binding in the past for a moving C++ library target, using boost::python, and keeping the C++ binding code up to date was a nightmare akin to maintaining a fork of any fast-moving project. Try cppyy [1]. It's very nice, though quite fresh. It's used extensively (as far as I can tell) at CERN, and derives from their cling C++ JIT interpreter. Plus it does lots of nice automatic transformations to make a C++ API more pythonic. [1] http://cppyy.readthedocs.io/en/latest/index.html http://cppyy.readthedocs.io/en/latest/index.html
- deleted 9y ago[deleted]
- wjakob 9y agoNot quite -- while pybind11 was originally inspired by boost::python, it is an entirely different project that is intended to address many of its problems.
- autopoiesis 9y agoIndeed; I was using "derived" somewhat loosely. But I think there is certainly a visible lineage there. Indeed, pybind11 is much nicer than boost::python (and I did evaluate it for the future of the project I mentioned above). However, the very nice feature of cppyy is that it does much (most?) of what pybind11 does, but it can also be completely on-the-fly, in the sense that it relies on the cling JIT interpreter. This means that there is absolutely no need to maintain a compiled C++ part for your bindings, and so the problem of keeping the interface up to date is greatly mitigated: the equivalent changes to match the interface when using cppyy are _much_ smaller. Often, one maintains a "C++ interface" layer in one's Python bindings (which could be created by pybind11, boost::python, or SWIG, for example), with a pure Python shim layer on top of that. cppyy allows you to do away with this two-layer structure entirely; all you need is the shim, if you need anything at all.