5 ms·
If you're using Python for performance-critical applications, simple use of the integrated ctypes module (uses libffi in background) can make a world of differe
by jnbiche 13y ago
If you're using Python for performance-critical applications, simple use of the integrated ctypes module (uses libffi in background) can make a world of difference. There's a modest performance overhead, but basically you're getting near C performance with proper use of ctypes.
Coincidentally, one of the usual examples given in the ctypes howto is a Point structure, just like in this post. It's simple:
from ctypes import *
class Point(Structure):
_fields_ = [("x", c_int), ("y", c_int)]
Then you can use the Point class the same way you'd use a regular Python class:
p = Point(3, 4)
p.x == 3
p.y == 4
Really, taking a half-day to learn how to use ctypes effectively can make a world of difference to your performance-critical Python code, when you need to stop and think about data structures. Actually, if you already know C, it's less than a half-day to learn... just an hour or so to read the basic documentation:
http://docs.python.org/2/library/ctypes.html http://docs.python.org/2/library/ctypes.html
If you plan to write an entire application using ctypes, it'd be worth looking at Cython, which is another incredible project. But for just one or two data structures in a program, ctypes is perfect.
And best of all, ctypes is included in any regular Python distribution -- no need to install Cython or any additional software. Just run your Pythons script like you normally do.
EDIT: I was just demonstrating how to get an easy-speed up using ctypes, which is included in the Python standard library. To be clear, you would usually use this type of data structure in ctypes along with a function from a C shared library.
Furthermore, if you're serious about optimizations, and you can permit additional dependencies, you should absolutely look at cython and/or numpy, both of which are much faster than ctypes, although they do bring along additional complexity. Other commenters are also pointing out cffi, which I've never used but also bears consideration.
- untothebreach 13y agoI am a python programmer who has yet to dive into ctypes, and I am curious if you have ever played with cffi[1], and if so, thoughts? The project seems to be aimed at the same use cases that you would use ctypes for. 1: http://cffi.readthedocs.org/en/release-0.8/ http://cffi.readthedocs.org/en/release-0.8/
- jnbiche 13y agocffi's API is similar to LuaJIT's, so if you're familiar and comfortable with that, then cffi might be a good option. If you're looking at creating your whole application around C extensions for Python, as in the case with some Cython apps, then libffi might be a good alternative. I've never used it, so I can't make any empirical observations about cffi (unlike Cython, which I've used -- it's awesome). The great thing about all these newish ways to interface with C code in Python is that we no longer have to write Python extensions in C. I like C, but writing Python extensions in C was very painful and tedious for me. So all these options are a good thing. Otherwise, I think ctypes is a more convenient option (it's always available, no extra installations required). Also, the ctypes API is much more Pythonic, in my opinion.
- untothebreach 13y agoThanks for the thoughts!
- vishvananda 13y agoI really like cffi for interacting with external c libraries. See for example my post on talking to openssl using cffi: http://unchainyourbrain.com/using-openssl-from-python-with-python-cffi http://unchainyourbrain.com/using-openssl-from-python-with-p...
- acdha 13y agoYou might also want to look at cffi – it's not in stdlib but the interface is pretty nice for this kind of thing: http://cffi.readthedocs.org/ http://cffi.readthedocs.org/ In the specific example above, however, I'd try to reduce that overhead by going even more C-style and allocating an array or two so you avoid the overhead of individual instances and instead have one Python object for many elements.
- jnbiche 13y ago>In the specific example above, however, I'd try to reduce that overhead by going even more C-style and allocating an array or two so you avoid the overhead of individual instances and instead have one Python object for many elements. How exactly would you avoid the overhead of individual instances using a ctypes structure? Sure, you can allocate an array very easily, using Point * 10 and then putting your instances in the array. In fact, you'd have to do this if you wanted a collection of these structs. I'm pretty sure that this array is stored continguously, but you still have the overhead of the Structure base class. How are you proposing to avoid this in ctypes? Or were you referring to cffi? If you really want to be serious about large, performance-intensive arrays in Python, it's time to pull out numpy. It actually would fit well here. Re: cffi, I do love luajit but I'm not a huge fan of its C interface. I've not used cffi, but it seems to be inspired by, or at least shares a lot with, luajit's C ffi. Which means you have to basically paste all the C header files into a string in your Lua or Python code to use the corresponding shared library. The beautiful thing about ctypes is that you don't have to declare all the function prototypes and type declarations when you use a shared library. You simply make sure to feed the functions the properly typed variables, and store the returned value properly, and it just works. As a result, I find ctypes a lot easier to use for simple optimizations. YMMV.
- acdha 13y ago> How exactly would you avoid the overhead of individual instances using a ctypes structure? This optimization only works if you have an upper bound for the total number of items but if so I'd at least try something like declaring point_x and point_y arrays so you'd have two Python objects accessed by indexes rather than many Point references. This technique can be particularly useful if you don't access all of the elements at the same time and you can get more efficient memory access patterns because you can access a subset of the elements in nice linear reads. > If you really want to be serious about large, performance-intensive arrays in Python, it's time to pull out numpy. It actually would fit well here. Very strong agreement - numpy is a treasure. > As a result, I find ctypes a lot easier to use for simple optimizations. YMMV. Ditto - I've found cffi to be faster in some cases but it's definitely more overhead to use.
- absherwin 13y agoUsing ctypes here is slower in CPython 2.7.5 than a simple class. Summing 10000 points takes 1.81ms using a simple class vs 2.22ms using the code above. ctypes is designed for compatibility not speed. Using slots is 0.06ms faster while namedtuple is slower. A bigger speedup in CPython can be achieved by just using the builtin sum function twice to avoid the for loop. Here's a gist: https://gist.github.com/absherwin/8976587 https://gist.github.com/absherwin/8976587
- jnbiche 13y agoPlease expand on your assertion. How are you storing your "two ints"? Two int tuple? Dict? List? I don't even have enough information to respond. I do agree that ctypes is primarily meant for compatibility, but it can be used for speed optimizations in certain cases, particularly when there is a speed penalty for non-continguous storage of data structures (like a list of dicts, or list of tuples). Edit: Thanks for the additional information. I totally agree that Python build-ins are often faster, as in this case. However, there are times when no build-in is available, and you have to use a custom function (usually from C). This gets back to the compatibility question, and I totally agree that ctypes is primarily meant for compatibility.
- absherwin 13y agoApologies for the confusion. The comparison was against a new-style class as described in the original post. All of the types have two ints. The code is now linked so you can inspect and respond. Edit to respond to edit: Built-ins and slots are even faster but the key point is that the ctypes-based class is actually slower than simply using a class with ordinary Python variables. The reason ctypes is slower is that addition isn't defined on c_ints which requires casting them back to Python ints for each operation. This can be avoided by using ctypes to call a C function to perform addition.
- jnbiche 13y agoResponse to edit: you may be right, I don't have time to do comprehensive benchmarking now. I never use ctypes on its own, but rather always with an external library. I assumed that the ctypes Structures in a ctypes array would be faster because its stored contiguously, but it's possible that is incorrect. I'll have to come back to this later when I have time. I stand by my assertion that ctypes along with an external C library is a great way to do Python speed-ups. It's very simple to do, see here: http://www.jiaaro.com/python-performance-the-easyish-way/ http://www.jiaaro.com/python-performance-the-easyish-way/ This is the kind of optimizations I usually use ctypes for, or for interfacing with a third-party shared library.
- Sauliusl 13y agoThis would be one of the approaches to use if you wanted to squeeze the last pieces of performance out of this class. I would recommend using Cython for this though. The problem with both of these approaches is that, well, use this too often, however, and you suddenly realise you are not really coding in Python anymore. Also, correct me if I'm wrong, overuse of tricks like these is one of the key reasons why we cannot have nice things like PyPy for all modules.
- jnbiche 13y agoI actually agree with everything you write here. If I could re-write the post to include these points, I would. I do think PyPy is very near to having full ctypes support, if they don't already.
- pekk 13y agoThis is like saying that if you call C programs from shell scripts, or run node programs that make use of C libraries, that you are just writing C.
- walshemj 13y agoIf you using python for performance critical work would you not prototype it in python and then rewrite it in C C++ or even Fortan if its HPC
- codygman 13y agoWould you call me a heretic for writing it all in Haskell then rewriting hot spots in C[1] if performance isn't good enough? 1. http://book.realworldhaskell.org/read/interfacing-with-c-the-ffi.html http://book.realworldhaskell.org/read/interfacing-with-c-the...
- pekk 13y agoNo. I don't know how Haskell fans feel about it, but I don't see anything wrong with it if it works and lets you write more good code than you would writing everything in C. C isn't that scary.
- freyrs3 13y agoC isn't that sca... Segmentation fault
- 1amzave 13y agoI've never understood why people always seem to immediately bring up segfaults as scary C behavior. Is it really that different than Python dying with "AttributeError: 'NoneType' object has no attribute 'foo'", or a NullPointerException in Java? Sure, in the latter two you get a stack trace, but you can rig up C to give you that too if you want. What's actually potentially scary in C is code that doesn't segfault, but just continues running along with silently-corrupted memory. (And that's something that doesn't have an analogous failure mode in most other languages, as far as I know.)
- andreasvc 13y agoYes it's actually really different because the error is non-specific (unhelpful) and can be caused by a completely unrelated operation, so debugging is harder.