11 ms·
Mixpanel: Scaling to the Internet - Writing C extensions for Python
- j_baker 16y agoDoes anybody write C extensions like this anymore? Most of the time it's easier to use swig or cython. If it's really simple, you can also use ctypes.
- jnoller 16y agoActually, a lot of people do write them this way. When you take something you can write quickly and with no dependencies other than python, and instead pull in swig or cython - you now have two problems instead of one. As the author shows, writing a python c extension is fundamentally easy which is why Python has such a huge ecosystem of them to begin with.
- cdavid 16y agoWriting C extensions is only easy if you do trivial things. As soon as you do something significant, writing all this C code by hand becomes increasingly complex, especially because of reference counting and error management (http://docs.python.org/release/2.5.2/ext/thinIce.html http://docs.python.org/release/2.5.2/ext/thinIce.html). If dependency for distribution is an issue, you can just include the generated C code (as we do in numpy and scipy, as a matter of fact). Also, cython can generate both python 2 and 3 compatible code.
- mturmon 16y agoIt depends on what kind of "trivial". I have a lot (> 30000 SLOC; hundreds of functions) of C code that does intricate math/stats/optimization, but from a control flow and memory allocation point of view is simple. This kind of code is perfect for the simple c-extension API of Python as described in the OP. In fact, if you set up the gateway interface to your C code cleanly, you can have the same C code callable directly from (e.g.) Matlab via the "mex" mechanism.
- cdavid 16y agoYep, in that case, that's relatively easy - but cython is even better. Cython is used a lot in the scipy community - if you don't know it, you may want to look at it. Since I am using cython, I am trying to avoid writing raw C API as much as possible, there is just no point.
- slug 16y agoI wrote a few things in cython to give it a try and it's amazing what static typing can do to improve the efficiency of the code. Just don't look at the auto-generated .c file though, it looks scary, but this seems to be a common occurrence (swig comes to mind).
- sfk 16y agoSWIG has its own problems. Dependencies have already been mentioned, trusting obfuscated code is another issue: http://www.artima.com/weblogs/viewpost.jsp?thread=95863 http://www.artima.com/weblogs/viewpost.jsp?thread=95863 "But even if I find the cause of the leak, I may not be able to plug it. M2Crypto is written using SWIG, which contains lots of obfuscated code and doesn't exactly give me a warm fuzzy feeling that it's always doing the reference counting correctly." I prefer a clean handwritten C extension any day. There is a lot of repetitive work involved, but most of that is really just copy&paste, so it tends to go quickly.
- StavrosK 16y agoThis is needlessly complicated. You can just use Cython and annotate the types, or, even better, use shedskin to compile your Python code, usually with no changes, to C++ and make a module out of it. There are many options to speed up your Python code by ~50 times effortlessly. See my blog posts: http://www.korokithakis.net/node/109 http://www.korokithakis.net/node/109 http://www.korokithakis.net/node/117 http://www.korokithakis.net/node/117
- hugh4life 16y agoKeep an eye out for Nuitka... http://kayhayen24x7.homelinux.org/blog/2010/09/nuitka-release-0-3-1/ http://kayhayen24x7.homelinux.org/blog/2010/09/nuitka-releas...
- StavrosK 16y agoI saw that, it looks quite good but I'm not sure how much better than Unladen Swallow it can get... RPython is another promising contender, if the PyPy guys ever polish it for end users...
- hartror 16y agoThese are great when you're worried about performant code: http://docs.python.org/library/profile.html http://docs.python.org/library/profile.html http://wiki.python.org/moin/PythonSpeed/PerformanceTips http://wiki.python.org/moin/PythonSpeed/PerformanceTips
- StavrosK 16y agoYep, they won't speed your code up 50 times, though.
- hartror 16y agoI am yet to be sold on using Cython or it's ilk but it is philosophical/personal preference thing rather than a commentary on the quality or usefulness of these projects. As I have no trouble switching code to C++ and integrating it with python these projects don't offer me the same thing as a coder with less C++ experience. As an aside some simple coding changes will allow Psyco to do much of the by hand optimisations that Cython gives you. Not all and it is easy for another coder to come along and ruin everything. All it takes is some understanding of how Psyco performs its magic.
- hartror 16y agoMy programming knowledge base is strongest with C++ and Python. I always start writing projects in Python. Then if psyco hasn't gotten me the performance in the (often very small) sections of code that do the heavy lifting a quick C++ extension is whipped up. This method is so productive it always surprises me how quickly things get done!
- StavrosK 16y agoIt's the best of both worlds, really. You write everything in a language that's fast to write, then you write the bottlenecks in a language that's fast to execute.
- cdavid 16y agoThis works only if your bottlenecks are a small portion of your work. For example, if your twisted application is slow, you will have a hard time optimizing it by writing a bit of C code.
- hartror 16y agoConfused. Why would replacing larger chunks of code be a problem?
- cdavid 16y agoBecause of the python <-> C marshalling cost. Let's say you have some code which uses a lot of small objects which interact with each other, maybe each object has a small list with a few integers. Replacing this in C would make sense because you can be much faster than python. But you still have the cost of interfacing with python (creating the objects, boxing integers back and forth, etc...). That's why something like numpy manages to be so fast if used well: it internally uses efficient (C-like) representation, but as soon as you need to interact with every item in your array, then it is not only much slower, but even slower than using standard python containers because of this cost.
- kqueue 16y ago>So imagine: You want to stick to Python because it’s so fast to develop in but need the performance of C/C++. Let me introduce you to C extensions in Python. Let me introduce you to Cython.
- nphase 16y agoWow, this looks so much easier than writing a PHP extension!
- slug 16y agoUntil you forget a decref/incref somewhere ;) Personally, I prefer to use either boost.python, cython, ctypes or swig. Although writing C routines to directly use numpy arrays is fairly easy.
- metamemetics 16y agoIs there any scalable\cloud python hosting that allows C extensions? I know app engine doesn't unless they changed it recently. I'm guessing a VPS is necessary?
- gpjt 16y agoI think you can upload your own extensions to Picloud.