6 ms·
This is just my opinion, but it's served me well in the 8 or so years I've done Python development: Why are you doing compute intensive work in Python? Python
by falcolas 13y ago
This is just my opinion, but it's served me well in the 8 or so years I've done Python development:
Why are you doing compute intensive work in Python? Python is not well optimized for doing compute intensive work. It's made some strides over the years, being able to do limited bytecode optimization and multiprocessing, but it's not a high performance computation language.
However, it is a fantastic glue language. Write your compute intensive portions in C, and use Python to glue the portions of C together. You can even do your thread generation in Python, release the GIL during your C execution, and you don't even have to worry about the complicated process of spinning up threads in your C.
In other words, use the language like it was designed to be used.
- cgh 13y agoNo offense, but I don't think you understand what numpy and scipy are. Numpy is the specific use case mentioned in the post.
- falcolas 13y agoI know precisely what Numpy and scipy are. Their choice to not release the GIL during their operations is not necessairly a limitation of Python. Of course, not releasing the GIL eases interaction with Python level variables, so that might be a tradeoff they decided is worth losing threading for. But please don't blame Python for the choices made by a library developer.
- pwang 13y ago> Their choice to not release the GIL during their operations is not necessairly a limitation of Python. Um, what? They do not hold on to the GIL when doing compute intensive operations.
- bad_user 13y ago> Write your compute intensive portions in C This advice is so often given by people, that I suspect that most people don't know what they are talking about, because in fact most people never do that, being unable to drop to C. I dare you to show me a piece of "compute intensive" portion that you optimized with C. I've worked with Python for about 3 years. After struggling with it to stretch the boundaries of what it could do, I finally gave up in frustration and the final solution was to find a less limiting environment. Worked great thus far and I don't miss anything about Python.
- beambot 13y agoWe do this in robotics (research) all the time, eg. for image and pointcloud processing. I've implemented algorithms in plain python, numpy, and then OpenCV / PCL. When a compute-intensive task starts becoming painful, it's profiled, improved, and/or eventually forked into a C project with python-friendly wrappers. I acknowledge this is very anecdotal... but my (previous) labs observed a tradeoff in developer productivity in Python vs C, and deemed Python worthwhile (avoiding classic premature optimization and whatnot).
- falcolas 13y agoI can't show, because it was at a job I no longer work for, but I've done it a fair bit (and it's extensively done in the standard library as well). I could ask you to trust me, but it is the internet. Honestly, it's not that hard to do - mostly a bit of wrapper code to announce to Python what methods you're exposing, and doing coercion from Python types to C types. The hardest part for me was identifying when to increment and decrement reference counters to variables being passed in to me.
- coldtea 13y ago>Why are you doing compute intensive work in Python? Because I'd rather have a unified code base, all higher level, than some BS mix of Python and C or other stuff that I'd have to support. >Python is not well optimized for doing compute intensive work That's a rather circular argument, when the topic under discussion is why Python is not well optimised for doing compute intensive work.
- falcolas 13y ago> Because I'd rather have a unified code base, all higher level What higher level language offers you the high performance computing you desire? > That's a rather circular argument, when the topic under discussion is why Python is not well optimised for doing compute intensive work. That's not the original post I read - it was berating Python for not being a high performance computing language. To which I reply: Of course it's not. It's the wrong tool. Expecting a mature high level glue language to also be exceptionally high performance... that language doesn't exist yet. Go is a good step along that route, but it's own hacky interoperability with C and lack of maturity creates its own problems.
- yummyfajitas 13y agoWhat higher level language offers you the high performance computing you desire? Scala and Haskell both strike a nice balance for my uses. They aren't quite as fast as C, but give me decent concurrency primitives and performance. (You might be surprised to discover that my use cases are primarily the realtime event processing systems described in the article.)
- coldtea 13y ago>What higher level language offers you the high performance computing you desire? The major pain problem with other high level languages is the lack of a large library ecosystem (to replace NumPy et al). As for which high level languages that beat the pants out of Python for high performance computing, there are plenty. Julia for one. Haskell for another. Scala. Go. LuaJIT It's a long list. Heck, even V8 is miles ahead.