3 ms·
I'm actually interested to know what the HN community does to achieve parallelism in Python? Do you use multiprocessing module, Stackless python, PP module (par
by amirmansour 14y ago
I'm actually interested to know what the HN community does to achieve parallelism in Python? Do you use multiprocessing module, Stackless python, PP module (parallel python)?
- willvarfar 14y agoI use workers (sometimes multiprocessing, sometimes I do my own starting of them). Often I have use-cases where my own shared memory system works well http://williamedwardscoder.tumblr.com/post/13363076806/buffcacher http://williamedwardscoder.tumblr.com/post/13363076806/buffc... > I was writing a custom video editing program in Python (interesting choice of language for that problem) and I wanted to cache decoded frames
- dagw 14y agoEither multiprocessing or shifting the bits that really benefit from parallelization out to C/C++ tend to cover the vast majority of my needs. Also numpy and numexpr does a lot of the multiprocessing stuff for me, so in many cases I don't even have to think about it.
- deleted 14y ago[deleted]
- dbaupp 14y agoFor sufficiently new version of Python (3.2+), concurrent.futures[1] is pretty nice for its use-case, but otherwise multiprocessing has been quite good. [1]: http://docs.python.org/py3k/library/concurrent.futures.html http://docs.python.org/py3k/library/concurrent.futures.html
- blitzprog 14y agoI'm using QThread for PyQt GUI applications. However the GIL makes it impossible to execute _Python_ code in parallel. I can understand the reasons for that, however it doesn't change the fact that it makes my application inherently slower and is therefore something I dislike. Apart from that I really like Python.
- smeg 14y agoSame here. As a nice side effect, QThread somehow makes signals work again, so you can Ctrl-C a MT Python program (you still have to set a handler for SIGINT but at least it works). For this reason, I actually use QThread in console apps aswell.
- sateesh 14y agoIf you are using QThread for console apps and are not using any other Qt libraries, isn't your console app has an unnecessary dependency on Qt ?
- smeg 14y agoWell I guess it is "unnecessary" to the extent it is unnecessary to be able to Control-C your programs. Personally I like my programs to respond to signals.
- lmm 14y agoQt's split up into separate libraries; QtCore is pretty much a general portable utility library like glib or APR, and it's perfectly reasonable for a console app to depend on it.
- heretohelp 14y agomultiprocessing or rewrite in something else, depending on the nature of the problem.
- Nursie 14y agoThat's usually the point at which I decide the project could really use being rewritten in C. If I need performance badly enough that I'm considering making major structural changes in order to utilise multi-process or find some other way around the GIL, I probably ought to be considering using a language that supports what I'm trying to do. Of course my python projects tend to be fairly small.
- smeg 14y agoOr just write the fast bits in C, call it from Python via ctypes and get the GIL unlocked and parallelism for free.
- Nursie 14y agoIf my projects were large enough to make that a more attractive option, sure, that's a great plan. Usually when I've hit this situation I'm writing something simple but processor intensive, and it's easier for me (12 years commercial parallel/distributed C programming, amateur python enthusiast) to go back to home ground at this point.
- cageface 14y agoIf the problem at hand isn't easily solved with a multi-process approach then this would probably be the time I'd start taking a hard look at another language. Scala, Go, Clojure etc make this so much easier that I think a language jump is reasonable.