5 ms·
I guess this is supposed to cure the GIL curse of threading in Python? i.e. the fact that only one thread can be running in a n interpreter process. There is
by t0mk 10y ago
I guess this is supposed to cure the GIL curse of threading in Python? i.e. the fact that only one thread can be running in a n interpreter process.
There is still the multiprocessing module which can spawn processes, so you can effectively run code in parallel. It's pain to manage though.
I think in the end I am grateful for no parallel threading in Python as it forces me to either do things so that they can be run in naive-parallel, or to use things that have the concurrency solved out of Python.
- gnipgnip 10y agoSadly you can only share pickleable objects in python multiprocessing.
- radarsat1 10y agoOr shared memory: http://stackoverflow.com/questions/5549190/is-shared-readonly-data-copied-to-different-processes-for-python-multiprocessing/5550156 http://stackoverflow.com/questions/5549190/is-shared-readonl... (but it's a little hacky..)
- cocoablazing 10y agoStill, multiprocessing objects like Array and Manager are limited to ctypes.
- sevensor 10y agoSometimes worth it, but often not. The ability to do shared-memory multi-threading is one of the things that tempts me away from Python. Message passing is great and all, but sometimes you want your messages to be passing around control of a shared 4GB data structure, instead of trying to copy it.
- sdbrown 10y agoA little hacky, yes, but extremely effective. I wrote an image processing application for somewhat large time-series datasets (> 1TB) on Linux which took liberal advantage of these details to run very nicely on v2 Xeon processors. It also worked quite well for GUIs which interacted with the datasets.
- Sean1708 10y agoThis looks more like MPI style parallelism (usually used on things like supercomputers and clusters) than anything the standard library provides.