4 ms·
the author seems to be unaware of threading and event based models. multiprocessing adds memory isolation through the CPU's protected memory.
by zobzu 12y ago
the author seems to be unaware of threading and event based models.
multiprocessing adds memory isolation through the CPU's protected memory.
- rasbt 12y agoRight. Yes, I am aware of the threading module. It depends on the type of application what to prefer: memory-bound vs CPU-bound tasks
- Igglyboo 12y agoJust curious but could this be possible to get around by having multiple copies of the python interpreter installed on your sytstem? Maybe possibly changing some config strings so the GIL thinks it's something else.
- quadlock 12y agoProcessPoolExecuter from the futures package essentially does this, it runs multiple instances of the interpreter and then distributes instructions to them.
- deleted 12y ago[deleted]
- freyrs3 12y agoIf you're working on CPU-bound tasks with NumPy/SciPy using threads then you have to think very hard to make sure most of the critical sections are hitting the NumPy calls in C which release the GIL. It's not a great reliable way to program. The way the author describes is basically the only pure-Python way of achieving parallelism for this kind of problem.
- zobzu 12y agoit doesnt make it any less true. besides, that C python's GIL sucks doesnt mean threading or event models sucks. In doubt, ask nginx how they feel about that. Then ask apache's mpn-prefork how that feels.
- freyrs3 12y agoIf you're holding a global mutex every 100 instructions while context switching between CPU bound tasks, then yes, the GIL does suck. There are a class of IO-bound problems where threading/evented models in Python can be used effectively but that's not the class of problems the author is talking about here.
- nemothekid 12y agoConsidering the author's use case (mathematical modeling) and language (Python), threading and event-based models would have no real performance benefit. Event based models only shine when you are doing IO-bound tasks. They won't help you when you are chewing CPU. Threading models in Python aren't attractive because of the GIL. If you are doing a parallel matrix operation you can only ever use one CPU because of the GIL. Not attractive.
- chuckledog 12y agoAny thoughts how software-transactional memory might apply for a use case like this?
- freyrs3 12y agoSTM is not a great fit for this kind of problem, there's no need for all the transaction machinery if the problem is embarrassingly parallel. In an ideal world what you want is threads which just split the work sections like they do in C/C++/Haskell.