2 ms·
Python threads are real OS threads that are scheduled by the kernel. Yes, there is a GIL, but it is released for blocking IO. Granted CPU heavy loads in threa
by influx 12y ago
Python threads are real OS threads that are scheduled by the kernel. Yes, there is a GIL, but it is released for blocking IO. Granted CPU heavy loads in threads that are pure Python are not great, but there is a subtle difference here most people miss.
The multiprocess module solves this and has almost exactly the same interface as the threading module, I'm confused why the author paints it as more complicated.
That said I have nothing against R, or the use case the author presents.
- bwood 12y agoUsing multiprocessing in Python is more complicated because each process has a separate memory space and interprocess communication can be quite expensive. If you don't carefully design your communication patterns, any performance you expect to gain by using multiple cores can easily be lost.
- influx 12y agoYou'd be mistaken to think just because you share memory space that you don't have to think about or design around the same issues with threads. The main issue I've ran into with multiprocessing is objects that are unpickleable, otherwise, it's pretty transparent usually.
- bwood 12y agoThat's interesting, what sort of communication issues have you run into with threading? I've always found it to be mostly a non-issue since you can use global variables or just pass along a reference.