5 ms·
(Standard reponse) Python does support parallelism, so long as you're calling into a C library to do the work, because they (usually) release the GIL while doi
by quietbritishjim 5y ago
(Standard reponse)
Python does support parallelism, so long as you're calling into a C library to do the work, because they (usually) release the GIL while doing the work. That sounds like a cop out but using C libraries is very common to do CPU intensive work anyway. Examples include numpy (e.g. you can call np.dot from multiple threads in Python and they will genuinely use multiple cores) and all the rest of the scipy stack, and even modules in the standard library (e.g. when using zipfile, even to process an in-memory buffer, the GIL is released).
- amelius 5y agoNot really. You can't build large data structures (not arrays, but trees, graphs, etc) with structural sharing between threads. Unless of course you translate all of your data to C too, at which point you are really doing everything in C (data and code).
- incrudible 5y ago> You can't build large data structures (not arrays, but trees, graphs, etc) with structural sharing between threads. You probably do not want to do that in the first place. The reason that the GIL still exists is that the overhead of fine-grained locking for Python datastructures would negate the speedup from multithreading.
- amelius 5y agoWhich is why Python doesn't really support parallelism. Other language (like GoLang) do support parallelism but they have a more advanced runtime and don't use a GIL.
- incrudible 5y agoIt's not specific to Python. If you want to do fine-grained "parallelism" on a set of complex shared datastructures, your cores might look busy, but really you're just wasting most of the time locking/unlocking stuff. That's not necessarily going to result in a speedup over a lock-free single-threaded implementation (that you didn't implement to compare against). In the case of Python, the single-threaded implementation was already there, the implementation that removed the GIL in favor of fine-grained locking failed to deliver the goods. If you want to actually take advantage of parallelism, you want data structures that are amenable to it, which is basically sections of arrays without a lot of potential read/write conflicts over multiple threads. If you're at that point, you probably want to use C/C++/Rust anyway, you don't want speed up something that's already 100x slower (the interpreter) by parallelizing it. Python offers that with C-extensions like NumPy.
- amelius 5y ago> your cores might look busy, but really you're just wasting most of the time locking/unlocking stuff. Not if you're reading most of the time and are using a modern concurrent GC.
- incrudible 5y agoIt doesn't matter if you're reading or writing, if there's a potential conflict you need to lock, or you need some complex and probably buggy "lock-free" data structure. The complexity and memory overhead of a concurrent GC just goes on top of that.
- quietbritishjim 5y agoLocking an uncontended lock is cheap (at least compared to a contended one) so actual conflict does matter more than potential conflict.
- quietbritishjim 5y agoOK, let's put it like this: In some contexts, but not all, Python supports a useful amount of parallelism. I have definitely worked on Python projects in the past that were easily parallisable and worked absoluately fine. I simply created threads with threading.Thread, passed messages around with queue.Queue and used some standard Python modules to process those messages. The messages were independent enough from each other that I didn't have any issues with marshalling access to them, while being large enough that the overhead of mulithreading was still outweighed by its benefits. Rewriting the whole thing in C++ (etc.) would have made negligable difference to performance. I can certainly imagine a program similar to what you describe, where there are lots of messages that operate on a single monolythic data structure, which couldn't easily be made to work with true parallelisation in Python.
- dekhn 5y agoif you call np.dot on the same underlying ndarray from multiple threads, what are the semantics? I don't know what they are but I think most people want a single invocation of np.dot in a single thread to use multiple worker threads (as described in https://scipy.github.io/old-wiki/pages/ParallelProgramming https://scipy.github.io/old-wiki/pages/ParallelProgramming) by pushing the work to a thread-aware matrix library.
- dekhn 5y agoHi, would the downvoter mind adding a comment explaining why? What I said is technically correct so it's unclear what the objection is.
- dekhn 5y agoNo need to downvote a request for explanation; this is within HN rules. Otherwise, you're giving an ambiguous signal so I don't know whether you're unhappy with the current state of Python, or disagree with my technical statement (in which case, please provide a counterargument instead of a vote).
- pitaj 5y agoFYI complaining about votes is against HN comment policy.
- dekhn 5y agoI'm not complaining about votes. I'm asking for people to clarifying why they are downvoting a technically correct statement.
- dragonwriter 5y agoRequesting explanation of downvotes is among the things the policy against discussion of the votes on a post is directed at, and, IMO, also demonstrates a fundamental failure to understand (or respect) the purpose of downvotes, which is about managing signal-to-noise ratio by, among other things, not polluting threads with meta-level discussion (abd invariably debates) of why a particular comment is (or is not) inappropriate.
- garyrob 5y ago"Python does support parallelism, so long as you're calling into a C library to do the work" Actually, you can use Numba to get the same effect while still writing Python. Numba allows you to simply apply a decorator to critical Python code and it is JITted into C-like performance. I do this in my code, and it has worked great for my uses.