4 ms·
It's not specific to Python. If you want to do fine-grained "parallelism" on a set of complex shared datastructures, your cores might look busy, but really you'
by incrudible 5y ago
It's not specific to Python. If you want to do fine-grained "parallelism" on a set of complex shared datastructures, your cores might look busy, but really you're just wasting most of the time locking/unlocking stuff. That's not necessarily going to result in a speedup over a lock-free single-threaded implementation (that you didn't implement to compare against).
In the case of Python, the single-threaded implementation was already there, the implementation that removed the GIL in favor of fine-grained locking failed to deliver the goods.
If you want to actually take advantage of parallelism, you want data structures that are amenable to it, which is basically sections of arrays without a lot of potential read/write conflicts over multiple threads. If you're at that point, you probably want to use C/C++/Rust anyway, you don't want speed up something that's already 100x slower (the interpreter) by parallelizing it. Python offers that with C-extensions like NumPy.
- amelius 5y ago> your cores might look busy, but really you're just wasting most of the time locking/unlocking stuff. Not if you're reading most of the time and are using a modern concurrent GC.
- incrudible 5y agoIt doesn't matter if you're reading or writing, if there's a potential conflict you need to lock, or you need some complex and probably buggy "lock-free" data structure. The complexity and memory overhead of a concurrent GC just goes on top of that.
- quietbritishjim 5y agoLocking an uncontended lock is cheap (at least compared to a contended one) so actual conflict does matter more than potential conflict.