3 ms·
If you're looking for a bit more info on the Windows-only aspect, there is a lot of detail here: https://speakerdeck.com/trent/pyparallel-how-we-removed-the-gil
by trentnelson 6y ago
If you're looking for a bit more info on the Windows-only aspect, there is a lot of detail here: https://speakerdeck.com/trent/pyparallel-how-we-removed-the-gil-and-exploited-all-cores https://speakerdeck.com/trent/pyparallel-how-we-removed-the-....
I think the TL;DR though is that PyParallel was a successful proof-of-concept (but seems to have failed at moving the bar anywhere) because the threadpool and async I/O primitives on Windows are so much more sophisticated than what's available on any other platform.
On Linux/Mac, I'd have to write so much scaffolding to get the same baseline functionality offered by Windows, and many parts wouldn't even be possible to replicate. It'd be a huge engineering effort that would take a team of people. (Just like all the Vista+ threadpool stuff took a team of kernel engineers working for years at Microsoft.)
That being said, I've been looking at stuff like the Chromium cross-platform threadpool stuff recently and that could potentially be used as a substitute (I believe it maps 1:1 with native threadpool APIs on Windows, and mimics the best it can on Linux/Mac). But that's an unwieldy 3rd party package for Python to suddenly depend on.
I also disagree with the sentiment that the GIL, parallel computing and async I/O are all separate, orthogonal pieces. The reason PyParallel was so performant was the fact that I treated all three as very intertwined concepts that had to be addressed all at once.