3 ms·

> NumPy core developer Sebastian Berg chimed in as well. He suggested that it could take up to a solid year of work to support subinterpreters in NumPy. Whoa,

> NumPy core developer Sebastian Berg chimed in as well. He suggested that it could take up to a solid year of work to support subinterpreters in NumPy. Whoa, that's a shame. I actually found it really easy to patch NumPy to make it PyParallel-compatible -- simply had to tweak the memory allocator stuff: https://github.com/pyparallel/numpy/commit/046311ac1d66cec789fa8fd79b1b582a3dea26a8 https://github.com/pyparallel/numpy/commit/046311ac1d66cec78... Example that loaded a 12GB NumPy array and serviced requests in parallel: https://github.com/pyparallel/pyparallel/blob/branches/3.3-px/examples/wiki/wiki.py https://github.com/pyparallel/pyparallel/blob/branches/3.3-p.... I really wish the PyParallel approach gained more traction. Having the solution Windows-only reaaaaally didn't help with having other core developers experiment with the approach. [*]: https://pyparallel.org https://pyparallel.org

anentropic 6y ago
This seems great!! Obviously, being Windows-only is a big drawback - is there any hope it could be implemented for other platforms some day? (I mean in terms of technical feasibility, rather than effort, money etc) I know literally nothing about the OS side of things relevant to this topic, but googling for "async io linux" turns up io_uring which seems to date from 2019ish and is maybe addressing some of what's lacking there?
- trentnelson 6y ago
  If you're looking for a bit more info on the Windows-only aspect, there is a lot of detail here: https://speakerdeck.com/trent/pyparallel-how-we-removed-the-gil-and-exploited-all-cores https://speakerdeck.com/trent/pyparallel-how-we-removed-the-.... I think the TL;DR though is that PyParallel was a successful proof-of-concept (but seems to have failed at moving the bar anywhere) because the threadpool and async I/O primitives on Windows are so much more sophisticated than what's available on any other platform. On Linux/Mac, I'd have to write so much scaffolding to get the same baseline functionality offered by Windows, and many parts wouldn't even be possible to replicate. It'd be a huge engineering effort that would take a team of people. (Just like all the Vista+ threadpool stuff took a team of kernel engineers working for years at Microsoft.) That being said, I've been looking at stuff like the Chromium cross-platform threadpool stuff recently and that could potentially be used as a substitute (I believe it maps 1:1 with native threadpool APIs on Windows, and mimics the best it can on Linux/Mac). But that's an unwieldy 3rd party package for Python to suddenly depend on. I also disagree with the sentiment that the GIL, parallel computing and async I/O are all separate, orthogonal pieces. The reason PyParallel was so performant was the fact that I treated all three as very intertwined concepts that had to be addressed all at once.

anentropic 6y ago

This seems great!! Obviously, being Windows-only is a big drawback - is there any hope it could be implemented for other platforms some day? (I mean in terms of technical feasibility, rather than effort, money etc) I know literally nothing about the OS side of things relevant to this topic, but googling for "async io linux" turns up io_uring which seems to date from 2019ish and is maybe addressing some of what's lacking there?

trentnelson 6y ago
If you're looking for a bit more info on the Windows-only aspect, there is a lot of detail here: https://speakerdeck.com/trent/pyparallel-how-we-removed-the-gil-and-exploited-all-cores https://speakerdeck.com/trent/pyparallel-how-we-removed-the-.... I think the TL;DR though is that PyParallel was a successful proof-of-concept (but seems to have failed at moving the bar anywhere) because the threadpool and async I/O primitives on Windows are so much more sophisticated than what's available on any other platform. On Linux/Mac, I'd have to write so much scaffolding to get the same baseline functionality offered by Windows, and many parts wouldn't even be possible to replicate. It'd be a huge engineering effort that would take a team of people. (Just like all the Vista+ threadpool stuff took a team of kernel engineers working for years at Microsoft.) That being said, I've been looking at stuff like the Chromium cross-platform threadpool stuff recently and that could potentially be used as a substitute (I believe it maps 1:1 with native threadpool APIs on Windows, and mimics the best it can on Linux/Mac). But that's an unwieldy 3rd party package for Python to suddenly depend on. I also disagree with the sentiment that the GIL, parallel computing and async I/O are all separate, orthogonal pieces. The reason PyParallel was so performant was the fact that I treated all three as very intertwined concepts that had to be addressed all at once.