7 ms·

Solving multi-core Python

166 points by ot 11y ago

zabbadabba 11y ago
This proposal is needlessly complicated. All Python has to do is put all interpreter state into a single struct to allow for completely independent interpreter instances. Then the GIL can be discarded altogether. And finally they have to change the C API to include a PythonInterpreter* as the first parameter rather than relying on the thread-local data hack.
- JoshTriplett 11y ago
  Interpreter state isn't the only problem; if that were the only issue, it'd be a more tractable problem. However, built on top of CPython's C API are a huge pile of libraries that assume they can manipulate global or per-data-structure state without locking.
  - viraptor 11y ago
    While technically correct, it's not as bad as explained here. They don't assume they can manipulate global state. They are guaranteed that it's right because native calls happen with GIL in place. That's an existing and real contract. If you don't want the lock in your library, you can release it. But your native code will always start in GIL-locked state.
  - zabbadabba 11y ago
    Avoiding fixing the C API for the sake of backwards compatibility is the reason for the GIL mess. Python 3 had a chance to correct the API situation since they were starting anew, but it didn't happen. All these API half measures are just dancing around the real problem of not having truly independent interpreter instances.
    - pekk 11y ago
      Python 3 wasn't really starting anew, they were also balancing the ongoing screaming about every difference from Python 2.
      - zabbadabba 11y ago
        It is curious how they completely revamped Python making most Python3 code incompatible with Python2 and yet they wanted to preserve C API "compatibility". Compatibility with what? Most Python3 modules had to be rewritten anyway - it was the perfect opportunity to fix the C API and get rid of the GIL once and for all.
        ericsnow 11y ago
        you forgot your <hand-wavy gross exaggeration> tags
  - frozenport 11y ago
    I don't understand. Can you give an example of a library that has a global resource that isn't thread safe? Also how is this different then problems faced when using multiprocessing style parallelism with the same library?
    - JoshTriplett 11y ago
      A few random examples, all from in-tree Python modules that are completely typical examples of what you'd find out-of-tree as well: The readline module uses libreadline, which maintains internal global state on the C side. The standard library has lists and dictionaries, which are not thread-safe to access from two threads simultaneously, and are not in any way locked.
      - exDM69 11y ago
        > The standard library has lists and dictionaries, which are not thread-safe to access from two threads simultaneously, and are not in any way locked. Locking lists and dicts, etc would not be a reasonable thing to do. There must be some granularity in synchronization, it's clearly not a good idea to attempt to allow mutating any object from multiple threads without explicit locking. Things like readline are issues that programmers have to deal with when writing multi threaded code in any language. There are libs which are not thread safe or which have global state or need special handling with threads (e.g. OpenGL). That's no reason to disallow threading completely. However, this might be a big culture shock issue if Python were to suddenly start supporting proper concurrency without GIL. None of the libraries document whether they are thread safe or not, so there would be surprises ahead.
        Erwin 11y ago
        Locking the basic data structures seems like it would destroy Python's simplicity. How do you determine the scope of the per-list lock? Do you add a lock to every single data structure that you are required to grab before reading or modifying it? If you require the programmer to do it, you can easily get race conditions which will only show up once in a blue moon causing incorrect results or corruption at worst - something Python's GIL guaranteed not to happen (as long as you didn't do anything risky in your C extension). Is there any solution that doesn't use purely functional data structures?
- comex 11y ago
  That would help, but in the chunk of the thread I read, most of the debate was about what to do after that: specifically (a) avoiding the cost of starting up a new interpreter, especially module import work, and (b) possibly taking advantage of the shared address space to be able to share some immutable objects somehow, rather than copying.
- srean 11y ago
  That's more or less how Tcl does it. They have had this support since ever. One coordinated by passing messages to the different Tcl interpreters, each running in their own thread.
- smegel 11y ago
  > All Python has to do is put all interpreter state into a single struct to allow for completely independent interpreter instances. But that is not what threading means. If you want "completely independent interpreters" just fork. Threading means you share everything. > Then the GIL can be discarded altogether And then you start sharing variables between interpreters, and you gets lots of little baby GILs to contend with! How fun will that be? Not very.
  - Frondo 11y ago
    That's how Tcl does it. Each thread gets its own Tcl interpreter, and you communicate among them via message passing. It lets you avoid the context switch that way.
    - frozenport 11y ago
      There is a difference between threads and processes. What you outline is how the python multiprocessing library works.
      - sitkack 11y ago
        Think of it like running multiple python processes inside the same memory space so that copying between interpreters is by reference, not value. Nothing has to be serialized.
        deleted 11y ago
        [deleted]
        onnoonno 11y ago
        How do you avoid damaging the consistency of Python objects then? I think the only way to do that would be to have a reentrant lock for each object individually. Isn't that potentially a big performance issue? I think that is the main reason behind this quote from the article: > Several attempts have been made over the years and failed to do it without sacrificing single-threaded performance.
- mappu 11y ago
  PHP did this (conditionally, at compile-time) with its ZTS builds, to allow multiple PHP runtimes in the same memory space. There is a not-insignificant performance loss from running this way, and so it's just not done, at least on Linux and similar scenarios where multiprocessing is feasible instead. Although PHP7 does have some workarounds to improve the performance of this mode, involving implementing thread-local storage on certain operating systems..
- sitkack 11y ago
  The Lua interpreter interface is so much cleaner, the number of globals in CPython is mind numbing. I looked into for a quick hack, not possible. At least 100+ hrs of work to put the globals inside a struct.
  - trentnelson 11y ago
    For PyParallel I just altered the decl for global variables to be TLS static, then tweaked initialization, and voila, thread safe globals. E.g. https://bitbucket.org/tpn/pyparallel/src/3be2954508f9938b85ae1bbc5c41587ec26b764a/diffs/Objects/bytesobject.c.patch?at=3.3-px https://bitbucket.org/tpn/pyparallel/src/3be2954508f9938b85a...
    - zabbadabba 11y ago
      The problem is that thread local storage is slow. It's an extra layer of indirection. Also, thread safety is just one concern in the GIL debate. One should be able to support many low-overhead independent interpreter instances on a single thread rather than relying on an arcane interpreter context switching scheme as presently exists in Python 2 and 3.
      - trentnelson 11y ago
        > The problem is that thread local storage is slow. It's slower than accessing a struct, sure: 5982: if (ctx) { 000000001E1A808B 8B 0D FF 7C 28 00 mov ecx,dword ptr [_tls_index (1E42FD90h)] 000000001E1A8091 BA 80 29 00 00 mov edx,2980h 000000001E1A8096 48 89 83 90 02 00 00 mov qword ptr [rbx+290h],rax 000000001E1A809D 65 48 8B 04 25 58 00 00 00 mov rax,qword ptr gs:[58h] 000000001E1A80A6 48 8B 04 C8 mov rax,qword ptr [rax+rcx*8] 000000001E1A80AA 48 8B 0C 02 mov rcx,qword ptr [rdx+rax] 000000001E1A80AE 48 85 C9 test rcx,rcx 000000001E1A80B1 74 70 je new_context+253h (1E1A8123h) But think of the big picture: I use TLS everywhere for PyParallel, and PyParallel has awesome performance, so eh, net win. > One should be able to support many low-overhead independent interpreter instances on a single thread Why? What problem does that solve?
maxerickson 11y ago
In case it isn't obvious, this got lots of discussion on the mailing list it was posted to: https://mail.python.org/pipermail/python-ideas/2015-June/thread.html#34177 https://mail.python.org/pipermail/python-ideas/2015-June/thr...
mianos 11y ago
If this comes I'll move from 2.7 to a 3.x
smegel 11y ago
Wow, Python will become the new Perl, how cool is that?! Not very.
- coldtea 11y ago
  What does that even mean?! Not much.
BuckRogers 11y ago
What problem does this solve? It does not solve the multicore "problem". I'm not a huge fan of this proposal. A few reasons. This doesn't really add much that wasn't possible before, and doesn't really solve any technical or PR issue. The PR issue will never be resolved with CPython, those people who don't understand it are free to write multithreaded Java apps. But I think explicitly spinning up pthreads should be reserved for writing systems software. I'm assuming subinterpreters means they'll be allocated as pthreads because this is meant to "use all your cores". Looking forward, this proposal sounds great as long as we're on 4 or 8 cores max, at a certain point this solution starts to look like something like a gimmicky ideal created to fit the technology way back in 2015. The ultimate multicore and multinode solution at a non-systems level, if we really need every single language to solve that specific problem- would be Erlang's approach. It's yet more technical churn rather than innovation in a language (Python3), that was born of technical churn. To offer something constructive as well, I think an ideal solution would be to find a more implicit approach. Think gevent for pthreads or subinterpreters. That would be a lot more work to figure out, this proposal looks more like a hack. I don't think this type of "improvement" will draw people to Python3 either, I wish they'd stop throwing crap at the wall seeing what will stick. Constantly expanding the language, which is bad. The core dev team doesn't know what to do, but usually in that case it's better to do nothing. Thus Python3 is looking more and more like a playground or experimental branch as time goes on. I'm increasingly thinking my "Python3 migration" will be to Swift or Go.
- pekk 11y ago
  I couldn't detect any real proposal in anything you said except that Python programmers should switch to Swift, and your main thesis seems to be that Python 3 is bad because it changes.
  - sfk 11y ago
    Even for HN standards, your comment is one of the most peculiar summaries I've read.
- mamerhst 11y ago
  I agree with this guy. I was not aware of the multicore "problem" So, basically problems that don't exist are made up and then complicated solutions proposed. Does seem to go well with the python3 philosophy overall.
crdoconnor 11y ago
>However, in CPython the GIL means that we don't have parallelism, except through multiprocessing which requires trade-offs. I can't speak for anybody else, but I personally haven't felt that inconvenienced by these trade offs at all. The major one is needing to pickle objects before sending them back and forth between processes. Since I prefer to keep the thread interfaces as tight as possible anyway, this only ended up being a major problem once - when I was trying to pickle a stacktrace before sending it to the parent process.
- ant6n 11y ago
  multiprocessing can be fickle. In ipython, ctrl-c during a computation using multiprocessing results in general unhappiness, and will probably be followed by a 'killall python'
  - vosper 11y ago
    Oh, yeah - and this one! Much more so than the inconvenience I posted!
  - crdoconnor 11y ago
    I've had this problem too, but after doing the (quite considerable) leg-work to fix it involving messing around with process groups and the ridiculous number of different types of signal, I'm more inclined to blame the horrible 30 year old UNIX APIs than python. I've had this signal handling problem in single process/single threaded code too, and in other languages. True, it would be nice if python layered on a nicer set of APIs to deal with the nastiness.
    - logicchains 11y ago
      >I'm more inclined to blame the horrible 30 year old UNIX APIs than python. Datapoint: I encountered the same thing on Windows, having to use the Powershell equivalent of `killall python` to deal with it.
    - easytiger 11y ago
      No, this is pretty easy in c. i think the python docs even apologise for it somewhere.
ngoldbaum 11y ago
The situation around multiple cores in Python (up to and including the link in the OP) is very nicely summarized in these notes by Nick Coghlan: http://python-notes.curiousefficiency.org/en/latest/python3/multicore_python.html http://python-notes.curiousefficiency.org/en/latest/python3/...
- dtxcoolbits 11y ago
  "...it’s very easy to envision a future where CPython is used for command line utilities (which are generally single threaded and often so short running that the PyPy JIT never gets a chance to warm up) and embedded systems, while PyPy takes over the execution of long running scripts and applications, letting them run substantially faster and span multiple cores without requiring any modifications to the Python code" IMHO this is the most sensible and logical approach and doesn't break any existing code. This also seems like what will likely happen anyways. Out of all the options PyPy-STM seems like the best approach because: "pypy-stm is fully compatible with a GIL-based PyPy; you can use it as a drop-in replacement and multithreaded programs will run on multiple cores." http://pypy.readthedocs.org/en/latest/stm.html http://pypy.readthedocs.org/en/latest/stm.html
trentnelson 11y ago
I attempted to describe the progress I've made with PyParallel already here: https://mail.python.org/pipermail/python-ideas/2015-June/034342.html https://mail.python.org/pipermail/python-ideas/2015-June/034... And here: https://mail.python.org/pipermail/python-ideas/2015-June/034260.html https://mail.python.org/pipermail/python-ideas/2015-June/034... I'm getting excellent scaling and performance across the board for the simple TEFB tests (https://bitbucket.org/tpn/pyparallel/src/8528b11ba51003a9821ceb75683ee96ed33db28a/examples/tefb/tefb.py?at=3.3-px https://bitbucket.org/tpn/pyparallel/src/8528b11ba51003a9821...), and have implemented something that really shows where PyParallel shines: an instantaneous wiki search REST API: https://bitbucket.org/tpn/pyparallel/src/8528b11ba51003a9821ceb75683ee96ed33db28a/examples/wiki/wiki.py?at=3.3-px https://bitbucket.org/tpn/pyparallel/src/8528b11ba51003a9821.... Quoting: I particularly like the wiki example as it leverages a lot of benefits afforded by PyParallel's approach to parallelism, concurrency and asynchronous I/O: - Load a digital search trie (datrie.Trie) that contains every Wikipedia title and the byte-offset within the wiki.xml where the title was found. (Once loaded the RSS of python.exe is about 11GB; the trie itself has about 16 million items in it.) - Load a numpy array of sorted 64-bit integer offsets. This allows us to do a searchsorted() (binary search) against a given offset in order to derive the next offset. - Once we have a way of getting two byte offsets, we can use ranged HTTP requests (and TransmitFile behind the scenes) to efficiently read random chunks of the file asynchronously. (Windows has a huge advantage here -- there's simply no way to achieve similar functionality on POSIX in a non-blocking fashion (sendfile can block, a disk read() can block, a memory reference into a mmap'd file that isn't in memory will page fault, which will block).)
- scott_s 11y ago
  Apparently, you should expect to hear from him soon: https://mail.python.org/pipermail/python-ideas/2015-June/034275.html https://mail.python.org/pipermail/python-ideas/2015-June/034... Antoine Pitrou said that Snow's proposal was similar to your work. What do you see as the similarities?
  - trentnelson 11y ago
rurban 11y ago
This looks like a new Python 4, but why not. It's a big problem that needs to be resolved for once.
angry_octet 11y ago
It seems to me that there is alot of muddled explanation going on here, if not muddled thinking. The goal fine grained parallelism python (fgpp) must be to improve the speed of code for which fat parallelism (user conconcted error prone 'multi-threading' or message passing) doesn't gain much, and for which calling out to an impl in another language or specialised library has overhead or doesn't make sense. IMHO, the most important part of fgpp must be that it is easily reasoned about, both by compiler/runtime and programmers, so as to maximise scope for transformations and user-driven design decisions (as opposed to VOODOO). It would ideally avoid the costs of premature optimisation, or the often inefficient flattening required for vectorisation. So I think any propsal should be addressing these questions, with appropriate benchmarks too, before it can be considered seriously.
- ericsnow 11y ago
  Thanks for the feedback.
Scramblejams 11y ago
Here's what I want: Something with programmer productivity similar to Python, but type-inferred for predictably good execution speed, with more support for functional programming style, with green threads preemptively executing in parallel and immutability baked deeply into the whole thing. And then a track record of deployment so the ugly GC and type system edge cases are ironed out and libraries are plentiful. Elixir/Erlang will probably never be fast enough, Go didn't commit to immutability and its functional programming support is minimal, Rust could have all that but it's too low-level... I guess OCaml is as close as it comes for the moment -- it's got everything I listed except a reasonable parallel execution story, but it looks like that's being worked on at the moment.
- beagle3 11y ago
  I don't think there's anything quite like that, but Nim seems to be a better fit than OCaml.
  - Scramblejams 11y ago
    Tried that. It's quite a big and complex language, there are lots of rough edges (I had it crashing from a double free within 5 minutes of trying it out), the bus factor is in the basement and although I didn't stick with it long enough to know for sure, there's much more mutability there than made me comfortable during my short sojourn.

166 points by ot 11y ago

This proposal is needlessly complicated. All Python has to do is put all interpreter state into a single struct to allow for completely independent interpreter instances. Then the GIL can be discarded altogether. And finally they have to change the C API to include a PythonInterpreter* as the first parameter rather than relying on the thread-local data hack.

JoshTriplett 11y ago
Interpreter state isn't the only problem; if that were the only issue, it'd be a more tractable problem. However, built on top of CPython's C API are a huge pile of libraries that assume they can manipulate global or per-data-structure state without locking.
- viraptor 11y ago
  While technically correct, it's not as bad as explained here. They don't assume they can manipulate global state. They are guaranteed that it's right because native calls happen with GIL in place. That's an existing and real contract. If you don't want the lock in your library, you can release it. But your native code will always start in GIL-locked state.
- zabbadabba 11y ago
  Avoiding fixing the C API for the sake of backwards compatibility is the reason for the GIL mess. Python 3 had a chance to correct the API situation since they were starting anew, but it didn't happen. All these API half measures are just dancing around the real problem of not having truly independent interpreter instances.
  - pekk 11y ago
    Python 3 wasn't really starting anew, they were also balancing the ongoing screaming about every difference from Python 2.
    - zabbadabba 11y ago
      It is curious how they completely revamped Python making most Python3 code incompatible with Python2 and yet they wanted to preserve C API "compatibility". Compatibility with what? Most Python3 modules had to be rewritten anyway - it was the perfect opportunity to fix the C API and get rid of the GIL once and for all.
      - ericsnow 11y ago
        you forgot your <hand-wavy gross exaggeration> tags
- frozenport 11y ago
  I don't understand. Can you give an example of a library that has a global resource that isn't thread safe? Also how is this different then problems faced when using multiprocessing style parallelism with the same library?
  - JoshTriplett 11y ago
    A few random examples, all from in-tree Python modules that are completely typical examples of what you'd find out-of-tree as well: The readline module uses libreadline, which maintains internal global state on the C side. The standard library has lists and dictionaries, which are not thread-safe to access from two threads simultaneously, and are not in any way locked.
    - exDM69 11y ago
      > The standard library has lists and dictionaries, which are not thread-safe to access from two threads simultaneously, and are not in any way locked. Locking lists and dicts, etc would not be a reasonable thing to do. There must be some granularity in synchronization, it's clearly not a good idea to attempt to allow mutating any object from multiple threads without explicit locking. Things like readline are issues that programmers have to deal with when writing multi threaded code in any language. There are libs which are not thread safe or which have global state or need special handling with threads (e.g. OpenGL). That's no reason to disallow threading completely. However, this might be a big culture shock issue if Python were to suddenly start supporting proper concurrency without GIL. None of the libraries document whether they are thread safe or not, so there would be surprises ahead.
      - Erwin 11y ago
        Locking the basic data structures seems like it would destroy Python's simplicity. How do you determine the scope of the per-list lock? Do you add a lock to every single data structure that you are required to grab before reading or modifying it? If you require the programmer to do it, you can easily get race conditions which will only show up once in a blue moon causing incorrect results or corruption at worst - something Python's GIL guaranteed not to happen (as long as you didn't do anything risky in your C extension). Is there any solution that doesn't use purely functional data structures?
comex 11y ago
That would help, but in the chunk of the thread I read, most of the debate was about what to do after that: specifically (a) avoiding the cost of starting up a new interpreter, especially module import work, and (b) possibly taking advantage of the shared address space to be able to share some immutable objects somehow, rather than copying.
srean 11y ago
That's more or less how Tcl does it. They have had this support since ever. One coordinated by passing messages to the different Tcl interpreters, each running in their own thread.
smegel 11y ago
> All Python has to do is put all interpreter state into a single struct to allow for completely independent interpreter instances. But that is not what threading means. If you want "completely independent interpreters" just fork. Threading means you share everything. > Then the GIL can be discarded altogether And then you start sharing variables between interpreters, and you gets lots of little baby GILs to contend with! How fun will that be? Not very.
- Frondo 11y ago
  That's how Tcl does it. Each thread gets its own Tcl interpreter, and you communicate among them via message passing. It lets you avoid the context switch that way.
  - frozenport 11y ago
    There is a difference between threads and processes. What you outline is how the python multiprocessing library works.
    - sitkack 11y ago
      Think of it like running multiple python processes inside the same memory space so that copying between interpreters is by reference, not value. Nothing has to be serialized.
      - deleted 11y ago
        [deleted]
      - onnoonno 11y ago
        How do you avoid damaging the consistency of Python objects then? I think the only way to do that would be to have a reentrant lock for each object individually. Isn't that potentially a big performance issue? I think that is the main reason behind this quote from the article: > Several attempts have been made over the years and failed to do it without sacrificing single-threaded performance.
mappu 11y ago
PHP did this (conditionally, at compile-time) with its ZTS builds, to allow multiple PHP runtimes in the same memory space. There is a not-insignificant performance loss from running this way, and so it's just not done, at least on Linux and similar scenarios where multiprocessing is feasible instead. Although PHP7 does have some workarounds to improve the performance of this mode, involving implementing thread-local storage on certain operating systems..
sitkack 11y ago
The Lua interpreter interface is so much cleaner, the number of globals in CPython is mind numbing. I looked into for a quick hack, not possible. At least 100+ hrs of work to put the globals inside a struct.
- trentnelson 11y ago
  For PyParallel I just altered the decl for global variables to be TLS static, then tweaked initialization, and voila, thread safe globals. E.g. https://bitbucket.org/tpn/pyparallel/src/3be2954508f9938b85ae1bbc5c41587ec26b764a/diffs/Objects/bytesobject.c.patch?at=3.3-px https://bitbucket.org/tpn/pyparallel/src/3be2954508f9938b85a...
  - zabbadabba 11y ago
    The problem is that thread local storage is slow. It's an extra layer of indirection. Also, thread safety is just one concern in the GIL debate. One should be able to support many low-overhead independent interpreter instances on a single thread rather than relying on an arcane interpreter context switching scheme as presently exists in Python 2 and 3.
    - trentnelson 11y ago
      > The problem is that thread local storage is slow. It's slower than accessing a struct, sure: 5982: if (ctx) { 000000001E1A808B 8B 0D FF 7C 28 00 mov ecx,dword ptr [_tls_index (1E42FD90h)] 000000001E1A8091 BA 80 29 00 00 mov edx,2980h 000000001E1A8096 48 89 83 90 02 00 00 mov qword ptr [rbx+290h],rax 000000001E1A809D 65 48 8B 04 25 58 00 00 00 mov rax,qword ptr gs:[58h] 000000001E1A80A6 48 8B 04 C8 mov rax,qword ptr [rax+rcx*8] 000000001E1A80AA 48 8B 0C 02 mov rcx,qword ptr [rdx+rax] 000000001E1A80AE 48 85 C9 test rcx,rcx 000000001E1A80B1 74 70 je new_context+253h (1E1A8123h) But think of the big picture: I use TLS everywhere for PyParallel, and PyParallel has awesome performance, so eh, net win. > One should be able to support many low-overhead independent interpreter instances on a single thread Why? What problem does that solve?

maxerickson 11y ago

In case it isn't obvious, this got lots of discussion on the mailing list it was posted to: https://mail.python.org/pipermail/python-ideas/2015-June/thread.html#34177 https://mail.python.org/pipermail/python-ideas/2015-June/thr...

mianos 11y ago

If this comes I'll move from 2.7 to a 3.x

smegel 11y ago

Wow, Python will become the new Perl, how cool is that?! Not very.

coldtea 11y ago
What does that even mean?! Not much.

BuckRogers 11y ago

What problem does this solve? It does not solve the multicore "problem". I'm not a huge fan of this proposal. A few reasons. This doesn't really add much that wasn't possible before, and doesn't really solve any technical or PR issue. The PR issue will never be resolved with CPython, those people who don't understand it are free to write multithreaded Java apps. But I think explicitly spinning up pthreads should be reserved for writing systems software. I'm assuming subinterpreters means they'll be allocated as pthreads because this is meant to "use all your cores". Looking forward, this proposal sounds great as long as we're on 4 or 8 cores max, at a certain point this solution starts to look like something like a gimmicky ideal created to fit the technology way back in 2015. The ultimate multicore and multinode solution at a non-systems level, if we really need every single language to solve that specific problem- would be Erlang's approach. It's yet more technical churn rather than innovation in a language (Python3), that was born of technical churn. To offer something constructive as well, I think an ideal solution would be to find a more implicit approach. Think gevent for pthreads or subinterpreters. That would be a lot more work to figure out, this proposal looks more like a hack. I don't think this type of "improvement" will draw people to Python3 either, I wish they'd stop throwing crap at the wall seeing what will stick. Constantly expanding the language, which is bad. The core dev team doesn't know what to do, but usually in that case it's better to do nothing. Thus Python3 is looking more and more like a playground or experimental branch as time goes on. I'm increasingly thinking my "Python3 migration" will be to Swift or Go.

pekk 11y ago
I couldn't detect any real proposal in anything you said except that Python programmers should switch to Swift, and your main thesis seems to be that Python 3 is bad because it changes.
- sfk 11y ago
  Even for HN standards, your comment is one of the most peculiar summaries I've read.
mamerhst 11y ago
I agree with this guy. I was not aware of the multicore "problem" So, basically problems that don't exist are made up and then complicated solutions proposed. Does seem to go well with the python3 philosophy overall.

crdoconnor 11y ago

>However, in CPython the GIL means that we don't have parallelism, except through multiprocessing which requires trade-offs. I can't speak for anybody else, but I personally haven't felt that inconvenienced by these trade offs at all. The major one is needing to pickle objects before sending them back and forth between processes. Since I prefer to keep the thread interfaces as tight as possible anyway, this only ended up being a major problem once - when I was trying to pickle a stacktrace before sending it to the parent process.

ant6n 11y ago
multiprocessing can be fickle. In ipython, ctrl-c during a computation using multiprocessing results in general unhappiness, and will probably be followed by a 'killall python'
- vosper 11y ago
  Oh, yeah - and this one! Much more so than the inconvenience I posted!
- crdoconnor 11y ago
  I've had this problem too, but after doing the (quite considerable) leg-work to fix it involving messing around with process groups and the ridiculous number of different types of signal, I'm more inclined to blame the horrible 30 year old UNIX APIs than python. I've had this signal handling problem in single process/single threaded code too, and in other languages. True, it would be nice if python layered on a nicer set of APIs to deal with the nastiness.
  - logicchains 11y ago
    >I'm more inclined to blame the horrible 30 year old UNIX APIs than python. Datapoint: I encountered the same thing on Windows, having to use the Powershell equivalent of `killall python` to deal with it.
  - easytiger 11y ago
    No, this is pretty easy in c. i think the python docs even apologise for it somewhere.

ngoldbaum 11y ago

The situation around multiple cores in Python (up to and including the link in the OP) is very nicely summarized in these notes by Nick Coghlan: http://python-notes.curiousefficiency.org/en/latest/python3/multicore_python.html http://python-notes.curiousefficiency.org/en/latest/python3/...

dtxcoolbits 11y ago
"...it’s very easy to envision a future where CPython is used for command line utilities (which are generally single threaded and often so short running that the PyPy JIT never gets a chance to warm up) and embedded systems, while PyPy takes over the execution of long running scripts and applications, letting them run substantially faster and span multiple cores without requiring any modifications to the Python code" IMHO this is the most sensible and logical approach and doesn't break any existing code. This also seems like what will likely happen anyways. Out of all the options PyPy-STM seems like the best approach because: "pypy-stm is fully compatible with a GIL-based PyPy; you can use it as a drop-in replacement and multithreaded programs will run on multiple cores." http://pypy.readthedocs.org/en/latest/stm.html http://pypy.readthedocs.org/en/latest/stm.html

trentnelson 11y ago

I attempted to describe the progress I've made with PyParallel already here: https://mail.python.org/pipermail/python-ideas/2015-June/034342.html https://mail.python.org/pipermail/python-ideas/2015-June/034... And here: https://mail.python.org/pipermail/python-ideas/2015-June/034260.html https://mail.python.org/pipermail/python-ideas/2015-June/034... I'm getting excellent scaling and performance across the board for the simple TEFB tests (https://bitbucket.org/tpn/pyparallel/src/8528b11ba51003a9821ceb75683ee96ed33db28a/examples/tefb/tefb.py?at=3.3-px https://bitbucket.org/tpn/pyparallel/src/8528b11ba51003a9821...), and have implemented something that really shows where PyParallel shines: an instantaneous wiki search REST API: https://bitbucket.org/tpn/pyparallel/src/8528b11ba51003a9821ceb75683ee96ed33db28a/examples/wiki/wiki.py?at=3.3-px https://bitbucket.org/tpn/pyparallel/src/8528b11ba51003a9821.... Quoting: I particularly like the wiki example as it leverages a lot of benefits afforded by PyParallel's approach to parallelism, concurrency and asynchronous I/O: - Load a digital search trie (datrie.Trie) that contains every Wikipedia title and the byte-offset within the wiki.xml where the title was found. (Once loaded the RSS of python.exe is about 11GB; the trie itself has about 16 million items in it.) - Load a numpy array of sorted 64-bit integer offsets. This allows us to do a searchsorted() (binary search) against a given offset in order to derive the next offset. - Once we have a way of getting two byte offsets, we can use ranged HTTP requests (and TransmitFile behind the scenes) to efficiently read random chunks of the file asynchronously. (Windows has a huge advantage here -- there's simply no way to achieve similar functionality on POSIX in a non-blocking fashion (sendfile can block, a disk read() can block, a memory reference into a mmap'd file that isn't in memory will page fault, which will block).)

scott_s 11y ago
Apparently, you should expect to hear from him soon: https://mail.python.org/pipermail/python-ideas/2015-June/034275.html https://mail.python.org/pipermail/python-ideas/2015-June/034... Antoine Pitrou said that Snow's proposal was similar to your work. What do you see as the similarities?
- trentnelson 11y ago

rurban 11y ago

This looks like a new Python 4, but why not. It's a big problem that needs to be resolved for once.

angry_octet 11y ago

It seems to me that there is alot of muddled explanation going on here, if not muddled thinking. The goal fine grained parallelism python (fgpp) must be to improve the speed of code for which fat parallelism (user conconcted error prone 'multi-threading' or message passing) doesn't gain much, and for which calling out to an impl in another language or specialised library has overhead or doesn't make sense. IMHO, the most important part of fgpp must be that it is easily reasoned about, both by compiler/runtime and programmers, so as to maximise scope for transformations and user-driven design decisions (as opposed to VOODOO). It would ideally avoid the costs of premature optimisation, or the often inefficient flattening required for vectorisation. So I think any propsal should be addressing these questions, with appropriate benchmarks too, before it can be considered seriously.

ericsnow 11y ago
Thanks for the feedback.

Scramblejams 11y ago

Here's what I want: Something with programmer productivity similar to Python, but type-inferred for predictably good execution speed, with more support for functional programming style, with green threads preemptively executing in parallel and immutability baked deeply into the whole thing. And then a track record of deployment so the ugly GC and type system edge cases are ironed out and libraries are plentiful. Elixir/Erlang will probably never be fast enough, Go didn't commit to immutability and its functional programming support is minimal, Rust could have all that but it's too low-level... I guess OCaml is as close as it comes for the moment -- it's got everything I listed except a reasonable parallel execution story, but it looks like that's being worked on at the moment.

beagle3 11y ago
I don't think there's anything quite like that, but Nim seems to be a better fit than OCaml.
- Scramblejams 11y ago
  Tried that. It's quite a big and complex language, there are lots of rough edges (I had it crashing from a double free within 5 minutes of trying it out), the bus factor is in the basement and although I didn't stick with it long enough to know for sure, there's much more mutability there than made me comfortable during my short sojourn.