4 ms·

What problem does this solve? It does not solve the multicore "problem". I'm not a huge fan of this proposal. A few reasons. This doesn't really add much that

What problem does this solve? It does not solve the multicore "problem". I'm not a huge fan of this proposal. A few reasons. This doesn't really add much that wasn't possible before, and doesn't really solve any technical or PR issue. The PR issue will never be resolved with CPython, those people who don't understand it are free to write multithreaded Java apps. But I think explicitly spinning up pthreads should be reserved for writing systems software. I'm assuming subinterpreters means they'll be allocated as pthreads because this is meant to "use all your cores". Looking forward, this proposal sounds great as long as we're on 4 or 8 cores max, at a certain point this solution starts to look like something like a gimmicky ideal created to fit the technology way back in 2015. The ultimate multicore and multinode solution at a non-systems level, if we really need every single language to solve that specific problem- would be Erlang's approach. It's yet more technical churn rather than innovation in a language (Python3), that was born of technical churn. To offer something constructive as well, I think an ideal solution would be to find a more implicit approach. Think gevent for pthreads or subinterpreters. That would be a lot more work to figure out, this proposal looks more like a hack. I don't think this type of "improvement" will draw people to Python3 either, I wish they'd stop throwing crap at the wall seeing what will stick. Constantly expanding the language, which is bad. The core dev team doesn't know what to do, but usually in that case it's better to do nothing. Thus Python3 is looking more and more like a playground or experimental branch as time goes on. I'm increasingly thinking my "Python3 migration" will be to Swift or Go.

pekk 11y ago
I couldn't detect any real proposal in anything you said except that Python programmers should switch to Swift, and your main thesis seems to be that Python 3 is bad because it changes.
- sfk 11y ago
  Even for HN standards, your comment is one of the most peculiar summaries I've read.
mamerhst 11y ago
I agree with this guy. I was not aware of the multicore "problem" So, basically problems that don't exist are made up and then complicated solutions proposed. Does seem to go well with the python3 philosophy overall.
deleted 11y ago
[deleted]
kyllo 11y ago
I agree and am not looking forward to debugging race conditions or implementing locks around lists and dicts in python code. The language is chock-full of mutable state and sharing that mutable state between threads is a recipe for disaster.
- ericsnow 11y ago
  That's the point of the subinterpreter approach. It's only a viable approach if we can achieve the data isolation of multiprocessing with the efficiency of threads.
ericsnow 11y ago
The problem is that in CPython the only mechanism to leverage multiple cores for CPU-bound workloads is the multiprocessing module. That module suffers from the cost of serializing all objects that are transferred between processes over IPC. Threading in CPython mostly doesn't utilize multiple cores for CPU-bound workloads due to the GIL. The goal of the project is to achieve good multi-core support without the serialization overhead and to make that support both obvious and undeniable. While very few Python programs actually benefit from true parallelism, it's a glaring gap that I'd like to see filled. My proposal is a means to an end. I'd be just as happy if the situation were resolved in some other way and without my involvement. However, I've found that in open source, waiting for someone else to do what you want done is a losing proposition. So I'm not going to hold my breath. The only project of which I'm aware that could make a difference is Trent Nelson's pyparallel. I hope to collaborate on that, but I'll likely continue pursuing alternatives at the same time for now. I'm certainly open to any serious recommendations on how to achieve the goal, if you're sincerely interested in making a difference. (I appreciate the the mention of gevent and Erlang, which are things I've already taken into consideration.) As to the details of my proposal, it's very early in the project and the python-ideas post is simply a high-level exploratory discussion of the problem along with a lot of unsettled details about how I think it might be solved in the Python 3.6 timeframe. A more serious proposal would be in the form of a PEP. Regarding your feedback, the post implies that you either misunderstood what I said or you don't understand the underlying technologies. To clarify: * the proposal is changing/adding relatively little, instead focusing on leverage as much existing features as possible * Python's existing threading support would be leveraged * subinterpreters, which already exist, would be exposed in Python through a new module in the stdlib * subinterpreters are already highly independent and share very little global state * the key change is enabling subinterpreters to run more or less without the GIL (leaving that to the main interpreter) * the key addition is a mechanism to efficiently and safely share objects between subinterpreters * the approach is drawing inspiration in part from CSP (Hoare's Communicating Sequential Processes) * think of it as shared-nothing threads with message passing It will most certainly improve multi-core support. It shares more in common with Erlang's approach than you think. It is neither a hack nor crap on the wall, as you put it.
ericsnow 11y ago
Also, the way you have described the CPython core developers and their efforts to improve the language is a bit disingenuous. I apologize if you've had some experience that left you with negative feelings. Keep in mind that like just about everyone working on Python (and most open source), I'm doing this work in my spare time. I take my role very seriously and work on what I believe will benefit the community the most (relative to what I can accomplish). This is also the case for most of the other committers. Furthermore, since we all do this in our free time, the pace of Python development is super slow when compared with subsidized projects (or languages). Wanting to make a difference in a reasonable timeframe, core developers take that into consideration when deciding about working on a large Python feature. Consider both these points before rushing to judgement about what core developers are working on.

pekk 11y ago

I couldn't detect any real proposal in anything you said except that Python programmers should switch to Swift, and your main thesis seems to be that Python 3 is bad because it changes.

sfk 11y ago
Even for HN standards, your comment is one of the most peculiar summaries I've read.

mamerhst 11y ago

I agree with this guy. I was not aware of the multicore "problem" So, basically problems that don't exist are made up and then complicated solutions proposed. Does seem to go well with the python3 philosophy overall.

deleted 11y ago

[deleted]

kyllo 11y ago

I agree and am not looking forward to debugging race conditions or implementing locks around lists and dicts in python code. The language is chock-full of mutable state and sharing that mutable state between threads is a recipe for disaster.

ericsnow 11y ago
That's the point of the subinterpreter approach. It's only a viable approach if we can achieve the data isolation of multiprocessing with the efficiency of threads.

ericsnow 11y ago

The problem is that in CPython the only mechanism to leverage multiple cores for CPU-bound workloads is the multiprocessing module. That module suffers from the cost of serializing all objects that are transferred between processes over IPC. Threading in CPython mostly doesn't utilize multiple cores for CPU-bound workloads due to the GIL. The goal of the project is to achieve good multi-core support without the serialization overhead and to make that support both obvious and undeniable. While very few Python programs actually benefit from true parallelism, it's a glaring gap that I'd like to see filled. My proposal is a means to an end. I'd be just as happy if the situation were resolved in some other way and without my involvement. However, I've found that in open source, waiting for someone else to do what you want done is a losing proposition. So I'm not going to hold my breath. The only project of which I'm aware that could make a difference is Trent Nelson's pyparallel. I hope to collaborate on that, but I'll likely continue pursuing alternatives at the same time for now. I'm certainly open to any serious recommendations on how to achieve the goal, if you're sincerely interested in making a difference. (I appreciate the the mention of gevent and Erlang, which are things I've already taken into consideration.) As to the details of my proposal, it's very early in the project and the python-ideas post is simply a high-level exploratory discussion of the problem along with a lot of unsettled details about how I think it might be solved in the Python 3.6 timeframe. A more serious proposal would be in the form of a PEP. Regarding your feedback, the post implies that you either misunderstood what I said or you don't understand the underlying technologies. To clarify: * the proposal is changing/adding relatively little, instead focusing on leverage as much existing features as possible * Python's existing threading support would be leveraged * subinterpreters, which already exist, would be exposed in Python through a new module in the stdlib * subinterpreters are already highly independent and share very little global state * the key change is enabling subinterpreters to run more or less without the GIL (leaving that to the main interpreter) * the key addition is a mechanism to efficiently and safely share objects between subinterpreters * the approach is drawing inspiration in part from CSP (Hoare's Communicating Sequential Processes) * think of it as shared-nothing threads with message passing It will most certainly improve multi-core support. It shares more in common with Erlang's approach than you think. It is neither a hack nor crap on the wall, as you put it.

ericsnow 11y ago

Also, the way you have described the CPython core developers and their efforts to improve the language is a bit disingenuous. I apologize if you've had some experience that left you with negative feelings. Keep in mind that like just about everyone working on Python (and most open source), I'm doing this work in my spare time. I take my role very seriously and work on what I believe will benefit the community the most (relative to what I can accomplish). This is also the case for most of the other committers. Furthermore, since we all do this in our free time, the pace of Python development is super slow when compared with subsidized projects (or languages). Wanting to make a difference in a reasonable timeframe, core developers take that into consideration when deciding about working on a large Python feature. Consider both these points before rushing to judgement about what core developers are working on.