4 ms·

I think the limitations of the GIL are not all in all a bad thing, but I do think that the Python community needs to seriously look at making Asynchronous execu

I think the limitations of the GIL are not all in all a bad thing, but I do think that the Python community needs to seriously look at making Asynchronous execution a priority in future versions. Asyncio is really complicated and provides very little in terms of performance. Multitasking is the most important issue that Python faces today (and maybe PyPy is the answer).

fijal 11y ago
The problem that python is facing (and pypy is not solving) is that in order to have a working async library ecosystem, everything, especially the stdlib has to be async aware. This is a far cry right now and I don't see any of the python devs really trying to address that. Otherwise you end up with the same situation as for twisted - it works and it's great, but every single library needs to be made twisted aware
vegabook 11y ago
So we're in 3.5 already, and one of its central features, async, is flawed, you say. So not only do we not have a killer feature, but even one of the nice-to-haves is a dog. Timeout. This project needs new leadership. The competition (Golang) is walking all over Python on the async issue.
- beagle3 11y ago
  I read your comments on this thread, and I frankly wonder where you are coming from. It seems you care a lot about Python, but have a crisis because your pet peeve (threading) is not being addressed, and as a result expect the team who has been guiding Python to the success that it is over the last 25 years to step down. (And it has been a huge success - it has none of the big corporate money backing that Java, C# and Go have - but mindshare and notability on the same scale, dominates significant niches like non-HPC scientific computation, and has very significant presence in almost every field). Could you tell me what (and how long) your experience is? My answers about your complaints are basically: 1. Solving threading is (relatively) easy, if you just give up backwards compatibility; Not the 2.x -> 3.x compatibility which is comparatively trivial - but in a major way, breaking every single extension library, and the vast majority of Python code (or slowing it down unbearably). You might be happy, but the rest of the Python users (basically, the reason you actually use Python) won't. 2. Assuming they agree with you about their failure (I dont, FWIW), someone has to step up and offer an alternative. Who is that, what are they doing these days, and why do you think they will succeed where Guido et al "failed"? 3. Perl at 28 years is not much older than Python at 24 years (relatively speaking), but has been sliding into obscurity for a long time now, whereas Python is flourishing. The Python 3 transition is actually happening as planned (IIRC, there was an expected 5 year period just for feature and speed parity!). Perl 6 is esoteric, Perl 5 is aging and dying. I think that for a living, popular language, the Python team is doing a commendable job, even if they do not address a specific issue (that many people care about, but would actually not make that much of a difference in practice if we are to learn from other languages)
  - Lofkin 11y ago
    The GIL issue has been solved without breaking backwards compat: http://pyparallel.org/ http://pyparallel.org/ It just need further dev work and acceptance into py3
    - acconsta 11y ago
      On Windows
    - beagle3 11y ago
      Did you actually look into it? The threads you run can't make any change to existing objects, and various other changes. It does break compatibility, and needs patched version of Numpy, ODBC (and I would guess, most other packages). Definitely not "solved without breaking backwards compat".
  - vegabook 11y ago
    I have been using Python for a decade, and intensively for 7 years. I am a domain expert (finance) but I do also have 4 years of CS so consider myself non-idiotic, if not haskell-genius, on programming languages. I adopted Python (before it was popular - it was behind ruby / PHP / perl at the time) because it was pragmatic above all. That last statement is being violated with 3.x, and that is the seed of my concern. Before answering your questions 1 thru 3, some context. It is my strong belief that the authorities are constantly looking at golang and JS as their competitors, in other words, the web world, whereas the real hardcore advantage of Python is in science and numerical computing. As evidence, witness numerous Python books which advise new users to hit the Continuum Analytics or Enthought sites for their full-stack Python installations, even texts which are not about numerics. On your questions: 1) I don't care about threading. I care about pushing as many compute bits through the Xeon as I can in a given amount of seconds (using Numpy). But as I am a data scientist, I need the REPL. C is out. Why can I still not do this easily? Multiprocessing is there, sure, but it's been unch for years, while Cuda, OpenCL etc are far too hard for guy like me whose intellectual bandwidth is occupied with the domain, not the CS. Isn't that what Python was supposed to be about? Getting stuff done? Why isn't Python vectoring my data through the CPU and GPU yet, 15 years after numeric was first introduced? 2) Continuum Analytics is doing an awesome job and I don't see why they, or Enthought, couldn't take up the mantle, 10gen/Datastax style to use a database analogy. They really know their customers, and the Continuum stack delivers real new value every 6 months, and not only for a scientific audience. More generally, real users in real domains should be driving the project. 3) I am less concerned about Python 3 happening as planned, than I am about the focus of the project. Type annotations? This is an intellectual indulgence if it does not increase performance. Asyncio? We've had async libraries for years! Even when I started Python we had async libraries (not as good but they were there). How is async something fantastic and new? It's nothing but polishing an existing capability a little bit further. Unicode. fine. But again, web focused. Nobody else cares. Xrange laziness. Okay. Leaves me cold. Print(). No CS benefit, but huge marketing loss as you can't go out to newbies anymore and say "hey, check this out... Python hello world?" >>> print "hello world" "hello world" It just doesn't get any simpler, and yet Python wants to throw out that unique hook for new people who care little about coding but a lot about domain. It's zero, genuinely zero, boilerplate, whereas there are a dozen languages where you can do print("hello world"). Seems trivial, but in print vs print() we have the key difference in philosophy (get-stuff-done-now vs take-me-oh-so-seriously). If you're so serious at a computer science level, you're not going to do Python. So. What is Python. A serious language? NO. A wonderfully malleable, not too serious, friendly language, into which you can insert some real hardcore stuff (Numpy, ML, website parsing, database transformations, game scripting, image processing....the list is endless) really easily? Yes. Where is Python genuinely way ahead of everyone else? Only on numerical computing. It strikes me that Guido and co are embarrassed by their weekend hack of 1.x and 2.x, when that is precisely what the user base loves about it. Their attempt to make Python serious, is killing Python's original spirit. There is nothing wrong with 2.7. Nobody wants Python to morph into Java. So, am I a Luddite wanting 2.7 to live forever? no. What I want is vectorization plus DAG-like workflows. These are the most important pieces of computer science that actually dovetail with real world use cases, today. Yes async is cool, but golang now owns that space. What I'd really like is for Python to give us a good framework for the Big Data world which is in almost everybody's use case now, and that means, Python needs to talk multiprocessor, Python needs to talk GPU, Python needs to talk cluster, and Python should long ago have been addressing this directly. Python needs to "go vectorized". That should be the project's obsession. SIMD, in a word, where the parallel granularity can go from GPU kernels,to CPU, to multi-machine clusters, and DAG-like workflows (with possible recursion) built in. This is not a nice-to-have capability anymore. It is what the next wildly popular mainstream programming language will have, built into the language. The signals from everything we're seeing added to 3.x is NOT this. It's web-like stuff. That battle is over. JS won. Let's not gift the opportunity of huge data to Java (Spark) and Cuda, or a new language that will see the future better than us. It should be Python! There you go. My view.

fijal 11y ago

The problem that python is facing (and pypy is not solving) is that in order to have a working async library ecosystem, everything, especially the stdlib has to be async aware. This is a far cry right now and I don't see any of the python devs really trying to address that. Otherwise you end up with the same situation as for twisted - it works and it's great, but every single library needs to be made twisted aware

vegabook 11y ago

So we're in 3.5 already, and one of its central features, async, is flawed, you say. So not only do we not have a killer feature, but even one of the nice-to-haves is a dog. Timeout. This project needs new leadership. The competition (Golang) is walking all over Python on the async issue.

beagle3 11y ago
I read your comments on this thread, and I frankly wonder where you are coming from. It seems you care a lot about Python, but have a crisis because your pet peeve (threading) is not being addressed, and as a result expect the team who has been guiding Python to the success that it is over the last 25 years to step down. (And it has been a huge success - it has none of the big corporate money backing that Java, C# and Go have - but mindshare and notability on the same scale, dominates significant niches like non-HPC scientific computation, and has very significant presence in almost every field). Could you tell me what (and how long) your experience is? My answers about your complaints are basically: 1. Solving threading is (relatively) easy, if you just give up backwards compatibility; Not the 2.x -> 3.x compatibility which is comparatively trivial - but in a major way, breaking every single extension library, and the vast majority of Python code (or slowing it down unbearably). You might be happy, but the rest of the Python users (basically, the reason you actually use Python) won't. 2. Assuming they agree with you about their failure (I dont, FWIW), someone has to step up and offer an alternative. Who is that, what are they doing these days, and why do you think they will succeed where Guido et al "failed"? 3. Perl at 28 years is not much older than Python at 24 years (relatively speaking), but has been sliding into obscurity for a long time now, whereas Python is flourishing. The Python 3 transition is actually happening as planned (IIRC, there was an expected 5 year period just for feature and speed parity!). Perl 6 is esoteric, Perl 5 is aging and dying. I think that for a living, popular language, the Python team is doing a commendable job, even if they do not address a specific issue (that many people care about, but would actually not make that much of a difference in practice if we are to learn from other languages)
- Lofkin 11y ago
  The GIL issue has been solved without breaking backwards compat: http://pyparallel.org/ http://pyparallel.org/ It just need further dev work and acceptance into py3
  - acconsta 11y ago
    On Windows
  - beagle3 11y ago
    Did you actually look into it? The threads you run can't make any change to existing objects, and various other changes. It does break compatibility, and needs patched version of Numpy, ODBC (and I would guess, most other packages). Definitely not "solved without breaking backwards compat".
- vegabook 11y ago
  I have been using Python for a decade, and intensively for 7 years. I am a domain expert (finance) but I do also have 4 years of CS so consider myself non-idiotic, if not haskell-genius, on programming languages. I adopted Python (before it was popular - it was behind ruby / PHP / perl at the time) because it was pragmatic above all. That last statement is being violated with 3.x, and that is the seed of my concern. Before answering your questions 1 thru 3, some context. It is my strong belief that the authorities are constantly looking at golang and JS as their competitors, in other words, the web world, whereas the real hardcore advantage of Python is in science and numerical computing. As evidence, witness numerous Python books which advise new users to hit the Continuum Analytics or Enthought sites for their full-stack Python installations, even texts which are not about numerics. On your questions: 1) I don't care about threading. I care about pushing as many compute bits through the Xeon as I can in a given amount of seconds (using Numpy). But as I am a data scientist, I need the REPL. C is out. Why can I still not do this easily? Multiprocessing is there, sure, but it's been unch for years, while Cuda, OpenCL etc are far too hard for guy like me whose intellectual bandwidth is occupied with the domain, not the CS. Isn't that what Python was supposed to be about? Getting stuff done? Why isn't Python vectoring my data through the CPU and GPU yet, 15 years after numeric was first introduced? 2) Continuum Analytics is doing an awesome job and I don't see why they, or Enthought, couldn't take up the mantle, 10gen/Datastax style to use a database analogy. They really know their customers, and the Continuum stack delivers real new value every 6 months, and not only for a scientific audience. More generally, real users in real domains should be driving the project. 3) I am less concerned about Python 3 happening as planned, than I am about the focus of the project. Type annotations? This is an intellectual indulgence if it does not increase performance. Asyncio? We've had async libraries for years! Even when I started Python we had async libraries (not as good but they were there). How is async something fantastic and new? It's nothing but polishing an existing capability a little bit further. Unicode. fine. But again, web focused. Nobody else cares. Xrange laziness. Okay. Leaves me cold. Print(). No CS benefit, but huge marketing loss as you can't go out to newbies anymore and say "hey, check this out... Python hello world?" >>> print "hello world" "hello world" It just doesn't get any simpler, and yet Python wants to throw out that unique hook for new people who care little about coding but a lot about domain. It's zero, genuinely zero, boilerplate, whereas there are a dozen languages where you can do print("hello world"). Seems trivial, but in print vs print() we have the key difference in philosophy (get-stuff-done-now vs take-me-oh-so-seriously). If you're so serious at a computer science level, you're not going to do Python. So. What is Python. A serious language? NO. A wonderfully malleable, not too serious, friendly language, into which you can insert some real hardcore stuff (Numpy, ML, website parsing, database transformations, game scripting, image processing....the list is endless) really easily? Yes. Where is Python genuinely way ahead of everyone else? Only on numerical computing. It strikes me that Guido and co are embarrassed by their weekend hack of 1.x and 2.x, when that is precisely what the user base loves about it. Their attempt to make Python serious, is killing Python's original spirit. There is nothing wrong with 2.7. Nobody wants Python to morph into Java. So, am I a Luddite wanting 2.7 to live forever? no. What I want is vectorization plus DAG-like workflows. These are the most important pieces of computer science that actually dovetail with real world use cases, today. Yes async is cool, but golang now owns that space. What I'd really like is for Python to give us a good framework for the Big Data world which is in almost everybody's use case now, and that means, Python needs to talk multiprocessor, Python needs to talk GPU, Python needs to talk cluster, and Python should long ago have been addressing this directly. Python needs to "go vectorized". That should be the project's obsession. SIMD, in a word, where the parallel granularity can go from GPU kernels,to CPU, to multi-machine clusters, and DAG-like workflows (with possible recursion) built in. This is not a nice-to-have capability anymore. It is what the next wildly popular mainstream programming language will have, built into the language. The signals from everything we're seeing added to 3.x is NOT this. It's web-like stuff. That battle is over. JS won. Let's not gift the opportunity of huge data to Java (Spark) and Cuda, or a new language that will see the future better than us. It should be Python! There you go. My view.