7 ms·

In other words, a lost opportunity. Asyncio and this new syntax is both hard for beginners and experienced python coders alike, and still doesn't do multicore!.

In other words, a lost opportunity. Asyncio and this new syntax is both hard for beginners and experienced python coders alike, and still doesn't do multicore!. We're clocking up 3.x version numbers as if this will magically provide the illusion of progress, but the killer feature is still not there. Instead we get type annotations. In a dynamic language. Which doesn't compile and therefore doesn't need them. With no performance advantage. If I have to type declare everything, I want 10x performance. Okay? If Python were a listed company, the CEO would have been replaced long ago. I'm tired of watching my favourite language flail around like this. Will Continuum Analytics or Enthought please fork 2.7?

sonicrocketman 11y ago
I think the limitations of the GIL are not all in all a bad thing, but I do think that the Python community needs to seriously look at making Asynchronous execution a priority in future versions. Asyncio is really complicated and provides very little in terms of performance. Multitasking is the most important issue that Python faces today (and maybe PyPy is the answer).
- fijal 11y ago
  The problem that python is facing (and pypy is not solving) is that in order to have a working async library ecosystem, everything, especially the stdlib has to be async aware. This is a far cry right now and I don't see any of the python devs really trying to address that. Otherwise you end up with the same situation as for twisted - it works and it's great, but every single library needs to be made twisted aware
- vegabook 11y ago
  So we're in 3.5 already, and one of its central features, async, is flawed, you say. So not only do we not have a killer feature, but even one of the nice-to-haves is a dog. Timeout. This project needs new leadership. The competition (Golang) is walking all over Python on the async issue.
  - beagle3 11y ago
    I read your comments on this thread, and I frankly wonder where you are coming from. It seems you care a lot about Python, but have a crisis because your pet peeve (threading) is not being addressed, and as a result expect the team who has been guiding Python to the success that it is over the last 25 years to step down. (And it has been a huge success - it has none of the big corporate money backing that Java, C# and Go have - but mindshare and notability on the same scale, dominates significant niches like non-HPC scientific computation, and has very significant presence in almost every field). Could you tell me what (and how long) your experience is? My answers about your complaints are basically: 1. Solving threading is (relatively) easy, if you just give up backwards compatibility; Not the 2.x -> 3.x compatibility which is comparatively trivial - but in a major way, breaking every single extension library, and the vast majority of Python code (or slowing it down unbearably). You might be happy, but the rest of the Python users (basically, the reason you actually use Python) won't. 2. Assuming they agree with you about their failure (I dont, FWIW), someone has to step up and offer an alternative. Who is that, what are they doing these days, and why do you think they will succeed where Guido et al "failed"? 3. Perl at 28 years is not much older than Python at 24 years (relatively speaking), but has been sliding into obscurity for a long time now, whereas Python is flourishing. The Python 3 transition is actually happening as planned (IIRC, there was an expected 5 year period just for feature and speed parity!). Perl 6 is esoteric, Perl 5 is aging and dying. I think that for a living, popular language, the Python team is doing a commendable job, even if they do not address a specific issue (that many people care about, but would actually not make that much of a difference in practice if we are to learn from other languages)
    - Lofkin 11y ago
      The GIL issue has been solved without breaking backwards compat: http://pyparallel.org/ http://pyparallel.org/ It just need further dev work and acceptance into py3
      - acconsta 11y ago
        On Windows
pdonis 11y ago
> Asyncio and this new syntax is both hard for beginners and experienced python coders alike, and still doesn't do multicore!. The fundamental limitation here isn't unique to Python. Multiple tasks that are CPU bound simply cannot be managed using cooperative multitasking, and that's all asyncio is: cooperative multitasking, with a friendlier syntax than the old asyncore module. Cooperative multitasking using async I/O works great when you have lots of tasks running that are I/O bound--they spend almost all of their time waiting for network or disk reads/writes. But as soon as you start piling up tasks that require CPU, you need either multiple threads or multiple processes. Python asyncio was never intended to handle that use case. Multiple threads is where Python has a unique limitation: the GIL prevents multiple threads from running on multiple cores, so if you want multiple CPU bound tasks to use multiple cores, you need to fork each one into a separate process. It also doesn't help that the multiprocessing module is, IMO, a huge ball of cruft that's overkill for a lot of multitasking use cases. But there are simpler ways to do it, at least on Unix systems, because the fork system call is very lightweight. Shameless plug: I wrote a library for this some time ago, which I use whenever I have a Python project that needs to fork worker processes; it's the "comm" sub-package in my plib-io Python distribution (this is the Python 3 version): https://bitbucket.org/pdonis/plib3-io/src https://bitbucket.org/pdonis/plib3-io/src
- vegabook 11y ago
  I know it's not easy to graft multicore onto Python. So why pretend? If you're serious about web programming you're not using Python anyway. You're using JS or Golang. And if you insist on python, we have plenty of options to do async i/o already. Optimising async is like polishing the doorknobs on the titanic. Waste of effort. What we need, is a concentrated focus on the only two areas where Python beats the competition hands down. Numerical computing, and newby accessibility. The first gets a token "@" operator (great, thanks), the second is moving backwards with 3.x. Asyncio, type annotations, and unicode, are answers to questions nobody is asking (except by those who refuse to use the right tool for the right job).
  - deleted 11y ago
    [deleted]
jaimebuelta 11y ago
I fail to see how asyncio should use multicore. Asynchronous programming uses a single thread... If you use CPU-bound tasks, using non-blocking I/O (which is what asyncio is about) is not going to work. You can use multiple processes (maybe from concurrent.futures[1]) to help that case. [1] https://docs.python.org/3.5/library/concurrent.futures.html#module-concurrent.futures https://docs.python.org/3.5/library/concurrent.futures.html#...
armitron 11y ago
Amen brother. If you make one mistake in your thinking, it's treating Python as more than a christmas hack (which it started out as). It's not a serious language, it has no design behind it and is decades behind the state of the art. Use it for throw-away scripts and such is what I say, it's an area where it shines. The moment you want to _architect_ something substantial, reach for something else cause chewing gum and duct tape is not going to do it.
- vegabook 11y ago
  Amen buddy. I hack in Python. I discover in Python. I script, and I pre-production in Python. Then I do C or Java or even JS. But none of my pro-code would have been possible without all the learning I accumulated about the problem in good ol' 2.7 interactive and its arsenal of tools. [edit] Actually I think you're right. I think it's time to graduate to something serious. My mistake is to think that Python can be that serious language. 3.x, by trying to be that, is horribly compromising the basics of what Python is all about, namely accessibility, friendliness, discovery, malleability. Not production.
- zeeg 11y ago
  We are pretty serious about the web, infrastructure, and platforms in general, and we use Python. As does Dropbox, Google, Facebook, Twitter, etc. I don't think concurrency is great in Python, but you're living in the wrong world if you dont think people are using the language for real things.
  - armitron 11y ago
    Not sure what you mean by pretty serious about the web, infrastructure and platforms. To me this reads as an oxymoron because Python is not a pretty serious language. It's a toy at best. There is not a single feature that Python introduced or pioneered in the computing arena. It's a mismash of bad ideas from other similarly bad languages, wrapped in user-friendly paper. In that regard, it's very similar to Perl, but a lot more dangerous. Unlike Perl, there is nothing immediate in Python to warn you of the pit you're digging for yourself, especially if you're a newcomer to programming (as a substantial chunk of the Python community is). The troubles begin when you make the colossal mistake of trying to use Python for more than it can do, try to design and architect actual systems with it but I digress. On the other point you made, Google, Facebook, Twitter etc are using C++ too. Does that make C++ a good language for architecting substantial systems? Show me (1) system of substantial size implemented in C++ that hasn't turned out to be a DISASTER in anything _but_ performance. When your browser can be taken over a million ways to Sunday _just because_ it's implemented in C++, do you think Google, Facebook, Twitter etc give a damn if _you_ don't? Same argument for PHP and Facebook. Would you say that these companies are "pretty serious" about web, infrastructure and platforms? There are good languages out there, suited to architecting systems that are maintainable, secure and evolve gracefully. You won't find them simply by looking at what Google, Facebook and Twitter are using. You will need to try harder.
Lofkin 11y ago
They pretty much already did, but with 3.x: http://pyparallel.org/ http://pyparallel.org/

sonicrocketman 11y ago

I think the limitations of the GIL are not all in all a bad thing, but I do think that the Python community needs to seriously look at making Asynchronous execution a priority in future versions. Asyncio is really complicated and provides very little in terms of performance. Multitasking is the most important issue that Python faces today (and maybe PyPy is the answer).

fijal 11y ago
The problem that python is facing (and pypy is not solving) is that in order to have a working async library ecosystem, everything, especially the stdlib has to be async aware. This is a far cry right now and I don't see any of the python devs really trying to address that. Otherwise you end up with the same situation as for twisted - it works and it's great, but every single library needs to be made twisted aware
vegabook 11y ago
So we're in 3.5 already, and one of its central features, async, is flawed, you say. So not only do we not have a killer feature, but even one of the nice-to-haves is a dog. Timeout. This project needs new leadership. The competition (Golang) is walking all over Python on the async issue.
- beagle3 11y ago
  I read your comments on this thread, and I frankly wonder where you are coming from. It seems you care a lot about Python, but have a crisis because your pet peeve (threading) is not being addressed, and as a result expect the team who has been guiding Python to the success that it is over the last 25 years to step down. (And it has been a huge success - it has none of the big corporate money backing that Java, C# and Go have - but mindshare and notability on the same scale, dominates significant niches like non-HPC scientific computation, and has very significant presence in almost every field). Could you tell me what (and how long) your experience is? My answers about your complaints are basically: 1. Solving threading is (relatively) easy, if you just give up backwards compatibility; Not the 2.x -> 3.x compatibility which is comparatively trivial - but in a major way, breaking every single extension library, and the vast majority of Python code (or slowing it down unbearably). You might be happy, but the rest of the Python users (basically, the reason you actually use Python) won't. 2. Assuming they agree with you about their failure (I dont, FWIW), someone has to step up and offer an alternative. Who is that, what are they doing these days, and why do you think they will succeed where Guido et al "failed"? 3. Perl at 28 years is not much older than Python at 24 years (relatively speaking), but has been sliding into obscurity for a long time now, whereas Python is flourishing. The Python 3 transition is actually happening as planned (IIRC, there was an expected 5 year period just for feature and speed parity!). Perl 6 is esoteric, Perl 5 is aging and dying. I think that for a living, popular language, the Python team is doing a commendable job, even if they do not address a specific issue (that many people care about, but would actually not make that much of a difference in practice if we are to learn from other languages)
  - Lofkin 11y ago
    The GIL issue has been solved without breaking backwards compat: http://pyparallel.org/ http://pyparallel.org/ It just need further dev work and acceptance into py3
    - acconsta 11y ago
      On Windows

pdonis 11y ago

> Asyncio and this new syntax is both hard for beginners and experienced python coders alike, and still doesn't do multicore!. The fundamental limitation here isn't unique to Python. Multiple tasks that are CPU bound simply cannot be managed using cooperative multitasking, and that's all asyncio is: cooperative multitasking, with a friendlier syntax than the old asyncore module. Cooperative multitasking using async I/O works great when you have lots of tasks running that are I/O bound--they spend almost all of their time waiting for network or disk reads/writes. But as soon as you start piling up tasks that require CPU, you need either multiple threads or multiple processes. Python asyncio was never intended to handle that use case. Multiple threads is where Python has a unique limitation: the GIL prevents multiple threads from running on multiple cores, so if you want multiple CPU bound tasks to use multiple cores, you need to fork each one into a separate process. It also doesn't help that the multiprocessing module is, IMO, a huge ball of cruft that's overkill for a lot of multitasking use cases. But there are simpler ways to do it, at least on Unix systems, because the fork system call is very lightweight. Shameless plug: I wrote a library for this some time ago, which I use whenever I have a Python project that needs to fork worker processes; it's the "comm" sub-package in my plib-io Python distribution (this is the Python 3 version): https://bitbucket.org/pdonis/plib3-io/src https://bitbucket.org/pdonis/plib3-io/src

vegabook 11y ago
I know it's not easy to graft multicore onto Python. So why pretend? If you're serious about web programming you're not using Python anyway. You're using JS or Golang. And if you insist on python, we have plenty of options to do async i/o already. Optimising async is like polishing the doorknobs on the titanic. Waste of effort. What we need, is a concentrated focus on the only two areas where Python beats the competition hands down. Numerical computing, and newby accessibility. The first gets a token "@" operator (great, thanks), the second is moving backwards with 3.x. Asyncio, type annotations, and unicode, are answers to questions nobody is asking (except by those who refuse to use the right tool for the right job).
- deleted 11y ago
  [deleted]

jaimebuelta 11y ago

I fail to see how asyncio should use multicore. Asynchronous programming uses a single thread... If you use CPU-bound tasks, using non-blocking I/O (which is what asyncio is about) is not going to work. You can use multiple processes (maybe from concurrent.futures[1]) to help that case. [1] https://docs.python.org/3.5/library/concurrent.futures.html#module-concurrent.futures https://docs.python.org/3.5/library/concurrent.futures.html#...

armitron 11y ago

Amen brother. If you make one mistake in your thinking, it's treating Python as more than a christmas hack (which it started out as). It's not a serious language, it has no design behind it and is decades behind the state of the art. Use it for throw-away scripts and such is what I say, it's an area where it shines. The moment you want to _architect_ something substantial, reach for something else cause chewing gum and duct tape is not going to do it.

vegabook 11y ago
Amen buddy. I hack in Python. I discover in Python. I script, and I pre-production in Python. Then I do C or Java or even JS. But none of my pro-code would have been possible without all the learning I accumulated about the problem in good ol' 2.7 interactive and its arsenal of tools. [edit] Actually I think you're right. I think it's time to graduate to something serious. My mistake is to think that Python can be that serious language. 3.x, by trying to be that, is horribly compromising the basics of what Python is all about, namely accessibility, friendliness, discovery, malleability. Not production.
zeeg 11y ago
We are pretty serious about the web, infrastructure, and platforms in general, and we use Python. As does Dropbox, Google, Facebook, Twitter, etc. I don't think concurrency is great in Python, but you're living in the wrong world if you dont think people are using the language for real things.
- armitron 11y ago
  Not sure what you mean by pretty serious about the web, infrastructure and platforms. To me this reads as an oxymoron because Python is not a pretty serious language. It's a toy at best. There is not a single feature that Python introduced or pioneered in the computing arena. It's a mismash of bad ideas from other similarly bad languages, wrapped in user-friendly paper. In that regard, it's very similar to Perl, but a lot more dangerous. Unlike Perl, there is nothing immediate in Python to warn you of the pit you're digging for yourself, especially if you're a newcomer to programming (as a substantial chunk of the Python community is). The troubles begin when you make the colossal mistake of trying to use Python for more than it can do, try to design and architect actual systems with it but I digress. On the other point you made, Google, Facebook, Twitter etc are using C++ too. Does that make C++ a good language for architecting substantial systems? Show me (1) system of substantial size implemented in C++ that hasn't turned out to be a DISASTER in anything _but_ performance. When your browser can be taken over a million ways to Sunday _just because_ it's implemented in C++, do you think Google, Facebook, Twitter etc give a damn if _you_ don't? Same argument for PHP and Facebook. Would you say that these companies are "pretty serious" about web, infrastructure and platforms? There are good languages out there, suited to architecting systems that are maintainable, secure and evolve gracefully. You won't find them simply by looking at what Google, Facebook and Twitter are using. You will need to try harder.

Lofkin 11y ago

They pretty much already did, but with 3.x: http://pyparallel.org/ http://pyparallel.org/