10 ms·
Python 3.5 and Multitasking
- gcb0 11y agoso is it a little syntax sugar on top of the multiprocess module to please ios developers? there's nothing new that i can see under the hood
- keypusher 11y agono, asyncio is a completely different module and paradigm from multiprocessing. the new standard syntax uses asyncio, and none of it has anything to do with ios
- sonicrocketman 11y agoI mentioned iOS because Apple's Grand Central Dispatch API is something I wish Python had. Python's multitasking is so complicated while Apple's GCD is so elegant. It's sad really.
- Lofkin 11y agoHave you seen this: http://dask.pydata.org/en/latest/ http://dask.pydata.org/en/latest/
- Lofkin 11y agoYour example is trivially parallelized with some decorators from the dask library. Example here: http://dask.pydata.org/en/latest/imperative.html http://dask.pydata.org/en/latest/imperative.html
- sonicrocketman 11y agoI hadn't seen Dask before this. Thanks!
- Lofkin 11y agoSure! If you like it, can you please blog about it (and post on hn)? Needs exposure outside the python data community.
- fijal 11y agoMaybe it's worth noting, maybe not, but this extremely trivial example (e.g. serial.py) gets executed 20x faster just by using pypy which speeds up serial execution. This is more than you would get by any sort of multiprocessing/threading shengenians just by using an optimizing VM (and granted, this example is very simple, but maybe addressing basic performance problems should come first)
- sonicrocketman 11y agoI really wanted to take a look to how PyPy handles this. Now I'm even more so. Thanks!
- keypusher 11y agoif you are blocking on io (such as read/write to disk or over the network), pypy execution speed will not help while threading will.
- fijal 11y agoobviously, but the time of blocking on IO should be the same on all strategies + CPU overhead of doing that
- sandGorgon 11y agopypy has not been updated in a while (it's compatibility is with Python 3.25). what's even more worrying is that it had almost run out of funds in July. Currently, it has 5000$ in the fund earmarked for Py3 support (http://pypy.org/py3donate.html http://pypy.org/py3donate.html) somehow I thought that Google was supporting Pypy.
- fijal 11y agoPyPy is being update very regularly with 3 releases a year on average. Most of the work however focuses on things our users really want - stability, performance, C extension support, warmup speed, memory consumption - you know the mundane stuff. If someone is willing to put effort into supporting more python 3, that's great! we would welcome the contributions. PS. We're closing onto 3.3 release soon PPS. Google donated a bit of money to PyPy when Guido was there, definitely "google supporting pypy" is a bit of a stretch. It was years ago though
- vegabook 11y agoIn other words, a lost opportunity. Asyncio and this new syntax is both hard for beginners and experienced python coders alike, and still doesn't do multicore!. We're clocking up 3.x version numbers as if this will magically provide the illusion of progress, but the killer feature is still not there. Instead we get type annotations. In a dynamic language. Which doesn't compile and therefore doesn't need them. With no performance advantage. If I have to type declare everything, I want 10x performance. Okay? If Python were a listed company, the CEO would have been replaced long ago. I'm tired of watching my favourite language flail around like this. Will Continuum Analytics or Enthought please fork 2.7?
- sonicrocketman 11y agoI think the limitations of the GIL are not all in all a bad thing, but I do think that the Python community needs to seriously look at making Asynchronous execution a priority in future versions. Asyncio is really complicated and provides very little in terms of performance. Multitasking is the most important issue that Python faces today (and maybe PyPy is the answer).
- fijal 11y agoThe problem that python is facing (and pypy is not solving) is that in order to have a working async library ecosystem, everything, especially the stdlib has to be async aware. This is a far cry right now and I don't see any of the python devs really trying to address that. Otherwise you end up with the same situation as for twisted - it works and it's great, but every single library needs to be made twisted aware
- vegabook 11y agoSo we're in 3.5 already, and one of its central features, async, is flawed, you say. So not only do we not have a killer feature, but even one of the nice-to-haves is a dog. Timeout. This project needs new leadership. The competition (Golang) is walking all over Python on the async issue.
- 11y ago
- keypusher 11y agoconcurrency in python is kind of a disaster, in my opionion. there are a lot of different options but they all seem to have significant drawbacks, and not just limited to ease of use. i know concurrency is a hard problem, but i wish there was one really good, straightforward solution instead of 3 or 4 different half-baked convoluted solutions (threading, multiprocessing, asyncio, subprocess in stdlib, plus twisted, gevent, pulsar etc as third-party).
- sonicrocketman 11y agoFrom what I see, the community keeps changing its mind every few years. We need to pick a multitasking method and really embrace it. If we want to keep the GIL, then we need to go whole hog on some other approach.
- vegabook 11y agoA community which changes its mind every few years is a community which does not have effective leadership. Python's leadership is ineffective because 3.x has erased its credibility. You have maximum one opportunity to change a language in mid course. That opportunity has already been used (wasted?) in the case of Python, and while it could have addressed the elephant in the room, multicore, it did not. Instead it indulged the wishes of its powers that be to tinker on the edges to make Python more "serious", in the process adding nothing that anybody needed, while reducing simplicity and accessibility, and, criminal sin, no backward compatibility with no corresponding forward killer feature. Now, nothing less than a wholesale change of the leadership can right the listing ship.
- 1st1 11y ago> Now, nothing less than a wholesale change of the leadership can right the listing ship. I'd suggest you to go to python-dev and post a concrete (and realistic!) proposal on how to improve whatever you think is broken. Suggesting to change the leadership isn't really productive.
- 11y ago
- ikken 11y agoI've been using the new async/await syntax to write beautiful asynchronous websockets server and I fell in love with it. It handles hundreds of thousands of concurrent connections (on top of aiohttp) and the code is so much cleaner than it would be with i.e. NodeJS and Express and Promises. It reads like a serial code. I think benchmarking asyncio with any type of CPU bound tasks misses the point. Previously we were relying on hacks like monkeypatching with gevent, but now we've been presented with clean, explicit and beautiful way to write massivly parralel servers in Python.
- findjashua 11y agofyi: ES2016 javascript also supports async/await, and you can use them today w babel
- Kudos 11y agoNo it doesn't. It just hit stage 3 in TC39 yesterday, meaning browsers should be considering implementing it experimentally in order to solicit feedback from developers. When it progresses beyond that you can start calling it ES2016.
- findjashua 11y agoi stand corrected, somehow i was under the impression that they were a part of es7 (es2016). So does this mean, there's still a chance that they might not make it?
- Kudos 11y agoIt's going to make it in some form, possibly even it's current form, but it isn't formally part of the spec yet.
- tomchristie 11y agoAnything you can share? Some good examples of networking code with async/await would be incredibly helpful. The documentation covers the primatives but I have a hard time putting it all together.
- ak217 11y agoThis mirrors my experience precisely. I was very excited about async/await, hoping that this would integrate coroutines into regular Python scripts, without the need to manage some complex dispatch engine. I was equally disappointed to learn that it's business as usual, with painful and inadequate semantics out of the box. At least we have pypy. The community should really be rallying behind that project.
- 1st1 11y ago> This mirrors my experience precisely. I was very excited about async/await, hoping that this would integrate coroutines into regular Python scripts, without the need to manage some complex dispatch engine. I was equally disappointed to learn that it's business as usual, with painful and inadequate semantics out of the box. You're not supposed to use async/await without a framework like asyncio or tornado. One way to provide a better UX is to merge asyncio into Python on a deeper level, but this is something that many people won't like.
- btreecat 11y ago>I did this for two reasons, the first being that I cannot, for the life of me, figure out how to use asyncio to do local file IO and not a network request, but maybe I'm just an idiot. I don't think you are an idiot I just think you didn't search well enough. There is a reason there is no local file IO with asyncio. Check these links for more info: * https://stackoverflow.com/questions/87892/what-is-the-status-of-posix-asynchronous-i-o-aio https://stackoverflow.com/questions/87892/what-is-the-status... * http://blog.libtorrent.org/2012/10/asynchronous-disk-io/ http://blog.libtorrent.org/2012/10/asynchronous-disk-io/ From what I understand the way libuv (what node.js uses) gets around OS limits is with a thread pool. Also, this documentation might be helpful: * https://docs.python.org/3.5/library/asyncio-dev.html#handle-blocking-functions-correctly https://docs.python.org/3.5/library/asyncio-dev.html#handle-... My experiences don't seem to mirror your frustration. I found asyncio quite useful for tinkering with a simple web scraper that hit multiple sites at once (each with a different response time) and munged all the data together into one data set. Thanks for the write-up!
- sonicrocketman 11y agoThanks for the suggestions!
- sandGorgon 11y agohave you used a database in a non-blocking way with asyncio ? I'm thinking of the way Psycogreen works or node-postgres works.
- ikken 11y agoPlease check aiopg [1]. There's also an interesting list of packages for asyncio [2]. [1] https://aiopg.readthedocs.org/en/stable/ https://aiopg.readthedocs.org/en/stable/ [2] http://asyncio.org/ http://asyncio.org/
- csytan 11y agoWhile I haven't had a chance to use 3.5's async/await syntax, I have used AsyncIO pretty heavily to deal with multiple sensor inputs/outputs on a Raspberry Pi. The author is right. If you write a coroutine, any code that uses it must also be a coroutine. This is pretty annoying when you're trying to test something manually. It bubbles up this way until you eventually hit the event loop. If you're trying to debug a coroutine in the interactive shell you've got to do something like this: loop = asyncio.get_event_loop() # Blocking call which returns when the hello_world() coroutine is done loop.run_until_complete(hello_world()) loop.close() That's my main beef with it. Debugging can also be painful because when you hit an exception, your stack trace will also involve the asyncio library. Aside from those complaints, I'm a fan. It works fine and reads better than callback-style code.
- mangecoeur 11y agoYou have a similar problem in nodejs - if the api you want to use is async it has to take callbacks, so if you have to do something with that result you have to nest in another callback etc... it gets ridiculous. But it's pretty much a fundamental issue with async code - to keep things async then you have to set up the whole interdependent network of functions to be able to work async, since you don't know when anything is going to return its value. It takes some getting used to - I think Python people are likely to have more trouble with this precisely because Python is a very clear, explicit, and mostly imperative language -you can read Python code as a sequence of instructions and that will be pretty much the way it gets executed, which makes understanding programs very easy. Async is simply a less intuitive way to program, so to adopt it you have to be sure it's worth the hassle of giving up easily understood code.
- deleted 11y ago[deleted]
- n0us 11y agoWhen he brings up requests in comparison to urllib he seems to not know about aiohttp https://github.com/KeepSafe/aiohttp https://github.com/KeepSafe/aiohttp
- justinlardinois 11y agoTangentially related: of the people and projects that are using Python 3, why? I've found that aside from syntax and a few features here and there, Python 2 and 3 are more or less the same technology-wise, especially since 2.7 has a lot of features backported from 3. Thus to me it seems better to stick with 2 because there's so many existing libraries and CPython is the only implementation with complete Python 3 support. If Python 3 had good concurrency and optimization (something neither version has right now), I'd consider using it, but is there an already existing reason that I'm just not seeing?
- fweespeech 11y agoPython 2 has an EOL in 2020. Why start a new project in a language that will have support dropped in ~4 years? If it isn't going to be around in less than 5 years, I'm not going to use it for new projects which have lifespans measured in decades. I've got projects still in production, almost unchanged [only changes were to update business rules], for 4 years. It moves millions of dollars a year. I sure as hell don't want to worry about OS support for it another 4 years.
- antoinealb 11y agoThe amount of libraries working on Python 2 only is really small and not a problem for a lot of us: http://py3readiness.org/ http://py3readiness.org/ The syntax is really nicer IMHO. Unicode strings are really nice to have too. Plus Python 2.x will be EOL'd soon. I just don't see the point of working with it anymore.
- mangecoeur 11y agoSimple answer - It's the current (and currently updated) version. Why would you not use it when starting a new project? More fancy answer - There's plenty of nice improvements to the language (better handling of iterators, tidied up std lib, modern objects, etc), though it's probably worth it for unicode support alone - trying to convince python2 to properly handle international text is just a huge pain (also - no you can't just replace all accented characters with non accented ones if you want to preserve the correct meaning of text and not piss off all your customers by misspelling their names). Yes you can eventually solve all problems with python2. But why bother when you can just use the latest version, there's basically no cost for new projects.
- aidenn0 11y agoI don't use python all that much, but out of curiosity, what's the problem with multiprocessing? In the languages I do develop in, I find it much easier to reason about than multithreading.
- andor 11y agoData exchanged between the processes is serialized with pickle. Pickling is slow, adds latency, and doesn't work on all objects. https://docs.python.org/2/library/multiprocessing.html#pipes-and-queues https://docs.python.org/2/library/multiprocessing.html#pipes...
- aidenn0 11y agoThat sucks; there's no shared-memory message-queue implementation?
- beagle3 11y agoany message queue implementation would require serializing (which the Python standard library provides with "pickling"). If there was a reasonable way to share live objects without the GIL, Python threads would use them.
- 1st1 11y agoFirst of all, see this pic: https://pbs.twimg.com/media/COLLg0TUAAA4j79.jpg:large https://pbs.twimg.com/media/COLLg0TUAAA4j79.jpg:large asyncio doesn't provide any nio abstractions for files because (a) it's not really needed, and (b) there is no easy way to implement it. (a) basically you shouldn't expect your code to block on disk io. But even if it does block for a very short amount of time it's probably fine. (b) one way to implement nio for files is to use a threadpool. Maybe we'll add this in later versions of asyncio, but it will require to write some pretty low level code in C (and reimplement big chunks of asyncio in C too). Another option is to use modern APIs like aio in Linux, but as far as I know almost nobody uses it for real. Bottom line -- you don't need coroutines or asyncio to do file io. What you need asyncio (and frameworks like aiohttp) is to do network programming in Python efficiently.
- sonicrocketman 11y agoAfter playing with Asyncio more, I certainly see your point. I guess my larger point was that, even with the new async/await syntax, Python's general multitasking is still rough and incomplete imo. I thought async/await was a solution, but it's not. I just imagine it being so much simpler.
- athrun 11y ago"you shouldn't expect your code to block on disk io" uh?! In what world are you living in? Blocking the main thread of your app on disk I/O makes for a poor user experience. Disk I/O is a big deal for a whole range of applications. I would even say almost all apps have to deal with disk I/O at some point, as opposed to network I/O.
- beagle3 11y ago> basically you shouldn't expect your code to block on disk io. But even if it does block for a very short amount of time it's probably fine. No, it's very often not fine. Magnetic disks, still the norm for many, and definitely with large storage, often go as low as 5KB/s for random access (or even sequential access to very fragmented files). Reading a 1MB file can easily take 5-10 seconds in some setups - which is not acceptable for any interactive service. It's not fine for a web server to not service any requests for 5 seconds. > Another option is to use modern APIs like aio in Linux, but as far as I know almost nobody uses it for real. Anyone I know who tried came back screaming. There is no way to do an async file open, for example - which means that if you rely on aio, you can block for 10 minutes waiting for an NFS or SMB mounted file to open. The only sane, portable thing to do for Unix/Posix is use a threadpool for async file io - or just use something like libuv which already abstracted async operations this way.
- Lofkin 11y agoBetter multitasking and concurrency syntax, with task scheduling and out of core support here: http://dask.pydata.org/ http://dask.pydata.org/