18 ms·
I don't understand Python's Asyncio
- stygiansonic 10y agoFor background, the author founded the Flask project [0] among others, and contributed to many other well-known Python projects. 0. http://lucumr.pocoo.org/projects/ http://lucumr.pocoo.org/projects/
- dom0 10y agoAlso https://www.palletsprojects.com/ https://www.palletsprojects.com/ Btw. does someone know why some things where moved from Pocoo to Pallets?
- ubernostrum 10y agoFor more background, Flask embodies a lot of what Armin likes about how to build things, and there's room to disagree with him on it. (for example, Flask attaches the request to a magic thread-local variable, while Django -- to pick the one I prefer -- requires you to explicitly pass it around and write functions to take it as an argument; there's a parallel in Armin's complaints about asyncio requiring you to explicitly pass things around instead of accessing magic thread-local storage)
- forgotpwtomain 10y agoFrom my understanding Armin's complaint isn't solely that you have to pass it around but the fragmentation of not having a standard way of doing it: >> pass the event loop to all coroutines. That appears to be what a part of the community is doing. Giving a coroutine knowledge about what loop is going to schedule it makes it possible for the coroutine to learn about its task. >> alternatively you require that the loop is bound to the thread. That also lets a coroutine learn about that. Ideally support both. Sadly the community is already torn of what to do. Personally I'm not clear on a lot of the minority-use cases that spawns some of this complexity (e.g. when and why would you need to move event loops across threads?) and am quite out of my depth as well.
- doh 10y agoSo I'm not the only one. I wrote larger-ish project while using asyncio and it is pain. The syntax is very unfamiliar (especially in 3.5 with async/await), the documentation is confusing and it's very hard to debug it in general. Also it's very hard to combine/stack multiple IO heavy events (make 5 calls to these URLs and whichever is done and returns these task run them in parallel).
- dom0 10y agoI feel you. Having worked with it a bit more I'd say that these things will work out as time goes on. Currently there is quite a bit of fragmentation of async-solutions in Python that are often somewhat-but-not-fully compatible; the docs need a lot of love as well. While the reference in itself is okay, cross-references and conceptual docs (the latter being extremely important IMHO) are lackluster or non-existent. Also there seems to be a lot of confusion caused by the lack of docs around the difference of the keywords and the "asyncio" module (and it's various other forms). The former is just a coroutine / suspension engine and has relatively little to do with asyncio. --- Specifically your last request: https://docs.python.org/3/library/asyncio-task.html#asyncio.gather https://docs.python.org/3/library/asyncio-task.html#asyncio.... It is inherently somewhat limited by requiring the same underlying loop. But this is hard to change on a principal level... --- In a broader sense, I like the increased visibility this gives to async APIs; with the "old ways" this could be easily overlooked. Since good API design is even more important in an asynch piece of software than in a sync piece of software I feel like this is quite an advantage, avoiding bugs and highlighting API issues directly. With the old way, for example, it was relatively easy to confuse a synchronous method and a coroutine, since it's completely silent. You'd only notice this when stuff doesn't happen that was supposed to happen, and you can only see the bug at the call site if you know whether it's a method or a coroutine. This can't happen with the new keywords anymore. I think this is arguably their greatest advantage.
- greglindahl 10y agoThat last part is exactly what my webcrawler does with asyncio plus threads: https://github.com/cocrawler/cocrawler https://github.com/cocrawler/cocrawler This is the part that sends work to a thread: https://github.com/cocrawler/cocrawler/blob/master/cocrawler/burner.py https://github.com/cocrawler/cocrawler/blob/master/cocrawler... I agree that this was confusing in the docs. Docs can be improved. It really helped that this isn't my first crawler written using cooperative multitasking.
- r4pha 10y agoI had a similar experience with python 3's asyncio. I have worked with gevent, which has an arguably less "elegant" interface (with monkey patching, for instance), but with which it is so much easier to be productive. I was frustated for having dificulties understanding and using asyncio. The author is a better python programmer than I am, so I supposed there is really a problem there.
- ddorian43 10y agoSo I guess it's ok for now at least to stay with gevent ? Does asyncio has any pros (except that it's explicit) compared to gevent ?
- rspeer 10y agoPeople don't write articles named "I don't understand gevent" because nobody expects to understand gevent. Armin's complaints are for library writers. You will do much better using an async framework with support from the core language (asyncio) than a monkeypatch.
- the_mitsuhiko 10y agoasyncio is clearly where the community is moving. Sooner or later everybody in the 3.x world will use it.
- jerf 10y agoI think I've made the point a few times that I don't like this style of programming at all, because the coroutine layer turns into an Inner Platform [1] replicating all the control-flow structures the original language has, which then has to integrate with the original language which causes more than twice the complexity to emerge. But it's hard to bash together an example of how that happens in a comment, when it's all faked up and easy to dismiss. This is a great example of the complexity that can emerge. Some of that is incidental and will be fixed in future releases, but some of that is pretty fundamental, such as the initial list of all the sorts of things you need to learn about to use asyncio. Languages like Erlang or Go that have the event loop simply embedded into the language from day one have a much shorter list of such things you have to know, and it does all the things that this complicated list of asyncio concepts do. But I will also admit some of this complexity is definitely Python-specific. I've been using Python since 1.5.2 was still the version you'd be most likely to encounter and I've liked it as a language for a while, but one release at a time, the language has been getting more and more complicated. By the time asyncio came around, the language was quite complicated, and sticking asyncio on top while integrating it with everything else is really a mess. Python has too darned many features at this point. I don't know exactly when it jumped the shark feature-wise, because individually they all make sense, but the sum total has gotten quite unwieldy. Watching new programmers try to learn Python, a language I used to suggest to such people as a good language to start with, has been a bit dispiriting lately. [1]: https://en.wikipedia.org/wiki/Inner-platform_effect https://en.wikipedia.org/wiki/Inner-platform_effect
- RodericDay 10y agoI don't think the number of features is too problematic if it is properly gated. Python having 'http.server' or a Fractions module or tkinter or a matrix multiplication operator comes off as a pleasant surprise, it doesn't hamper the onboarding experience. However, async, typing syntax, multiple string formatting syntaxes, and so on all do. The ship is being helmed irresponsibly in this regard. Letting decorator and keyword asyncio coexist, as well as threads and multiprocess, just feels like a huge mess.
- wslh 10y ago
- dschiptsov 10y agoThis is what happens when coders are rushing to code without real understanding what they are doing and why. What is worse - they borrow "features" without understanding from amateur JavaScript projects or C#. What a bloated mess. This is clearly the second system syndrome, described in the Mystical Man-month. In good old times futures were macros on top of delay and force special forms, and explicit message passing a-la Erlang would do the job.
- orf 10y agoWhat a confused comment. You're saying its bad because it's borrowed hastily from other languages, and a better solution is... borrowing features from Erlang?
- dschiptsov 10y agoNot at all. Erlang's approach to concurrency has been well-researched and validated (Akka). Async, await and friends are mere standardized kludges - popular syntactic sugar without clear semantics and real world connection (explicit message passing mimics how biological systems do self-regulation). So called enterprise languages, especially C++ are full of similar stuff (kludges).
- orf 10y ago> popular syntactic sugar without clear semantics and real world connection Uhh... the point of them is that its as close to the semantics of synchronous code as possible. That's the 'real world connection' - your single threaded code can become asynchonous with just a few keyword changes. Rather than "sendRequest()" you do "await sendRequest()".
- blakeyrat 10y agoC# does all this stuff far clearer and cleaner (and probably more performant). If they had taken more ideas from C# (especially ExecutionContexts), a lot of his complaints would fade away. He actually explicitly calls this out towards the bottom of the essay.
- 10y ago
- adolgert 10y agoThe Fluent Python book has a nice set of chapters on coroutines, futures, and async.io. They present not the whole of what it does but one way to do it, which helps.
- earthnail 10y agoI just got started on Python's asyncio. It took me a long time to understand how to get things done, but after about a week of doing small example projects in the evenings, it finally 'clicked'. For the record, I never worked with gevent before, and have only done a small webserver with Tornado a long time ago, so I can't comment on how it compares to other solutions. One thing that made me wait for asyncio is the fact that, despite being experimental, it has a lot more of an "official" feeling to it. I'm trying to stay away entirely of the Python 3.4 way with coroutine decorators, and am using only await and async in Python 3.5. The async code I wrote has to live in parallel with regular synchronous Python code in a large scientific code base, but migrating our custom database adapter to an asynchronous codebase without breaking old synchronous code was surprisingly easy. Debugging is, in my opinion, a pain. Stacktraces can be extremely long and very hard to understand. The only profiler that seemed to give useful results was pprofile (in sampling mode). I also still don't fully understand why there's both Futures and Tasks - I probably didn't spend enough time understanding the difference, but that just means the author of this blog post is right. Mixing asyncio with threads and/or processes, however, is surprisingly easy and elegant. I hope the Python developers will have the courage to break backward compatibility in the asyncio module, and will remove the old yield from and @coroutine way of doing things. That would probably help a lot in reducing confusion. There's still not a lot of information about asyncio when you google for it, so the amount of existing code and examples that that would invalidate would not be too high. All in all, we are very happy with asyncio. We use it mainly to add concurrency to small sections of our code base. By default, all our code is synchronous, with some heavy I/O-bound functions exposing async versions, too. asyncio allows us to parallelise these sections without the use of a thread pool, and thanks to Futures/Tasks and queues, it's actually very easy to do this in a "streaming" fashion if the order of processing of the outcome of your concurrent tasks matters. Add to that the executors which allow you to run stuff in sub-processes when you're CPU-bound (instead of I/O), and it makes for a fairly solid tool.
- takeda 10y ago> I also still don't fully understand why there's both Futures and Tasks - I probably didn't spend enough time understanding the difference, but that just means the author of this blog post is right. Because they are essentially the same. Task is just extending Future and adds extra functionality (like for example keeping track of tasks schedule in given event loop). It is created when you call ensure_future() or loop.create_task(). You are not supposed to create it directly, so if you're wondering whether you should use Future or Task you should use Future.
- 991821911 10y agoI still prefer Twisted, it feels more natural to me. But this is something that one cannot say aloud in many Python circles.
- rgacote 10y agoLong-time Twisted user (and lover) here. Lots of what we build are fairly web-transaction data translators (http to http or http to db APIs). I find that async.io, by avoiding callbacks, allows us to create a more classical-looking code structure. It works for us. That being said: a) the Twisted environment is extremely robust and battle tested. b) I know that after six months of writing async.io code I'm still just tipping my toe into the capabilities and have yet to fully wrap my head around the details. Would love to see one of those single-topic O'Reilly books take a deep dive into async.io details.
- bogomipz 10y ago>"But this is something that one cannot say aloud in many Python circles" Can you explain why this is? What's the central issue?
- X86BSD 10y agoTwisted ROCKED. I remember when it came out everyone I knew thought it was pretty kick ass, at least those who developed in Python. Did something happen that it's now fallen out of favor?
- the_mitsuhiko 10y agoTwisted is still popular. Problem is that in 2.x the async ecosystems do not work together and many things might not exist everywhere. Twisted seems to have lost a bit of ground with newer things like Kafka etc.
- rspeer 10y agoBy now, people expect libraries to have thorough documentation on the Web, not just an O'Reilly book. The fact that the port of Twisted to Python 3 is slow-going, and far from complete, also gives the suggestion that there are corners of the code that developers don't even understand anymore.
- orf 10y agoI'm working on some Asyncio stuff right now. Asynchronous programming seems pretty natural to me, but other people do struggle to wrap their heads around somewhat confusing terminology: tasks, co-routines, awaitables, event loops. Underneath the terminology the theory is pretty simple. And boy is it powerful. If you ever find yourself doing network requests in a loop (for url in list: requests.get(url)) then a small bit of refactoring and a sprinkling of asyncio will speed this up immensely. But it's not just for network calls, you can `await` on threads and processes. It's a joy and I think it's one of the best things in Python right now.
- sidlls 10y agoHave you tried comparing the performance of asyncio based network requests versus multithreaded requests? And also compared the relative complexity of the code? I have never used asyncio in Python, mainly because the very use case you described is solved with multitheading, but that doesn't mean it's solved best that way of course.
- grok2 10y agoThe problem with multi-threading is knowing how to structure your program correctly to avoid issues with concurrent data-structure accesses, etc -- it requires more careful coding. With asyncio, you don't have to think about these issues.
- sidlls 10y agoI...disagree. And I'll leave it at that.
- orf 10y ago> Have you tried comparing the performance of asyncio based network requests versus multithreaded requests Nope, but the performance of an individual request overhead isn't much of a data point. The advantage is that it scales to thousands of connections easily, there are no concurrency/threading problems, and you can mix and match protocols easily. None of that is easy with threading. Threads can also be quite expensive to start and manage. That specific case is handled by threading, yes, but if you're making a webservice that makes requests to a bunch of endpoints during when processing a HTTP request and also sends output to IRC/Slack whilst simultaneously serving files over FTP and launching a bunch of external processes for good measure then asyncio has your back.
- jftuga 10y agoI wrote a fast tcp port scanner using Python 3.5 and concurrent.futures.ThreadPoolExecutor. See lines 123-124: https://github.com/jftuga/universe/blob/master/tcpscan.py https://github.com/jftuga/universe/blob/master/tcpscan.py I have used this under Linux, OSX and Windows. It's cool to add the Thread Count field in Task Manager and then see something I wrote use so many threads! I am more of a sys admin, so this code could be better - but it seems to work very well. :-)
- orf 10y agoWhatever works for you, but changing 20 lines of code in this to use asyncio will send your throughput through the roof.
- revelation 10y agoThe limiting factor for a fast TCP scanner is not your favorite async toolkit.
- orf 10y agoWell, no, of course not. Don't be stupid. The default ThreadPoolExecutor size is the number of cores you have on your laptop. In my case, 8. So that's 8 concurrent connections at once. That's, frankly, shit. Use asyncio (or roll your own toolkit, or just spawn 'nmap') and you get 20000 concurrent connections. Which is better?
- rantanplan 10y agoYou're conflating concurrency with parallelism. Your 20000 concurrent connections with asyncio are not parallel. You can have 20000 threads if you want.
- orf 10y agoThe connections are actually handled by the OS in parallel, it's the callbacks to Python that are not. Why do you need a whole OS thread to handle sending a SYN/ACK? As you well know 20000 threads is not a great way to do anything. Especially in Python.
- scardine 10y agoArmin criticism of Python 3 is nothing new. He is the author of several popular projects including the web framework "Flask". This makes him a person very respected in the Python community - personally I love his taste for interface design. I wish he could interact better with the core team, because some of his rants are not as constructive as they might be.
- whyever 10y ago> I wish he could interact better with the core team, because some of his rants are not as constructive as they might be. Can you be more specific? How could it be more constructive?
- scardine 10y agoSimplicity is a core value of the Python community; this kind of change goes through a long discussion before getting implemented, so I wonder how we ended up with such convoluted design (the reasoning is documented in the PEP system but most of it is beyond my wizardry level). Perhaps this is me showing some cultural bias (I'm Brazilian), but in my social context it is considered rude to make this kind of criticism in public unless you tried your best to address the problem in a more restricted exchange. The more authority you have, more you are expected to come with a better proposal instead of just pointing out the mess.
- the_mitsuhiko 10y agoasyncio is not my construction site and I do not intend to make it mine. I am not an expert in anything asynchronous and as such will not try to make myself that person. This entire adventure started because I was looking for a way to get logical call contexts working in it and this shows that there is not enough internal machinery to support that at the moment. I'm not going to fight for that however. I hope my post was rude.
- rch 10y agoThanks for taking the time to write out your concrete experiences and qualitative impressions in such detail. It's really a tremendous service and I'm sure it'll be well received.
- shad42 10y agoThe complexity is the reason why I came back to gevent, especially since it now supports Python 3... Gevent monkey patching isn't perfect but it works and gets you closer to how an event loop should be used with standard libs IMO, closer to Go.
- anacrolix 10y agoI agree with Armin that Python is becoming burdensomly complicated. Since Python 3.2, the language has started to accumulate a lot of features, Guido seems to be saying yes to everything. Furthermore, the Python 3 fiasco has severely affected Python's popularity. It would be in much greater use now if the clean-up had been done more gradually.
- krick 10y agoI don't quite see why it's being downvoted. Like it or not, but that's what is happening: Python's 2.4-2.7 success was largely due to it's simplicity. It more or less did correspond to values defined in `import this`. It wasn't particularly performant nor "safe/foolproof", it wasn't even that powerful as a language. It lacked (and still lacks) good error reporting system. But it was lean, easy to get started with, agile enough, expressive enough. It was always possible to make something unintelligible using reflection and magic methods, but it is easily distinguishable form "how stuff should be written" and otherwise you could be quite sure there won't be any surprises. This is why python is still used for it's purpose. CLI utils, bots/crawlers, tf/pandas/scikit-learn, REST-APIs on top of flask. Then hard times came. First, many year long story (still not concluded) of transition between 2 and 3. Then all this stuff. Sure, there is such thing as progress. Stuff is invented for some purpose. But now, in 2016 and at v3.6 — Python isn't what it has been loved for. Not easy-to-start-with, nor simple. This return/yield fuck up, showed in the article is absolutely huge deal, for example, and it is not about asyncio per se. Async stuff is always complicated, it wouldn't be that bad if it was all it's about. If some 5 years ago one would use Python just because "why not, I just need to get stuff done", now it's quite likely that after struggling with all this micro-nuisances he would go with golang/js/php/whatever instead.
- int_19h 10y agoThe thing about Python is that unlike most other languages, you don't have to deal with that complexity. You can still write Python 2.7 style code, no problem. This approach to async, though, is just a language feature that's becoming mainstream right now. C# has it, ES7 has it, C++ has a working paper on it etc. Python actually had the benefit of watching how things work out elsewhere before implementing it all.
- lyschoening 10y agoThere are a few new concepts that need to be learned when beginning work with asyncio as well as some confusing naming choices (who would expect that a coroutine is not the same as a coroutine function?). Some of the tooling is also lacking. Armin points out, no doubt with a look at server frameworks, that there is no elegant way to access the context of a task. That being said, as someone who started working with asyncio in Python 3.5 none of it feels particularly difficult to understand. Asyncio needs more work, sure, but the API so far is relatively straightforward. The overloading of iterators/generators is a bit odd. But the same approach that was taken with JavaScript -- and it's nothing anyone working with Python 3.5 or above will be exposed to. If I recall correctly, Python 3.6 will even feature async generators. Any developers diving into this using that release won't have to ask themselves why they can't 'yield' from within an 'async' function.
- the_mitsuhiko 10y agoI don't find it hard to write asyncio code but I find it hard to generically interface with other people's asyncio code and that is lacking basic support like a context object or logical call contexts or just the lack of agreed upon usage patterns. It does not help that asyncio evolves in the stdlib and changes with every major Python version. It might be less of an issue if this was pip installable I suppose. Right now writing utility code for asyncio is targeting many things.
- nerdwaller 10y agoDefinitely fair, as time progresses I think cleaner patterns will emerge. By time many of the major libraries were around, there was a semi-standard Python programming style that was fairly easy to adhere to. Since this is somewhat of a paradigm shift for the stdlib it will take a little time too. Personally I have been looking at some of the patterns taken by the team on aiohttp - I think they've done a good job and, iirc, one of their members is a core contributor to Python.
- sitkack 10y agoI was talking with David Beazley and he had some of the same confusions around asyncio. I think it would be nice to revisit it with a round table and fix it in 3.7.
- woah 10y agoI'm not really that well versed in Python, but wasn't it supposed to be a language for beginners?
- X86BSD 10y agoI wouldn't go so far as to say it's a language FOR beginners. But it is easier to understand some concepts in python than staring at god awful C or Perl or most other languages as a totally new person to programming.
- pdonis 10y ago> wasn't it supposed to be a language for beginners? No. It was supposed to be a language in which programmers at widely different skill levels, from beginner to expert, could be productive. Easy to pick up the basics, but also easy to use more advanced techniques when you find you need them.
- poletopole 10y agoI didn't need to understand everything about asyncio to use it successfully. It's definitely a lifesaver when it comes to making lots of requests; hours of waiting turn into seconds.
- kamyarg 10y agoPython core developer Brett Cannon wrote a nice(and long) post about understanding it a couple of months ago[1], that might help. After reading that I felt like I could use it, but never got the chance to try. [1] http://www.snarky.ca/how-the-heck-does-async-await-work-in-python-3-5 http://www.snarky.ca/how-the-heck-does-async-await-work-in-p...
- balloob 10y agoAt home-assistant.io we just migrated our core from being based on threads + locks to use asyncio. We managed to keep a full backwards compatible API available so we can slowly migrate other parts of the system over to async. Asyncio has a steep initial learning curve (especially in our hybrid setup) but it's well worth it. We target low resource computers like Raspberry Pi and using async over threads has speed up things a lot. The biggest catch is that while writing code you have to think about every function that you use. Is it doing I/O, is it a coroutine or is it callback/async friendly.
- ddorian43 10y ago... why do you need locks in threads and not in asyncio ?
- paulddraper 10y agoBecause with asyncio you can run on one thread. Before, you needed multiple threads, or else incur high I/O costs. But then you have to manage your shared state across threads, hence locks. Their code is now probably async single threaded code, like JS, where locks are irrelevant.
- ddorian43 10y agoBut don't threads run cpu-stuff in 1 thread, by not switching? (in python) The same as async but more overhead/slower.
- paulddraper 10y agoOnly one thread runs at a time because of the GIL, but the threads can preempt each other wherever between opcodes (http://effbot.org/zone/thread-synchronization.htm#atomic-operations http://effbot.org/zone/thread-synchronization.htm#atomic-ope...) Imagine two threads running a += 1 That's four opcodes. Only one operation will run at a time, but one thread might preempt the other midway through the four operations, and `a` will have been incremented by 1, not two. In some sense, mulithreading with Python's GIL is the worst of both worlds: you can use only one CPU, but you still need to account for concurrent state mutation. (Of course, it doesn't have the function color problem of asyncio http://journal.stuffwithstuff.com/2015/02/01/what-color-is-your-function/ http://journal.stuffwithstuff.com/2015/02/01/what-color-is-y...)
- coleifer 10y agoI've been a gevent user for a long time and Python's decision to "bless" twisted by adopting it's patterns was a watershed moment for me, and basically was the beginning of the end of my belief that I'd ever adopt Python 3. User jerf's comment that asyncio "more than [doubles] the complexity" is absolutely correct. Watch this video of Guido talking about tulip...or struggling to talk about tulip, rather. It's clear the dude is out of his depth and my god the recent changes to the language show that the inmates are now running the asylum... Seems like Python, in it's effort to chase the latest fads, is no longer the language I would endorse to someone new to programming. Whether you think that's a meaningful litmus test or not, the staggering amount of crap that's infiltrated the language now completely flies in the face of the zen of python's statement that there should be one and preferably only one way of doing things. Fuck. I'm going to go code some lua now. https://www.youtube.com/watch?v=1coLC-MUCJc https://www.youtube.com/watch?v=1coLC-MUCJc
- mangeletti 10y agoHey Charles, I had the pleasure of working over IRC with you (at Ellington) when I worked for CMG Digital, and I found you to be one of the most knowledgeable Python developers I had met. But... I think you're blowing this out of proportion. I think that yes, the implementation details are far too complicated, and yes there has been some serious (IMHO) mistakes made (e.g., type hints, format strings, etc.) in Python 3, but overall the language is terrific and getting better.
- hyperbovine 10y agoOh come on. Hyperbole much? You're acting as if it's impossible to write any sort of meaningful code in 3.x without embracing asyncio. A few 2to3-ish language changes excepted, you can punch the exact same code into the Python 3.x interpreter that you have been for the past 15 years.
- carapace 10y agoNot much. I agree with coleifer (and others) that Python 3 is losing "the zen". The new f-string stuff is a perfect example: a bad, retrogressive idea that never should have been approved. I have no plans to adopt or use Python 3.
- omginternets 10y agoWhat's missing from the asyncio docs, IMHO, is a section on common asyncio idioms and patterns. I understand the pieces individually, but I struggle to get a more global, Gestalt understanding of the system.
- the_mitsuhiko 10y agoThe problem there is that not everybody who develops asyncjo/tulip agrees on the patterns.
- omginternets 10y agoYou're probably right! I just (for once!) wish a few opinionated people would tell me what they think!
- agumonkey 10y agoI consider A.Ronacher as a very very competent guy. I felt really bad not mastering asyncio; now I'm laughing green.
- agumonkey 10y agoI consider A.Ronacher as a very very competent guy. I felt really bad not mastering asyncio; now I'm laughing green.
- mangeletti 10y agoThe one thing that I think is absolutely lovely is the syntax: async def spam(eggs): ... It can't get much better than that. My hope is that the implementation will become: A) simpler B) more optimized C) out of the box functional (e.g., no need to manage event loops and other things yourself) D) unified (e.g., the zen of python even states that there should be one way to do something) I also strongly agree with the comment @RodericDay made herein, pertaining the new typing syntax and the addition of yet another string formatting syntax (a dangerously unexplicit one).
- voltagex_ 10y agoHave a look at the way C# does it. I think it comes pretty close.
- sametmax 10y agoActually using asyncio tools is quite easy and straighforward. However, writting an asyncio lib, or god saves you, an asyncio framework, is really, really hard.
- carapace 10y agoCoroutines are goto. That's not a joke or anything, the semantics of coroutines are the semantics of goto statements. All this async business is just the old spaghetti sneaking back in while people are distracted by the nomenclature.
- 1st1 10y agoThen for-loops are goto as well.
- y11 10y agoA for loop is localized. I've seen Python code bases that use so many interdependent iterators that the code is non-refactorable and the developers just pray that it does the right thing. That's plain iterators. Now people throw in coroutines.
- xapata 10y agoAn exception is a goto... Your conclusion doesn't follow from your premise.
- carapace 10y ago> An exception is a goto Not quite. > Your conclusion doesn't follow from your premise. What conclusion? ----- The difference between a coroutine and a subroutine is that the subroutine cannot select the destination of a return statement: control always "returns" to the caller. Coroutines can choose where the control flow goes after they run. This is exactly what GOTO statements permit. There is no formal difference between allowing coroutines and allowing goto statements. (And we all know what goto is considered, don't we?) Exceptions are not the same: the control is passed to a handler but the raiser/thrower (in general) does not choose the catcher.
- int_19h 10y ago> And we all know what goto is considered, don't we? A very powerful and occasionally useful tool that's often misused, and misunderstood because of the aforementioned misuse?
- plq 10y agoThe general sentiment in this thread seems to be that Guido et al "blessed" the Twisted way, it's a pity and now using asyncio and friends is the only way to suspend a Python stack in a concurrent program besides using OS-level threads. This is not correct. Green threads, as a programming paradigm, are just a sub-optimal (but cheaper) way of doing preemptive multitasking. Yes, switch is explicit, but user code can't know what will switch. So you are not supposed to treat green threads any differently from OS threads. ie you still need your mutexes, semaphores etc. if you want to avoid race conditions. Again, that's because the caller doesn't know whether a function will yield execution to the next queued task at some point. Guido explains this point pretty well in one of his PyCon keynotes. So green threads may be great but they don't bring anything new to the table. However, Twisted-style concurrency (aka cooperative multitasking) is a different paradigm. In Twisted, you know that you only have one thread running at a time, so you actually don't need any thread synchronization primitives when accessing non-local state. This simplifies things a great deal. Yes, not having to spawn a thread for every single concurrent IO operation has other great benefits, but that's not the reason why CPython now has a blessed event loop -- it's cooperative-multitasking-the-paradigm. Before asyncio, there was no standard way of doing cooperative multitasking. Now there is and it's baked right into the language. Use it if you like it. If not, the old ways of doing things work just fine in Python 3. I'll admit that the concurrency model in Python 3.4 was not perfect. However, what we have with Python 3.5 and up looks quite polished.
- int_19h 10y agoAnother issue with green threads is that they usually do not work very well (or at all) across API, much less ABI, boundaries. Basically, the moment your control flow is calling into some shared library, you probably want C API at that boundary for the sake of portability and stability. Exposing what is an, essentially, promise-based system that way is not hard. Better yet, if the other side has some analogous construct, you can map to that. But how do you do it with green threads? Even platforms that have an OS-wide unified green thread primitive, like Win32 fibers, which would presumably solve this problem, find their use very lackluster, because many languages and frameworks just plain don't support fibers correctly. Even CLR tried to do it once and dropped the feature; forget about Python etc.
- 10y ago
- erikb 10y agoFrom what I learned through youtube videos and meetups it seems to solve a problem that doesn't exist (getting all these patterns into python) and in return doesn't solve the problem people hope it would solve (multiprocessing made easy and pythonic). That's why people who want to pretend to be smart (like me a few years back) find this totally attractive, king's new clothes style. Nobody really understands it so they can act like they really do something meaningful with it. How do I get to this painful conclusion? Well, as I said, I was (and probably still am to some degree) just like that. And to me it looks nearly irresistably interesting. But at the same time I also don't know what I would use it for, what it would actually improve for me. And since the need to pay my rent forced me to use my time more practically I didn't get around to looking at asyncio more in depth. Both these things together make me believe it's not really solving a real problem.
- scribu 10y agoNot sure what you mean by "getting all these patterns into Python". Just because you don't work on projects that can benefit from concurrent IO doesn't mean they don't exist. I work on such a project.
- pekk 10y agoThe name "asyncio" includes the substring "io" and is clearly about asynchronous I/O, so it's hard to see why anyone would think its purpose was "multiprocessing made easy."
- deleted 10y ago[deleted]
- jdnier 10y agoThe post mentions in passing David Beazley's curio[1] project. Is anyone using curio for async programming instead of asyncio? The curio docs are fun to read through and didn't leave me feeling lost. They're also full of understandable examples of using the new async/await syntax. I've made some simple scripts with curio but have found I keep hesitating take on the asyncio docs to learn it "the real way". Any thoughts on whether curio might be a plausible alternative to asyncio? [1] http://curio.readthedocs.io/en/latest/tutorial.html http://curio.readthedocs.io/en/latest/tutorial.html
- scribu 10y agoSince it's polling for I/O events and doesn't use threads [1], it does indeed sound like a plausible alternative for doing non-blocking I/O. I think I'm going to take a stab at using it, since asyncio doesn't have a mature ecosystem around it anyway. [1] http://curio.readthedocs.io/en/latest/index.html#under-the-covers http://curio.readthedocs.io/en/latest/index.html#under-the-c...
- yladiz 10y agoMaybe I'm not the intended audience for this article, but I don't really have a problem with a lot of the warts of asyncio. It really is pretty simple to use if you don't have your mind hard set to a different way of thinking. It wasn't as simple in Python 3.4, but then the async/await keywords became part of the language in Python 3.5 and simplified things. Understanding asyncio took me a few days to understand the complexities: the event loop usage and how this works with async/await, how an async function is different than a normal function, and a few other things. Once I understood those complexities (which I don't mind because it is a new part of the language) it was easy. Granted, I feel like it's easier to just work around issues rather than complain about them but I didn't find particularly many issues about asyncio. As an aside, not to pick on Armin but he was also complaining about Python 3 about things that may have been valid, but were more that he didn't like the way that Python 3 worked because he liked Python 2 more, and hid that fact by writing lengthy blog articles about how he doesn't like certain Python 3 things. I do find it a little strange that he does complain about these things publicly rather than trying to fix the things he doesn't like (if they are fixable), especially given his reputation in the community and his Python projects like Flask, because it makes him seem whiny and solves none of the issues that he's presenting.
- the_mitsuhiko 10y ago> I do find it a little strange that he does complain about these things publicly rather than trying to fix the things he doesn't like (if they are fixable), especially given his reputation in the community and his Python projects like Flask, because it makes him seem whiny and solves none of the issues that he's presenting. It's easy write code, it's harder to write specs and design systems and it's hardest to convince others. I'm very bad at the last part. My only real attempt to improve python 3 that went anywhere was to get the u prefix back. My suggestions for bytestrings were not very popular for instance.
- yladiz 10y agoOkay, fair enough, it's harder to think about a problem than write the solution to the problem in general, at least for software development. But it would be better if you did try to talk to the Python developers in a constructive manner and try to get whatever you feel is wrong with Python fixed. Writing blog posts is fine, and I'm not saying that voicing your opinion is bad, but there is a threshold between voicing an opinion and actually doing something about it, and while I feel you do contribute to the Python community, you don't contribute to Python itself even if you have the ability to and after years of blog posts you've crossed that threshold. The reasons that you don't attempt to contribute I'm not sure -- lack of patience for the process, unable to convince others, designing a system that fixes the issues you're finding. The biggest thing is that if you want something fixed and have the ability to fix it (which I feel you would have) don't just sit around and complain -- do something about it. The best way to convince a developer is to write code to prove your point.
- coldtea 10y ago>Since I'm not clever enough to actually propose anything better I just figured I share my thoughts about what confuses me instead so that others might be able to use that in some capacity to understand it. Coming from a well known Python developer, this gives a somewhat passive-aggressive vibe (which might not be meant at all, just sayin').
- gdamjan1 10y agoIt's too late to complain now. I did like gevent/greenlet a lot, but the wider community, for many years, was unforthcoming to it. Now asyncio is in the stdlib, including the language changes for coroutines, better than the status-quo.
- denfromufa 10y agoI find that Armin's blog posts about Python 3 are generally destructive, not constructive. A constructive way would be to post suggestions on python-ideas and python-dev mailing lists or report bugs and feature requests on issue tracker. Python is open-source project and there is not a single core developer working full-time on the language unlike his beloved Rust.
- raymondh 10y agoI've always found Armin's posts to be intellectually honest and thought provoking. And unlike your ad-hominem comment, he itemizes his concerns and provides details (such as the performance comparison with David Beazley's curio project). At the very least, it is a warning sign that a notable and highly experienced Python expert is having a hard time grappling with the best practices (or even workable practices) for a significant new feature set: "I know at least that I don't understand asyncio enough to feel confident about giving people advice about how to structure code for it." As far as I can tell, not a single respondent to this thread has indicated that to the contrary, they have been able to say that they do feel confident enough to give people advice on how to structure code with asyncio. At the very least, that means that we have a documentation and communication problem which is either intrinsic to the new API or something that will work itself out over the next few years.
- ubernostrum 10y agoFor what it's worth, I read Armin's critiques, but I take them with a grain of salt; it's clear that what he wants from a language, and what other people want from Python, diverged a while back and are probably irreconcilable at this point. That doesn't mean he doesn't have good points, but does mean that I read his articles through a lens of "the language he really wants probably is never going to be Python again".
- the_mitsuhiko 10y ago> I read his articles through a lens of "the language he really wants probably is never going to be Python again" I always had that opinion that what I would like Python to be like is never going to happen. This is not something new with Python 3.
- marmaduke 10y agoNo one has to use this stuff if they don't want to. Another complexity in Python is metaclasses. I've written meta classes which generate data descriptors, and was greatful that that was there when I needed it, but I also needed to look at the data model reference constantly and wrote 200% coverage tests.
- brettcannon 10y agoIf people would like some more illustrative examples for using asyncio there's https://asyncio.readthedocs.io https://asyncio.readthedocs.io
- morty16 10y agoFrom early on in the article > asyncio.get_event_loop() returns the thread bound event loop, it does not return the currently running event loop. How can these be different objects? In order to ask for the thread-bound event loop, you must be in the thread, right? When/why would you expect anything else? fyi, I don't have any background with asyncio/twisted.