8 ms·

Python and Async Simplified (2018)

• sedeki 4y ago
  Can anyone recommend a good book/primer on "concurrency models" (is that a term?) for a self-taught programmer? While I am self-taught, I'm used to (academic) books that strive for completeness. It is also what I prefer. Rather than something more pragmatic like a blog post. It doesn't mean I want to read overly complicated prose on the subject, which I'm sure is possible.
  • uniqueuid 4y ago
    I don't have a book recommendation ready, but if you want to see how async can be used in a large codebase, have a look at the telethon [1] library. It's a python library for telegram and one of the few that actually implement MTProto. It's huge, generates a large chunk of its machinery automatically from the MTProto specs and is extremely (!) well structured. This is much more useful than the typical "let's write a single-run example with async" blog post. [1] https://github.com/LonamiWebs/Telethon https://github.com/LonamiWebs/Telethon
  • bombolo 4y ago
    I think you should start by reading how "async" really works… that's call on a poll() (or epoll on linux) function, a loop, and a list of "call this function when this file descriptor can be written/read". The whole async thing is there to abstract away and not have the program structured around the main loop… but in reality you have to keep in mind you are in a main loop that calls poll() and then all the registered functions.
  • cdavid 4y ago
    The book https://pragprog.com/titles/pb7con/seven-concurrency-models-in-seven-weeks/ https://pragprog.com/titles/pb7con/seven-concurrency-models-... is actually pretty good, and much better than the title may suggest.
• kqr 4y ago
  Similar question to the other one at the time of writing, but more specific: does anyone have a good, thorough introduction to the "async event loop" (sometimes known as "asyncio") pattern? By thorough I mean that it goes beyond a starter tutorial, into both examples of various supporting libraries and implementation details that matter for usage. I'm fine with a book, too. There are popular libraries for it in both Python and Perl and I suspect I could make good use for it if I understood it. Unfortunately, I've only ever used it in a cargo cult manner of sticking together functions until the error messages go away (yeah yeah, it was only for "throwaway" "prototypes") so I really don't understand how it all is meant to fit together.
  • hbrn 4y ago
    I found this post to be amazing intro that shows you how to go from simple generators to async event loop. https://mleue.com/posts/yield-to-async-await/ https://mleue.com/posts/yield-to-async-await/
    • kqr 4y ago
      This was very good! But true to what's common, it stops just as it gets really juicy!
• BiteCode_dev 4y ago
  It gives good pointers but it falls short on the usual suspects for an article on asyncio. When teaching it, it's important to emphasis: - await is locally blocking, so you should isolate linear workflows into their own coro, which is the unit of concurrency. - to allow concurrency, you should use asyncio.create_task on coro (formerly ensure_future). - you should always explicitly delimitate the life cycle of any task. Right now, this means using something like gather() or wait(). TaskGroup will help when it becomes mainstream. A HN comment is not great to explain that, but if you read the article, you should investigate those points. There is no good asyncio code without them, only pain and disapointment.
  • uniqueuid 4y ago
    > you should always explicitly delimitate the life cycle of any task Unless you want a hacky actor system, in which case it's totally fine to `create_task` a ton of corountines which have their own spin loop with await sleep :)
    • lozenge 4y ago
      One coroutine crashing and the others continuing to send it messages without noticing was my $40,000 bug.
      • Spivak 4y ago
        I’m so confused by this architecture. It makes total sense in a threaded world but why would you want a coroutine constantly scheduling itself in a loop to pull messages off a queue like thing than just having the thing generating the message fire off a task to process it directly right there? It feels almost the same to me and then you can’t crash the coroutine.
    • BiteCode_dev 4y ago
      At some point even those tasks must be cleanly stopped and unless you want to play erlang and "let it crash", the actors have a lifecycle as well. Making it explicit will avoid much pain, and ease testing a lot. Also it will make resources consumption more predictable.
• landersg 4y ago
  Async is overengineered and bolted on. If you must use Python, I'd still recommend Twisted, which is more accessible. Otherwise, of course use Go, Elixir, etc. in the first place.
  • ogogmad 4y ago
    I don't know. It seems to me Elixir requires a new and rather restrictive programming paradigm. For me, that's even worse than being overengineered.
    • throwamon 4y ago
      > new Erlang has been around since what, the 80s? Elixir is "just" Erlang with a different face and extra features. > restrictive which is? Functional programming? Immutability? Interestingly, Erlang is often called a "true" object-oriented language thanks to its actor model. It's incredibly powerful and flexible, pretty much the opposite of restrictive. Just for a simple example, you can inspect, debug and modify your program while it's running. From your comment it just seems you're not familiar with it.
  • bsenftner 4y ago
    It reminds me of the cooperative multitasking used by the original Mac OS.
  • throwamon 4y ago
    Promising contenders include algebraic effects in OCaml and JVM's Project Loom.
  • kortex 4y ago
    I find Twisted less accessible and more opaque than async. It's also more "bolted on" in that it's an entirely separate library/framework outside the standard lib. Async might technically be bolted on, but no worse than async in most languages which weren't designed de novo for async (eg go/elixir).
• ogogmad 4y ago
  I thought Python couldn't multithread because of GIL? I understood from the article that async derives all its benefit from certain OS-level operations which don't need to run in a coroutine, like reading from a network socket or waiting for timers to finish. Another question: Is Python's implementation of async/await identical to other languages? In particular, do they always use coroutines instead of threads?
  • diarrhea 4y ago
    I understand async/await in Python to be entirely single-threaded. So is, for example, C#'s implementation: https://learn.microsoft.com/en-us/dotnet/csharp/programming-guide/concepts/async/task-asynchronous-programming-model#BKMK_Threads https://learn.microsoft.com/en-us/dotnet/csharp/programming-... ("The async and await keywords don't cause additional threads to be created.")
    • tomnipotent 4y ago
      Eliminates blocking on IO requests, letting the event loop spend CPU cycles doing non-IO work. Alternative is CPU doing nothing while waiting on IO, which for something like a web app doing lots of small network requests to database/cache can add up to a lot. CPU work is still single-threaded.
  • zzzeek 4y ago
    since it's my job to clear these things up, a few pointers: 1. python has threads. they just cannot perform CPU bound tasks in parallel due to the GIL. The GIL is released for IO, so threads can perform IO waiting in parallel, just like asyncio 2. asyncio runs in one thread, and has the exact same limitations as threads as implemented in Python, CPU operations are serialized, async tasks can yield for IO. the advantages offered by asyncio are: 1. you can have thousands of tasks extremely quickly cheaply, which is not as much the case for threads in Python . this can allow for massive concurrent architectures more expediently, provided your concurrency is very IO bound (if you are CPU bound, disaster) 2. people just like asyncio's programming model, IMPO this is largely due to the popularity of Javascript's event-based model being natural for lots of newer programmers
• samsquire 4y ago
  I am really interested in this space. There's an article that Cal Paterson wrote that async doesn't speed up code - it is not parallel. The GIL prevents Python from being parallel. So even if you create a thread to run an async method in Python, it shall not run in parallel to the main thread of execution. (In fact, it shall block the main thread of execution if you start a thread in the thread you are in, due to the blocking run_in_executor) https://calpaterson.com/async-python-is-not-faster.html https://calpaterson.com/async-python-is-not-faster.html I wrote a multithreaded userspace 1:M:N scheduler (1 scheduler thread, M kernel threads and N lightweight/green threads) which resembles Golang M:N model. I implemented the same design in Rust, C and Java. I am thinking it could be combined with my epoll-server and it would be an application server. https://github.com/samsquire/preemptible-thread https://github.com/samsquire/preemptible-thread https://github.com/samsquire/epoll-server https://github.com/samsquire/epoll-server I am also interested in structured concurrency. This article by Vala developers is good. https://verdagon.dev/blog/seamless-fearless-structured-concurrency https://verdagon.dev/blog/seamless-fearless-structured-concu... I am trying to find a concurrent software design that is scalable and is easy to write and hides complicated lock programming. I document my studies and ideas in the open in ideas4. https://github.com/samsquire/ideas4 https://github.com/samsquire/ideas4 I've implemented multithreaded parallel multiversion concurrency control in Java, which is the same approach used by Postgresql and MySQL for concurrent read and writing to the same data atomically. I still think concurrency is hard to write and understand. Even with async/await. // 3 requests in flight result1 = async_task1(); result2 = async_task2(); result3 = async_task3(); await result1; await result2; await result3; I ported a parallel multiconsumer multiproducer ringbuffer from Alek https://www.linuxjournal.com/content/lock-free-multi-producer-multi-consumer-queue-ring-buffer https://www.linuxjournal.com/content/lock-free-multi-produce... I use Python threads in https://github.com/samsquire/devops-schedule https://github.com/samsquire/devops-schedule and https://github.com/samsquire/parallel-workers https://github.com/samsquire/parallel-workers to parallelise a topologically sorted graph of IO of devops programs. This allows efficient scheduling and blocking with thread.join() for each split of the work graph and then a regrouping before doing other things, also potentially in parallel. This pattern is efficient and easy to use.
  • throwway2332 4y ago
• intrepidhero 4y ago
  This is the bit that should be at the very top of the official docs. It's tripped me up every time I go to write async code and until you learn it, the error message is very confusing. > In particular, calling it will immediately return a coroutine object, which basically says "I can run the coroutine with the arguments you called with and return a result when you await me". > The code in the target function isn't called yet - this is merely a promise that the code will run and you'll get a result back, but you need to give it to the event loop to do that. If I try to pass the async function to gather (for example) without calling it, which makes some intuitive sense, since functions are first class objects and I know I'm not calling it, the event loop is, the error message reads something like, "gather only accepts coroutines." But I thought it was a coroutine because I declared it with async! For some reason it took me a silly amount of time to notice that in all the examples, the async function is called when it's passed to gather (or whatever). That's not intuitive to me and the distinction made in the article should be clearer in the docs.
  • hbrn 4y ago
    > If I try to pass the async function to gather (for example) without calling it, which makes some intuitive sense That intuition breaks immediately when you realize that those functions can have arguments, and you have no way to pass them.