3 ms·

I'm not in a much of a position to evaluate Python's asyncio since I really have not used it very much. However, over the last few days, I started to dig into i

I'm not in a much of a position to evaluate Python's asyncio since I really have not used it very much. However, over the last few days, I started to dig into it and tried to get some (what I think are) very basic examples working and really struggled. That alone is not fully dispositive because I've used many async implementations in various languages and each of them have a bit of a learning curve and their own wrinkles you have to ramp up on. That said, my limited experience at least tells me that Python's async has a FUX that leaves a lot to be desired. I've found other languages make it a lot easier to do concurrency, parallelism, async vs sync, blocking vs non-blocking and all that. I don't really know if the issue is poor documentation, the semantics of the APIs themselves...really not sure. I do know that I'm left with the feeling that "this seems like it's going to be a pretty big PITA".

andrewstuart 4y ago
It IS easy. The Python project just don’t make it obvious how to do it easy.
- xwowsersx 4y ago
  Right. So you think it's mainly an education/documentation issue? I'm totally open to that in fact being the case. One really simple use case I had was just annotating a method to run async and then calling that in a loop elsewhere which did speed up a bunch of work I was doing pretty dramatically. import asyncio def background(f): def wrapped(*args, **kwargs): return asyncio.get_event_loop().run_in_executor(None, f, *args, **kwargs) return wrapped @background def my_io_bound_function():...
  - m3047 4y ago
    You need to run the executor somewhere ( loop.run_forever() or loop.run_until_complete() ), that can be in the current thread or a separate thread, keep in mind Python is still conceptually a single core. Things I've found particularly useful: * Optional / debug logging of coroutine start/exit. * Stats logging, including count, runtime, request queue (multiple instances of the same function) depth. * Printing tracebacks within the coroutine context. Haven't expended any effort on decorators, good for you.
    - xwowsersx 4y ago
      Those do sound useful, thanks for this. Will keep in mind as I continue exploring.
    - m3047 4y ago
      I think I hit a foul ball there, as run_in_executor() is for a threadpool. So yeah you are running inside of run_forever() or run_until_complete(), we just don't see it here, and that's where your sending things to run in a different thread. ;-)
      - xwowsersx 4y ago
        Ah got it, thanks for the correction and info.
- btilly 4y ago
  It is easy if you know what to do. But you'll CONSTANTLY run into things where it is obvious that this is a toy that hasn't been thought through. I've found random slowdowns of a factor of 10 for no reason. A context manager needs to have __enter__ and __exit__ methods. An async context manager needs to have __aenter__ and __aexit__ methods. The error you get if you use a context manager as an async context manager or vice versa is big and scary and offers no clue to most programmers of what the problem is. Using async with SQLAlchemy it isn't hard to wind up exiting the event loop before all cleanup has happened. They fixed the main cause of this, but it isn't hard to still trigger it. And so on.
matsemann 4y ago
Asyncio is annoying, and often unexpectedly slow. You think things are parallel, but then one misbehaving coroutine can hog your cpu bringing everything to a halt. GIL makes it useless for anything other than _heavily_ IO bound tasks. And yeah, Python's documentation is useless. Never how to use stuff, only listing of everything that's possible to do / the API. Unfortunately that style is being mimicked by most other Python projects as well.
- rpep 4y ago
  I think the big point of interest will come with Python 3.12 where it's proposed that as part of the faster CPython project, the GIL-per-process will be removed.

andrewstuart 4y ago

It IS easy. The Python project just don’t make it obvious how to do it easy.

xwowsersx 4y ago
Right. So you think it's mainly an education/documentation issue? I'm totally open to that in fact being the case. One really simple use case I had was just annotating a method to run async and then calling that in a loop elsewhere which did speed up a bunch of work I was doing pretty dramatically. import asyncio def background(f): def wrapped(*args, **kwargs): return asyncio.get_event_loop().run_in_executor(None, f, *args, **kwargs) return wrapped @background def my_io_bound_function():...
- m3047 4y ago
  You need to run the executor somewhere ( loop.run_forever() or loop.run_until_complete() ), that can be in the current thread or a separate thread, keep in mind Python is still conceptually a single core. Things I've found particularly useful: * Optional / debug logging of coroutine start/exit. * Stats logging, including count, runtime, request queue (multiple instances of the same function) depth. * Printing tracebacks within the coroutine context. Haven't expended any effort on decorators, good for you.
  - xwowsersx 4y ago
    Those do sound useful, thanks for this. Will keep in mind as I continue exploring.
  - m3047 4y ago
    I think I hit a foul ball there, as run_in_executor() is for a threadpool. So yeah you are running inside of run_forever() or run_until_complete(), we just don't see it here, and that's where your sending things to run in a different thread. ;-)
    - xwowsersx 4y ago
      Ah got it, thanks for the correction and info.
btilly 4y ago
It is easy if you know what to do. But you'll CONSTANTLY run into things where it is obvious that this is a toy that hasn't been thought through. I've found random slowdowns of a factor of 10 for no reason. A context manager needs to have __enter__ and __exit__ methods. An async context manager needs to have __aenter__ and __aexit__ methods. The error you get if you use a context manager as an async context manager or vice versa is big and scary and offers no clue to most programmers of what the problem is. Using async with SQLAlchemy it isn't hard to wind up exiting the event loop before all cleanup has happened. They fixed the main cause of this, but it isn't hard to still trigger it. And so on.

matsemann 4y ago

Asyncio is annoying, and often unexpectedly slow. You think things are parallel, but then one misbehaving coroutine can hog your cpu bringing everything to a halt. GIL makes it useless for anything other than _heavily_ IO bound tasks. And yeah, Python's documentation is useless. Never how to use stuff, only listing of everything that's possible to do / the API. Unfortunately that style is being mimicked by most other Python projects as well.

rpep 4y ago
I think the big point of interest will come with Python 3.12 where it's proposed that as part of the faster CPython project, the GIL-per-process will be removed.