3 ms·

> posted in Jan. 2017 Now we have asyncio and awesome libraries like aiohttp[1] you can get a much, much higher throughput than you'd ever achieve with threads

by orf 9y ago

> posted in Jan. 2017 Now we have asyncio and awesome libraries like aiohttp[1] you can get a much, much higher throughput than you'd ever achieve with threads with less code. 1. http://aiohttp.readthedocs.io/ http://aiohttp.readthedocs.io/

sametmax 9y ago
Plus asyncio has futures too, and with run_in_executor(), you can await something in a thread/multiprocessing pool from inside the event loop transparently.
- greglindahl 9y ago
  Yes, I use this extensively in a web crawler -- network I/O is done in a main thread that's cpu bound doing networking stuff, and cpu-burning tasks like html parsing are done in child threads done by run_in_executor. Without trying very hard I can get to a load of 3-4 during ordinary web crawling.
miracle2k 9y ago
I found this article extremely persuasive, and it matches my own experiences with asyncio: The performance gain might be there if most of what you do is waiting for a network response, but even a small amount of data processing will make your program CPU bound pretty quickly. http://techspot.zzzeek.org/2015/02/15/asynchronous-python-and-databases/ http://techspot.zzzeek.org/2015/02/15/asynchronous-python-an...
- zackelan 9y ago
  Note that it's not either/or - you can dispatch work from an event loop to a thread pool (or a process pool) with loop.run_in_executor [0], while loop.call_soon_threadsafe [1] can be used by worker threads to add callbacks to the event loop. This means that the "frontend" of a service can be asyncio, allowing it to support features like WebSockets that are non-trivial to support without aiohttp or a similiar asyncio-native HTTP server [2], while the "backend" of the service can be multi-threaded or multi-process for CPU-bound work. 0: https://docs.python.org/3/library/asyncio-eventloop.html#executor https://docs.python.org/3/library/asyncio-eventloop.html#exe... 1: https://docs.python.org/3/library/asyncio-eventloop.html#asyncio.AbstractEventLoop.call_soon_threadsafe https://docs.python.org/3/library/asyncio-eventloop.html#asy... 2: Flask-SocketIO, for example, requires that you use eventlet or gevent, which are the "legacy" ways of doing asynchronous IO: https://flask-socketio.readthedocs.io/en/latest/ https://flask-socketio.readthedocs.io/en/latest/
  - deleted 9y ago
    [deleted]
- xtrapolate 9y ago
  > "even a small amount of data processing will make your program CPU bound pretty quickly" I don't know what "a small amount" means. You are bound by hardware (cores, hyper-threading). It simply makes no sense to spawn 32 threads executing computation intensive code (no IO), on a machine with 2 cores.
trentnelson 9y ago
You can get the best of both worlds with PyParallel! Async I/O and multiple threads. (Experimental project, not intended for production use, so I say this some what facetiously.) http://pyparallel.org/ http://pyparallel.org/
- Rotareti 9y ago
  Last commit Oct 2016. Not sure where this is going.

by orf 9y ago

sametmax 9y ago

Plus asyncio has futures too, and with run_in_executor(), you can await something in a thread/multiprocessing pool from inside the event loop transparently.

greglindahl 9y ago
Yes, I use this extensively in a web crawler -- network I/O is done in a main thread that's cpu bound doing networking stuff, and cpu-burning tasks like html parsing are done in child threads done by run_in_executor. Without trying very hard I can get to a load of 3-4 during ordinary web crawling.

miracle2k 9y ago

I found this article extremely persuasive, and it matches my own experiences with asyncio: The performance gain might be there if most of what you do is waiting for a network response, but even a small amount of data processing will make your program CPU bound pretty quickly. http://techspot.zzzeek.org/2015/02/15/asynchronous-python-and-databases/ http://techspot.zzzeek.org/2015/02/15/asynchronous-python-an...

zackelan 9y ago
Note that it's not either/or - you can dispatch work from an event loop to a thread pool (or a process pool) with loop.run_in_executor [0], while loop.call_soon_threadsafe [1] can be used by worker threads to add callbacks to the event loop. This means that the "frontend" of a service can be asyncio, allowing it to support features like WebSockets that are non-trivial to support without aiohttp or a similiar asyncio-native HTTP server [2], while the "backend" of the service can be multi-threaded or multi-process for CPU-bound work. 0: https://docs.python.org/3/library/asyncio-eventloop.html#executor https://docs.python.org/3/library/asyncio-eventloop.html#exe... 1: https://docs.python.org/3/library/asyncio-eventloop.html#asyncio.AbstractEventLoop.call_soon_threadsafe https://docs.python.org/3/library/asyncio-eventloop.html#asy... 2: Flask-SocketIO, for example, requires that you use eventlet or gevent, which are the "legacy" ways of doing asynchronous IO: https://flask-socketio.readthedocs.io/en/latest/ https://flask-socketio.readthedocs.io/en/latest/
- deleted 9y ago
  [deleted]
xtrapolate 9y ago
> "even a small amount of data processing will make your program CPU bound pretty quickly" I don't know what "a small amount" means. You are bound by hardware (cores, hyper-threading). It simply makes no sense to spawn 32 threads executing computation intensive code (no IO), on a machine with 2 cores.

trentnelson 9y ago

You can get the best of both worlds with PyParallel! Async I/O and multiple threads. (Experimental project, not intended for production use, so I say this some what facetiously.) http://pyparallel.org/ http://pyparallel.org/

Rotareti 9y ago
Last commit Oct 2016. Not sure where this is going.