5 ms·

> If these hundred tasks make heavy use of the CPU, then the sync and async solutions would have similar performance, since the speed at which the CPU runs is f

> If these hundred tasks make heavy use of the CPU, then the sync and async solutions would have similar performance, since the speed at which the CPU runs is fixed, Python's speed of executing code is always the same and the work to be done by the application is also equal. I think one of the things the author has not discussed is that if you have high CPU then you are going to get starvation as async does not have a fairness mechanism - timesharing is co-operative and based on FIFO in all the implementations I've seen. If you start to saturate CPU (even a bit) you get uneven distribution of cpu time between your various async threads. This translates into chronic timeouts. For async to be applicable you really need to have a) high io load and b) lowish cpu load.

sltkr 6y ago
> ... you are going to get starvation as async does not have a fairness mechanism - timesharing is co-operative and based on FIFO in all the implementations I've seen. If it's FIFO based, why would you get starvation?
- euiq 6y ago
  What would the CPU-heavy code await on?
- calpaterson 6y ago
  FIFO systems can have starvation when dealing with time bound things like web requests - slow request handlers can cause other (waiting) requests to timeout. In a thread-per-request or process-per-request model where you run say 4 * cpu workers the operating system scheduler is able to interrupt stuff that's hogging the CPU but that isn't the case in a typical 1 * cpu worker async model. When there's little cpu load this is not a real threat but when there is material load you can easily run into problems.
  - sltkr 6y ago
    I see; it sounds like latency would become very unpredictable under that model. I can imagine that's a serious problem in practice, but it's not typically what's meant by starvation. https://en.wikipedia.org/wiki/Starvation_(computer_science) https://en.wikipedia.org/wiki/Starvation_(computer_science)
    - formerly_proven 6y ago
      "Task does not receive resources to advance within the required time interval" is exactly starvation. A "task" here is responding to a request. The required time interval is the timeout.
      - squeaky-clean 6y ago
        ctrl+f'd "time interval", zero matches. The first linked source in Wikipedia, "Modern operating systems" even says a FIFO prevents starvation, "Starvation can be avoided by using a first-come, first-serve, resource allocation policy. In due course of time, any given process will eventually become the oldest and thus get the needed resource"
      - Joker_vD 6y ago
        "Starvation" is being denied the necessary resource indefinitely. What you describe is "soft real-time failure".
        formerly_proven 6y ago
        I think these are equivalent when applicable, because starving a task long enough of a resource for either the resource to expire or the task's deadline to expire has the same effect as starving the task of the resource indefinitely: task fails to progress.
        Joker_vD 6y ago
        That's called "firm real-time failure". Again, the standard definitions of starvation/liveness only use "eventually" and "never" without mentioning actual physical time IIRC, and the real-time systems use a whole another bunch of concepts to deal with actual, physical time. And it doesn't really fail to progress if it's cancelled on deadline expiration, it progresses to a non-successful completion. If it instead hanged indefinitely, yes, that'd be a problem.
- matheusmoreira 6y ago
  Because any function can use as much time as it wants before returning. Event loops process events by calling functions which must return in order to allow the loop to make progress. Complex handlers will stall the loop and increase latency. The recent innovations in asynchronous programming involve ways to signal to the programming language that the current task is blocked, thereby allowing the underlying loop to move on to the next event. While this improves the efficiency of the asynchronous processing system, programmers still need to use these features correctly just like coroutines.
BiteCode_dev 6y ago
In production systems, you don't use coroutines alone. You have several processes, some may contain several threads, some thread may contain an event loop. In fact, it's already the case with WSGI setups: you may have async nginx on front, multi-process gunicorn behind, and a thread pool for you db connections. And asyncio already provides API for sending tasks to process pools and threads and getting them back as async Future as a convenience. It's turtles all the way down. Bottom line, async, threads or processes are not silver bullets, and you should use the right tool for the right job. Just like when Go users realized that spawning goroutines for everything actually didn't work at scale, throwing awaits everywhere is not a magical solution. Having asyncio at your disposal make some tasks easier, like websockets, crawling your network, etc. It opened the door for new architectures, and maybe new features (live settings, task queues that feeds the results back to you, etc.). But it's been oversold, as with anything new and shiny.
- 3pt14159 6y ago
  Async is perfect for things like bulk uploading to DynamoDb while you munge through input files or save metadata to postgres. But for 97% of the use cases it's easier to use multiple processes and a normal worker queue.
zzzeek 6y ago
can attest to this fact, I've isolated this issue in Openstack components once or twice where we identified CPU-bound tasks blocking other greenlets leading to new MySQL database connections rejecting the client, because there's an initial auth phase that has a very short 10 second timeout while it waits for the client's response to a security challenge.
worldmerge 6y ago
Would it be possible to implement async to push the functions to run on the GPU and bypass the CPU? Like build out a queuing system in shaders and all of your functions into shaders and just pass data to the shader functions as uniforms?
luckycharms810 6y ago
This is a great point! If you use an established framework like tornado / twisted - they have idiomatic ways to deal with these. For e.g. If you have a loop - that can potentially starve the rest of your service when the iterable is too long - go ahead and throw a `yield gen.moment` in there. `yield gen.sleep` can also be incredibly handy. Once you realize that you are just living in a world of partially executed functions - you can always have your coroutine phone home to the event loop and ask if anyone else needs some CPU. Of-course for this to work you need lumpiness in your workload, if the CPU is just always starved this is more likely a problem of under-allocation of compute.
miguelgrinberg 6y ago
Hi, author here. I believe I did discuss this, and in fact, I list the same a) and b) conditions you mention using slightly different words. Quoting the article: "To benefit from the async style, an application needs to have tasks that are often blocked by I/O and don't have too much CPU work." So I think we agree.

sltkr 6y ago

> ... you are going to get starvation as async does not have a fairness mechanism - timesharing is co-operative and based on FIFO in all the implementations I've seen. If it's FIFO based, why would you get starvation?

euiq 6y ago
What would the CPU-heavy code await on?
calpaterson 6y ago
FIFO systems can have starvation when dealing with time bound things like web requests - slow request handlers can cause other (waiting) requests to timeout. In a thread-per-request or process-per-request model where you run say 4 * cpu workers the operating system scheduler is able to interrupt stuff that's hogging the CPU but that isn't the case in a typical 1 * cpu worker async model. When there's little cpu load this is not a real threat but when there is material load you can easily run into problems.
- sltkr 6y ago
  I see; it sounds like latency would become very unpredictable under that model. I can imagine that's a serious problem in practice, but it's not typically what's meant by starvation. https://en.wikipedia.org/wiki/Starvation_(computer_science) https://en.wikipedia.org/wiki/Starvation_(computer_science)
  - formerly_proven 6y ago
    "Task does not receive resources to advance within the required time interval" is exactly starvation. A "task" here is responding to a request. The required time interval is the timeout.
    - squeaky-clean 6y ago
      ctrl+f'd "time interval", zero matches. The first linked source in Wikipedia, "Modern operating systems" even says a FIFO prevents starvation, "Starvation can be avoided by using a first-come, first-serve, resource allocation policy. In due course of time, any given process will eventually become the oldest and thus get the needed resource"
    - Joker_vD 6y ago
      "Starvation" is being denied the necessary resource indefinitely. What you describe is "soft real-time failure".
      - formerly_proven 6y ago
        I think these are equivalent when applicable, because starving a task long enough of a resource for either the resource to expire or the task's deadline to expire has the same effect as starving the task of the resource indefinitely: task fails to progress.
        Joker_vD 6y ago
        That's called "firm real-time failure". Again, the standard definitions of starvation/liveness only use "eventually" and "never" without mentioning actual physical time IIRC, and the real-time systems use a whole another bunch of concepts to deal with actual, physical time. And it doesn't really fail to progress if it's cancelled on deadline expiration, it progresses to a non-successful completion. If it instead hanged indefinitely, yes, that'd be a problem.
matheusmoreira 6y ago
Because any function can use as much time as it wants before returning. Event loops process events by calling functions which must return in order to allow the loop to make progress. Complex handlers will stall the loop and increase latency. The recent innovations in asynchronous programming involve ways to signal to the programming language that the current task is blocked, thereby allowing the underlying loop to move on to the next event. While this improves the efficiency of the asynchronous processing system, programmers still need to use these features correctly just like coroutines.

BiteCode_dev 6y ago

In production systems, you don't use coroutines alone. You have several processes, some may contain several threads, some thread may contain an event loop. In fact, it's already the case with WSGI setups: you may have async nginx on front, multi-process gunicorn behind, and a thread pool for you db connections. And asyncio already provides API for sending tasks to process pools and threads and getting them back as async Future as a convenience. It's turtles all the way down. Bottom line, async, threads or processes are not silver bullets, and you should use the right tool for the right job. Just like when Go users realized that spawning goroutines for everything actually didn't work at scale, throwing awaits everywhere is not a magical solution. Having asyncio at your disposal make some tasks easier, like websockets, crawling your network, etc. It opened the door for new architectures, and maybe new features (live settings, task queues that feeds the results back to you, etc.). But it's been oversold, as with anything new and shiny.

3pt14159 6y ago
Async is perfect for things like bulk uploading to DynamoDb while you munge through input files or save metadata to postgres. But for 97% of the use cases it's easier to use multiple processes and a normal worker queue.

zzzeek 6y ago

can attest to this fact, I've isolated this issue in Openstack components once or twice where we identified CPU-bound tasks blocking other greenlets leading to new MySQL database connections rejecting the client, because there's an initial auth phase that has a very short 10 second timeout while it waits for the client's response to a security challenge.

worldmerge 6y ago

Would it be possible to implement async to push the functions to run on the GPU and bypass the CPU? Like build out a queuing system in shaders and all of your functions into shaders and just pass data to the shader functions as uniforms?

luckycharms810 6y ago

This is a great point! If you use an established framework like tornado / twisted - they have idiomatic ways to deal with these. For e.g. If you have a loop - that can potentially starve the rest of your service when the iterable is too long - go ahead and throw a `yield gen.moment` in there. `yield gen.sleep` can also be incredibly handy. Once you realize that you are just living in a world of partially executed functions - you can always have your coroutine phone home to the event loop and ask if anyone else needs some CPU. Of-course for this to work you need lumpiness in your workload, if the CPU is just always starved this is more likely a problem of under-allocation of compute.

miguelgrinberg 6y ago

Hi, author here. I believe I did discuss this, and in fact, I list the same a) and b) conditions you mention using slightly different words. Quoting the article: "To benefit from the async style, an application needs to have tasks that are often blocked by I/O and don't have too much CPU work." So I think we agree.