4 ms·

That's exactly the opposite of what it says really: if you need to do CPU stuff, then you can do that, it just won't be using asyncio. So it doesn't really infe

That's exactly the opposite of what it says really: if you need to do CPU stuff, then you can do that, it just won't be using asyncio. So it doesn't really infect your whole program. You could easily have, say, a thread to do all your asyncio stuff, another to do some CPU intenstive stuff (so long as it blocks the GIL) and yet another to do some blocking I/O e.g. interacting with a database with its own blocking APIs. In that case, you could spawn separate threads and exchange messages between them like usual. Where this falls down a little is if you want asyncio to infer your whole program - or more accurately, if you want to use it in all of the task-management code at the top level of your app. In that case, you would use asyncio's thread API [1] although it's a bit clunky for when you want to maintain state bound to a specific worker thread (like SQLite database objects). [1] https://docs.python.org/3/library/asyncio-task.html#running-in-threads https://docs.python.org/3/library/asyncio-task.html#running-...

brrrrrm 4y ago
I think the issue is more like this: 1. You jump into a huge codebase and find a very useful function you want to use in your code (it uses asyncio) 2. Your code doesn't use asyncio, so you start converting functions to be async (or else you can't use the `await` keyword) 3. It turns out your function is called by a bunch of different users, some of which do not use the asyncio runner 4. You end up having several meetings with upstream teams asking them to use the asyncio runner 5. A single team says "no" 6. Scrap the project and just rewrite that original function you found synchronously
- quietbritishjim 4y ago
  I guess I was focusing more on the CPU task problem mentioned at the top. But you're talking about blocking IO - more specifically, blocking IO that you want to convert to async (not just run in a worker thread using the thread-to-async API I mentioned). What you've written is true, but "infects" a smaller proportion of the code than you'd expect, in my experience. In fact, the better organised the program is, the less needs to change when converting to async. For example, reading from a connection your top-level async function would often be something like this: async def read_and_handle_messages(connection): while True: next_message_bytes = await connection.get_next_message() next_message_parsed = my_parser.parse(next_message_bytes) my_message_handler(next_message_parsed) All the application-specific code is in the parser and the message handler but neither of those are async. (If you need to send a response, your message handler could post a message to an async queue that is read by a separate writer task.) In your hypothetical scenario, you'd maybe write two a wrapper functions, one for blocking and one for async, which both read the message, parse it and handle it. But that only needs to be two versions of a three line function while all the subtantial code is completely shared. I find that there's very little actual async code in async programs that I write, even with no blocking IO historical baggage. Admittedly, that's partly because I organise programs to reach that goal, but it actually works out for the best because all the async-task lifecycle management ends up in one place, which makes following overall program flow particularly easy.
- gpderetta 4y ago
  Depending on the details you can often call an async function from a sync function, by just running the event loop, it is just a few lines of code. The problem is that the event loop is not reentrant [1], so if your sync function is being called from an async function, things do not work. Why would you ever do this you might think? Well, asyncio might be an implementation detail of whatever environment you are using. Jupyiter or ipython for example. [1] there are... hacks to make this sort of work of course.
  - nineplay 4y ago
    > there are... hacks to make this sort of work of course. The python motto.
    - gpderetta 4y ago
      Monkey patch like there is no tomorrow (so you do not have to maintain it)!
  - quietbritishjim 4y ago
    Another possibility, if it's a matter of passing callback parameter that must be sync where you wish it could be an async function, is that the callback can be the `put_nowait()` method of an async queue. You can then have a background async task that pulls from that queue and does async stuff. This works well if the callback is just there to notify you about a result (which is the most common case in fairness) but doesn't really work if the callback is meant to synchronously determine and return a result that is then used within the function that's calling the callback.
    - gpderetta 4y ago
      Yes, been there done that :). You can spawn a whole background thread running its own event loop, and post the async function with run-coroutine_threadsafe, the wait on the returned concurrent.future. But it feels wrong having to spawn a thread and event loop just for that.
dekhn 4y ago
No, the point is that asyncio is viral, polluting large existing systems with its paradigm. Modern frameworks should be orthogonal and compositional. Even orthogonality (when you use multiple frameworks, each framework solves a problem along a different "axis" and does not interact with any of the other axes) is optional. Compositionality means I should be able to combine several systems without one affecting the other unnecessarily. async does not fulfill that requirement.

brrrrrm 4y ago

I think the issue is more like this: 1. You jump into a huge codebase and find a very useful function you want to use in your code (it uses asyncio) 2. Your code doesn't use asyncio, so you start converting functions to be async (or else you can't use the `await` keyword) 3. It turns out your function is called by a bunch of different users, some of which do not use the asyncio runner 4. You end up having several meetings with upstream teams asking them to use the asyncio runner 5. A single team says "no" 6. Scrap the project and just rewrite that original function you found synchronously

quietbritishjim 4y ago
I guess I was focusing more on the CPU task problem mentioned at the top. But you're talking about blocking IO - more specifically, blocking IO that you want to convert to async (not just run in a worker thread using the thread-to-async API I mentioned). What you've written is true, but "infects" a smaller proportion of the code than you'd expect, in my experience. In fact, the better organised the program is, the less needs to change when converting to async. For example, reading from a connection your top-level async function would often be something like this: async def read_and_handle_messages(connection): while True: next_message_bytes = await connection.get_next_message() next_message_parsed = my_parser.parse(next_message_bytes) my_message_handler(next_message_parsed) All the application-specific code is in the parser and the message handler but neither of those are async. (If you need to send a response, your message handler could post a message to an async queue that is read by a separate writer task.) In your hypothetical scenario, you'd maybe write two a wrapper functions, one for blocking and one for async, which both read the message, parse it and handle it. But that only needs to be two versions of a three line function while all the subtantial code is completely shared. I find that there's very little actual async code in async programs that I write, even with no blocking IO historical baggage. Admittedly, that's partly because I organise programs to reach that goal, but it actually works out for the best because all the async-task lifecycle management ends up in one place, which makes following overall program flow particularly easy.
gpderetta 4y ago
Depending on the details you can often call an async function from a sync function, by just running the event loop, it is just a few lines of code. The problem is that the event loop is not reentrant [1], so if your sync function is being called from an async function, things do not work. Why would you ever do this you might think? Well, asyncio might be an implementation detail of whatever environment you are using. Jupyiter or ipython for example. [1] there are... hacks to make this sort of work of course.
- nineplay 4y ago
  > there are... hacks to make this sort of work of course. The python motto.
  - gpderetta 4y ago
    Monkey patch like there is no tomorrow (so you do not have to maintain it)!
- quietbritishjim 4y ago
  Another possibility, if it's a matter of passing callback parameter that must be sync where you wish it could be an async function, is that the callback can be the `put_nowait()` method of an async queue. You can then have a background async task that pulls from that queue and does async stuff. This works well if the callback is just there to notify you about a result (which is the most common case in fairness) but doesn't really work if the callback is meant to synchronously determine and return a result that is then used within the function that's calling the callback.
  - gpderetta 4y ago
    Yes, been there done that :). You can spawn a whole background thread running its own event loop, and post the async function with run-coroutine_threadsafe, the wait on the returned concurrent.future. But it feels wrong having to spawn a thread and event loop just for that.

dekhn 4y ago

No, the point is that asyncio is viral, polluting large existing systems with its paradigm. Modern frameworks should be orthogonal and compositional. Even orthogonality (when you use multiple frameworks, each framework solves a problem along a different "axis" and does not interact with any of the other axes) is optional. Compositionality means I should be able to combine several systems without one affecting the other unnecessarily. async does not fulfill that requirement.