11 ms·

Just want to note and curio and trio, two non-stdlib python async libraries, have file io as batteries included. I would also like to point out that asyncio ma

Just want to note and curio and trio, two non-stdlib python async libraries, have file io as batteries included. I would also like to point out that asyncio made a huge mistake here. IO is not limited to interactions with the network buffer. I don't know how something called asynio can be released in to stdlib, and not include disk io. The fact that there is no actual async file io on the OS doesn't matter. Dealing with files is a core concept in almost every application, and so pushing this to stdlib and hoping the python ecosystem takes up the slack, or people handbake threaded access in themselves, is a joke. Some of this is a little dated, but a friend of mine wrote this and included my complaints about file io in there somewhere too: https://web.archive.org/web/20171206105117/https://veriny.tf/asyncio-a-dumpster-fire-of-bad-design/ https://web.archive.org/web/20171206105117/https://veriny.tf...

sitkack 6y ago
In my alternate universe David Beazley [1] does a hard fork on Python, separates the runtime from the VM and brings PyPy into the fold. [1] http://www.dabeaz.com/ http://www.dabeaz.com/ https://dabeaz.com/talks.html https://dabeaz.com/talks.html
orf 6y ago
I disagree that dealing with files is a core concept in almost every asyncio application. I would say that the most common instances of file IO is at the start of an asyncio process (parsing config files) rather than in the “hot” asynchronous part. Some problems might require extensive file IO, sure. But most don’t, and something like reading from a database or making a HTTP request is orders of magnitude more common.
- earthboundkid 6y ago
  You’re just writing off being able to write an HTTP file server in Python? That’s crazy.
  - falcor84 6y ago
    The parent never said that. You can of course write a file server, they just said it's not as common, and I agree. From personal experience, even for serving files, I find myself relying a lot more on web services rather than a local disk.
- aprdm 6y ago
  What about something like sqlite?
quietbritishjim 6y ago
The way that the linked thread passionately presents asyncio as very obviously wrong, like it was written by idiots, is a bit of a turn off. The author seems especially offended that asyncio's interface isn't as good as curio (of which trio is a fork): > Then I found curio. And I realised how badly designed asyncio was. That quotation, the rest of the text at that link, and the comment I'm replying to, all hinge on the fact that asyncio was the first async library available. There are two consequences of this: 1. asyncio was used to drive library design forward. Yes it's easy to look at curio (or trio) and say that the design is obviously better than asyncio, but even the author admits they only realised it after they saw curio. This shows how legitimately difficult it is to design an async API. The best way to come up with any design is really to start off with a first attempt and use that (especially by analysing its weaknesses) to come up with a new one. That's what happened here; I don't think curio could ever have been created without asyncio coming first to inspire them, even if it was partly the mistakes that inspired them. Indeed the asyncio API has changed a lot since that post was written, partly learning from its own mistakes directly (e.g. there's almost no reference to the event loop in the API now) and partly learning from the API in curio/trio (e.g. asyncio.run()). 2. asyncio was used to drive async language features forward, many of which are relied upon by curio and trio for the designs referred to in that post. These were ultimately used in asyncio too (as you said, many arguments in that link are now out of date - the main objection is surely "Ignoring the generator API" which is no longer true). I don't think an external library could have pushed language evolution as quickly as a part of the standard library did. For language stability reasons, the standardised version of asyncio always lagged the language features by one Python version. In many ways, the primary job of asyncio was to get something working to allow these two things to be iteratively improved. Missing out file IO is reasonable because it still allows the library to accomplish those goals, and people can still call out to a thread to do it (which needs to have a good API anyway because there's no avoiding it for CPU-bound tasks). In many platforms, that's all that would happen under the hood anyway. ------ A couple of specific responses to the objections in the link (many are simply "that's no longer true" and I'm not going to try and list all those): > Additionally, writing is not async. Is all your data sent when you write? Nope! ... The work around, of course, is to call writer.drain(). Why it doesn't do this automatically, who knows? Sometimes you want to queue up data to be written to a socket and then do something else (in that coroutine), and sometimes to you want to queue up that data and wait until it actually gets written. The only time I've used asyncio in a real-world project, I've wanted the former; waiting would've been a disaster! The obvious way to allow this flexibility is to split up those two operations - and they really are separate operations - into two separate functions. This isn't a "workaround", it's clean design. The alternative is to make it all in one function. That still technically allows both use cases: if you want to just queue up data then you can spawn a new coroutine for doing that with a queue/stream to send it data from other coroutines. But that's causing one of the two API uses to have way more code than the other, whereas having to choose between calling one or two functions is really not a big deal. It also means there's the runtime overhead of two queues, since the internal queue of data to write still really exists, and it's harder to arrange for smaller packets to be coallesced for writing... and so on. > Even the call to enter thread-land is bad: loop.run_in_executor(executor, callable). None is almost always passed as an argument to executor. You almost always don't want to pass None as the executor - I suspect the argument is deliberately first to discourage that! It's pretty normal in a threaded application to have different threads (or thread pools) for different types of work. This API lets you use different executors representing different types of thread or thread pool. Even if you only need one "processing" thread (/pool) for now, it's still good hygeine to be explicit about it. > Once you're in thread-land, you're basically SOL if you want to interact with async-land. Now I do concede this one could be a bit simpler. If you async thread is ultimately in charge of everything then the pattern that worked for me was a queue to pass the data to a processing coroutine, which in turn passes the data to the thread: async def process(self): while True: input_data = await self.queue.get() if isinstance(my_data, Exception): raise input_data else: output_data = await self.loop.run_in_executor( self.my_executor, processing_fn, input_data ) yield output_data (To be clear: my_executor is shared between multiple instances of the class with this method, so multiple connections end up sharing the same thread pool for this type of work, but there are other executors for unrelated work.) You can then use an `async for` to loop over the processed data. If you want something more complex then you can use asyncio.run_coroutine_threadsafe. This was introduced in Python 3.5.1, which is probably after that article was written (so that's a case of "no longer true").

sitkack 6y ago

In my alternate universe David Beazley [1] does a hard fork on Python, separates the runtime from the VM and brings PyPy into the fold. [1] http://www.dabeaz.com/ http://www.dabeaz.com/ https://dabeaz.com/talks.html https://dabeaz.com/talks.html

orf 6y ago

I disagree that dealing with files is a core concept in almost every asyncio application. I would say that the most common instances of file IO is at the start of an asyncio process (parsing config files) rather than in the “hot” asynchronous part. Some problems might require extensive file IO, sure. But most don’t, and something like reading from a database or making a HTTP request is orders of magnitude more common.

earthboundkid 6y ago
You’re just writing off being able to write an HTTP file server in Python? That’s crazy.
- falcor84 6y ago
  The parent never said that. You can of course write a file server, they just said it's not as common, and I agree. From personal experience, even for serving files, I find myself relying a lot more on web services rather than a local disk.
aprdm 6y ago
What about something like sqlite?

quietbritishjim 6y ago

The way that the linked thread passionately presents asyncio as very obviously wrong, like it was written by idiots, is a bit of a turn off. The author seems especially offended that asyncio's interface isn't as good as curio (of which trio is a fork): > Then I found curio. And I realised how badly designed asyncio was. That quotation, the rest of the text at that link, and the comment I'm replying to, all hinge on the fact that asyncio was the first async library available. There are two consequences of this: 1. asyncio was used to drive library design forward. Yes it's easy to look at curio (or trio) and say that the design is obviously better than asyncio, but even the author admits they only realised it after they saw curio. This shows how legitimately difficult it is to design an async API. The best way to come up with any design is really to start off with a first attempt and use that (especially by analysing its weaknesses) to come up with a new one. That's what happened here; I don't think curio could ever have been created without asyncio coming first to inspire them, even if it was partly the mistakes that inspired them. Indeed the asyncio API has changed a lot since that post was written, partly learning from its own mistakes directly (e.g. there's almost no reference to the event loop in the API now) and partly learning from the API in curio/trio (e.g. asyncio.run()). 2. asyncio was used to drive async language features forward, many of which are relied upon by curio and trio for the designs referred to in that post. These were ultimately used in asyncio too (as you said, many arguments in that link are now out of date - the main objection is surely "Ignoring the generator API" which is no longer true). I don't think an external library could have pushed language evolution as quickly as a part of the standard library did. For language stability reasons, the standardised version of asyncio always lagged the language features by one Python version. In many ways, the primary job of asyncio was to get something working to allow these two things to be iteratively improved. Missing out file IO is reasonable because it still allows the library to accomplish those goals, and people can still call out to a thread to do it (which needs to have a good API anyway because there's no avoiding it for CPU-bound tasks). In many platforms, that's all that would happen under the hood anyway. ------ A couple of specific responses to the objections in the link (many are simply "that's no longer true" and I'm not going to try and list all those): > Additionally, writing is not async. Is all your data sent when you write? Nope! ... The work around, of course, is to call writer.drain(). Why it doesn't do this automatically, who knows? Sometimes you want to queue up data to be written to a socket and then do something else (in that coroutine), and sometimes to you want to queue up that data and wait until it actually gets written. The only time I've used asyncio in a real-world project, I've wanted the former; waiting would've been a disaster! The obvious way to allow this flexibility is to split up those two operations - and they really are separate operations - into two separate functions. This isn't a "workaround", it's clean design. The alternative is to make it all in one function. That still technically allows both use cases: if you want to just queue up data then you can spawn a new coroutine for doing that with a queue/stream to send it data from other coroutines. But that's causing one of the two API uses to have way more code than the other, whereas having to choose between calling one or two functions is really not a big deal. It also means there's the runtime overhead of two queues, since the internal queue of data to write still really exists, and it's harder to arrange for smaller packets to be coallesced for writing... and so on. > Even the call to enter thread-land is bad: loop.run_in_executor(executor, callable). None is almost always passed as an argument to executor. You almost always don't want to pass None as the executor - I suspect the argument is deliberately first to discourage that! It's pretty normal in a threaded application to have different threads (or thread pools) for different types of work. This API lets you use different executors representing different types of thread or thread pool. Even if you only need one "processing" thread (/pool) for now, it's still good hygeine to be explicit about it. > Once you're in thread-land, you're basically SOL if you want to interact with async-land. Now I do concede this one could be a bit simpler. If you async thread is ultimately in charge of everything then the pattern that worked for me was a queue to pass the data to a processing coroutine, which in turn passes the data to the thread: async def process(self): while True: input_data = await self.queue.get() if isinstance(my_data, Exception): raise input_data else: output_data = await self.loop.run_in_executor( self.my_executor, processing_fn, input_data ) yield output_data (To be clear: my_executor is shared between multiple instances of the class with this method, so multiple connections end up sharing the same thread pool for this type of work, but there are other executors for unrelated work.) You can then use an `async for` to loop over the processed data. If you want something more complex then you can use asyncio.run_coroutine_threadsafe. This was introduced in Python 3.5.1, which is probably after that article was written (so that's a case of "no longer true").