3 ms·

If the author or someone cares to share I’m curious about the decision not to build this on asyncio

Bogdanp 8y ago
Here are my reasons: * I am not a fan of asyncio's API (`ensure_future`), docs (though I understand these two are improving in 3.8) and of the fact that it is incompatible[1] with much of the existing ecosystem of libraries (eg. the SQLAlchemy ORM or any existing libraries that talk to... anything over the network (postgres, redis, memcached, etc.) which need to be written from the ground up in asyncio style or rewritten so their protocol parsing logic is completely separate from the IO that they do (not a bad thing, mind you, but taking battle-tested libraries and rewriting them from the ground up is not the greatest idea)), tools and even builtins (queue.Queue vs asyncio.Queue). * I don't buy into the claimed benefits of explicitly marking yield points: any sufficiently complex code will have enough yield points (`async with`, `async for`, `await`) that reasoning about concurrency in the program isn't going to be any easier than if it had implicit yield points. * I think ergonomics matter for developer happiness and programming with asyncio is not ergonomic in the slightest. * File IO is blocking. Put all of those together and you end up with a system where you can't use any of the popular, battle-tested, libraries out there, where your code is more verbose for minimal gain and where seemingly benign things (reading a file, using a `for` instead of an `async for`) may greatly impact the performance of your program all for a small increase in throughput (assuming it does increase!). [1]: http://journal.stuffwithstuff.com/2015/02/01/what-color-is-your-function/ http://journal.stuffwithstuff.com/2015/02/01/what-color-is-y...
- thomasahle 8y ago
  1) This is fair. 2) Explicit yield points certainly save you from most of the classic threading errors, and having to use special maps and datastructures everywhere. 3) Other than tying into (1) I guess this is a matter of taste. 4) Supposedly this refers to the `open` function? This can be solved using the `aiofiles` module, but again I suppose this ties into (1).
  - nicolaslem 8y ago
    > Explicit yield points certainly save you from most of the classic threading errors You know what also saves you from most classic threading errors? The GIL, and yet people writing multi-threaded code in Python still use locks and thread-safe datastructures, because most errors is not enough. Explicit yield points when the developer is forced to yield does not bring much more safety than executing normal Python bytecode.
  - Bogdanp 8y ago
    Yes, a lot of it boils down to the ecosystem split for me. > 2) Explicit yield points certainly save you from most of the classic threading errors Can you provide an example of such a threading error? At most I've heard people claim that asyncio removes the need for mutexes[1]. I think that may be true at the local level, but as soon as your function needs to synchronize with other async functions in the system that ceases to be the case. At any rate, that's not enough of a win to justify the split in my opinion. > having to use special maps and datastructures everywhere. The GIL already protects you from corrupting builtin datastructures. Neither the GIL nor asyncio protect you from corrupting your data (the need for some form of synchronization), however. > 4) Supposedly this refers to the `open` function? This can be solved using the `aiofiles` module, but again I suppose this ties into (1). And most operations on opened files (like `read()`) as well as any OS bindings provided by the `os` module. [1]: Clearly, asyncio itself still thinks they're useful: https://docs.python.org/3/library/asyncio-sync.html#asyncio.Lock https://docs.python.org/3/library/asyncio-sync.html#asyncio....
  - shrimpx 8y ago
    Wrt (2): Personally I find coroutine programming to be unintuitive and often lead to unreadable code, where the control flow is hard to reason about. Additionally, it introduces the honestly idiotic programming task of profiling to find chunks of code that "don't yield enough" and then breaking them up to yield more. The possibility is always there that some engineer on your team will commit some nasty blocking API handler that stalls your endpoint. Threads don't have any of these problems. Conceptually they are sequential code. You just have to clearly state what the communication model is between threads, and use a good one. The actor model works really well.
- no_wizard 8y ago
  My only counterpoint to this is two fold. I know writing asyncio code is newer and therefore less established and that will certainly crop more bugs for less experienced programmers I’ve witnesses this myself. With that said my 2 points are as follows: 1. Asyncio has comapbility for running threadsade code (asyncio.Lock, loop.run_thread_safe etc) which can take those blocking functions and run them without having to give up all the benefits of awaitable code. In my albeit somewhat brief testing SqlAlqchemy worked okay in this context (I know peewee does its my personal ORM of choice) I happen to agree however that fileIO should’ve been the first thing they implemented as async as it’s the biggest gain instead of leaving it up to the community to re implemented In fact thinking about it it’s the explicit use of Futures that allow me to often bridge sync code like an ORM into asyncio without too much trouble IMO 2. Given that, did you test any of your io heavy code against an asyncio version? I’m just curious. I’ve been seeing personally huge bumps in performance by switching our production apps from Django to Tornado
- zzzeek 8y ago
  > I don't buy into the claimed benefits of explicitly marking yield points: any sufficiently complex code will have enough yield points (`async with`, `async for`, `await`) that reasoning about concurrency in the program isn't going to be any easier than if it had implicit yield points. agree. Another point where programmers seem to not be realizing this and I can only assume they need to gain more experience to see this.
BandaAria 8y ago
The author doesn’t want to learn new language features. Worry not, this framework will be forgotten in a week due to this.

Here are my reasons: * I am not a fan of asyncio's API (`ensure_future`), docs (though I understand these two are improving in 3.8) and of the fact that it is incompatible[1] with much of the existing ecosystem of libraries (eg. the SQLAlchemy ORM or any existing libraries that talk to... anything over the network (postgres, redis, memcached, etc.) which need to be written from the ground up in asyncio style or rewritten so their protocol parsing logic is completely separate from the IO that they do (not a bad thing, mind you, but taking battle-tested libraries and rewriting them from the ground up is not the greatest idea)), tools and even builtins (queue.Queue vs asyncio.Queue). * I don't buy into the claimed benefits of explicitly marking yield points: any sufficiently complex code will have enough yield points (`async with`, `async for`, `await`) that reasoning about concurrency in the program isn't going to be any easier than if it had implicit yield points. * I think ergonomics matter for developer happiness and programming with asyncio is not ergonomic in the slightest. * File IO is blocking. Put all of those together and you end up with a system where you can't use any of the popular, battle-tested, libraries out there, where your code is more verbose for minimal gain and where seemingly benign things (reading a file, using a `for` instead of an `async for`) may greatly impact the performance of your program all for a small increase in throughput (assuming it does increase!). [1]: http://journal.stuffwithstuff.com/2015/02/01/what-color-is-your-function/ http://journal.stuffwithstuff.com/2015/02/01/what-color-is-y...

thomasahle 8y ago
1) This is fair. 2) Explicit yield points certainly save you from most of the classic threading errors, and having to use special maps and datastructures everywhere. 3) Other than tying into (1) I guess this is a matter of taste. 4) Supposedly this refers to the `open` function? This can be solved using the `aiofiles` module, but again I suppose this ties into (1).
- nicolaslem 8y ago
  > Explicit yield points certainly save you from most of the classic threading errors You know what also saves you from most classic threading errors? The GIL, and yet people writing multi-threaded code in Python still use locks and thread-safe datastructures, because most errors is not enough. Explicit yield points when the developer is forced to yield does not bring much more safety than executing normal Python bytecode.
- Bogdanp 8y ago
  Yes, a lot of it boils down to the ecosystem split for me. > 2) Explicit yield points certainly save you from most of the classic threading errors Can you provide an example of such a threading error? At most I've heard people claim that asyncio removes the need for mutexes[1]. I think that may be true at the local level, but as soon as your function needs to synchronize with other async functions in the system that ceases to be the case. At any rate, that's not enough of a win to justify the split in my opinion. > having to use special maps and datastructures everywhere. The GIL already protects you from corrupting builtin datastructures. Neither the GIL nor asyncio protect you from corrupting your data (the need for some form of synchronization), however. > 4) Supposedly this refers to the `open` function? This can be solved using the `aiofiles` module, but again I suppose this ties into (1). And most operations on opened files (like `read()`) as well as any OS bindings provided by the `os` module. [1]: Clearly, asyncio itself still thinks they're useful: https://docs.python.org/3/library/asyncio-sync.html#asyncio.Lock https://docs.python.org/3/library/asyncio-sync.html#asyncio....
- shrimpx 8y ago
  Wrt (2): Personally I find coroutine programming to be unintuitive and often lead to unreadable code, where the control flow is hard to reason about. Additionally, it introduces the honestly idiotic programming task of profiling to find chunks of code that "don't yield enough" and then breaking them up to yield more. The possibility is always there that some engineer on your team will commit some nasty blocking API handler that stalls your endpoint. Threads don't have any of these problems. Conceptually they are sequential code. You just have to clearly state what the communication model is between threads, and use a good one. The actor model works really well.
no_wizard 8y ago
My only counterpoint to this is two fold. I know writing asyncio code is newer and therefore less established and that will certainly crop more bugs for less experienced programmers I’ve witnesses this myself. With that said my 2 points are as follows: 1. Asyncio has comapbility for running threadsade code (asyncio.Lock, loop.run_thread_safe etc) which can take those blocking functions and run them without having to give up all the benefits of awaitable code. In my albeit somewhat brief testing SqlAlqchemy worked okay in this context (I know peewee does its my personal ORM of choice) I happen to agree however that fileIO should’ve been the first thing they implemented as async as it’s the biggest gain instead of leaving it up to the community to re implemented In fact thinking about it it’s the explicit use of Futures that allow me to often bridge sync code like an ORM into asyncio without too much trouble IMO 2. Given that, did you test any of your io heavy code against an asyncio version? I’m just curious. I’ve been seeing personally huge bumps in performance by switching our production apps from Django to Tornado
zzzeek 8y ago
> I don't buy into the claimed benefits of explicitly marking yield points: any sufficiently complex code will have enough yield points (`async with`, `async for`, `await`) that reasoning about concurrency in the program isn't going to be any easier than if it had implicit yield points. agree. Another point where programmers seem to not be realizing this and I can only assume they need to gain more experience to see this.

BandaAria 8y ago

The author doesn’t want to learn new language features. Worry not, this framework will be forgotten in a week due to this.