4 ms·

So refreshing to see a new Python web framework that isn't trying to hoist more event-based non-blocking IO on us.

by zzzeek 8y ago

coolsunglasses 8y ago
foist
deleted 8y ago
[deleted]
scribu 8y ago
For those who are looking for web frameworks that use async I/O: https://github.com/huge-success/sanic https://github.com/huge-success/sanic https://gitlab.com/pgjones/quart https://gitlab.com/pgjones/quart
- devbug 8y ago
  I don't get the hate. I can reason about asyncio code without fuss and the throughput/latency impact is worth it. My only pain point is the derth of libraries. There are few if any palatable libraries that provide the powerful abstractions I'm used too. Things like ORMs.
  - rsyring 8y ago
    Grandparents concern about async is documented here: http://techspot.zzzeek.org/2015/02/15/asynchronous-python-and-databases/ http://techspot.zzzeek.org/2015/02/15/asynchronous-python-an...
dfee 8y ago
I get that SA ORM is unfit to run on the eventloop thread, but I’m increasingly unclear if you hold your position that asyncio is bad solely because it makes SA ORM less competitive - and less of a ecosystem standard. The alternative being just using SA Core and something like aiopg, using asyncpg, or using SA in a threadpool (unfortunately, a technique that’s involved enough that the recipe for using SA isn’t a 5 min “gotcha”, re: flask). I like SA. I’ve used it for years, and you’ve helped me on the forums, so I’m appreciative for the work you’ve done. I just don’t understand why you’ve written off an entire style of programming.
- zzzeek 8y ago
  > but I’m increasingly unclear if you hold your position that asyncio is bad solely because it makes SA ORM less competitive - and less of a ecosystem standard. It's not that the ORM is less "competitive" because I am certainly able to make an async version of SQLAlchemy, it's just I don't have the time / interest / resources to do so, and as I've gone through the effort to write about in detail, it is a bad idea. If you want to pay me a salary to make an async SQLAlchemy I can totally do that. It would be a worthwhile effort in that it would make people happy. But also it would be a wasteful effort in terms of advancing the field of Python database programming. That's the decision I'm facing and it would be a lot easier if folks would just realize asyncio isn't getting them much overall. It's like, suppose I make bikes. But everyone wants to get to work on pogo sticks instead. Everyone who uses pogo sticks is terminally late to work, they're getting injured all the time, they are miserable, but someone told them "hey pogo sticks are FASTER than bikes!", people were just bored with bikes so much that they didn't even bother testing this assertion, and now everyone just has discussions like "oh well I have this shiny new pogo stick, so I only broke three ribs this week! hooray!" and people have just forgotten that there is nothing wrong with bikes at all and they are in fact a lot better than pogo sticks for the task of getting to work every day (you might find pogo sticks to be more interesting and fun in the moment, but as a real solution to the problem, they are not any better than bikes and probably worse). Someone who makes bikes tries to point all of this out. That puts you in the position of saying, "all your arguments about bikes being better than pogo sticks for getting to work, who cares. you are just concerned pogo sticks are a threat to your bike sales". Well, yes, they are. But the bigger issue is that it is ridiculous everyone is killing themselves trying to get to work on pogo sticks. Somehow I feel that should not be lost. > I just don’t understand why you’ve written off an entire style of programming. I've written and tweeted about this a lot so feel free to point out the reasons that might not be clear. To recap http://techspot.zzzeek.org/2015/02/15/asynchronous-python-and-databases/ http://techspot.zzzeek.org/2015/02/15/asynchronous-python-an...: asyncio is essentailly intended to provide an interface around the concept of non-blocking IO. The relational database use case, for the vast majority of uses (e.g. CRUD), gains nothing from using non-blocking IO (most databases don't even offer a non-blocking API anyway) and only complicates the application, introduces subtle bugs, and does not improve performance. The other argument made about asyncio is that replacing the OS'es task scheduler with a cooperative one defined by when IO happens to occur is "safer", because context switches are no longer implicit. The second section of the post above makes many arguments why this is not the case and especially not for modern server-side application design which is inherently multi-process.
hguhghuff 8y ago
Async programming becomes very appealing once you’ve done enough of it to really sink in. At first though, learning async programming tastes like eating a garden snail.
- shrimpx 8y ago
  Do you have any high level wisdom/takeaways you could share?
  - hguhghuff 8y ago
    Just that my experience learning async programming really was a pain cause the concepts are so alien compared to synchronous programming. Eventually however the bell in your head rings and it all becomes second nature. At that point you have the ability to do a bunch of things in parallel in an intuitive and robust way. Also async is good for solving certain categories of problem that are harder in synch, like starting a sub process and reading and immediately responding to stderr, whislt also doing other things, without the issues that comes with multithreading. What async programming really gives you is event driven programming - the ability just to set up functions to run when certain things happen, as opposed to traditional ... do this step one then do this step two until program finishes. Again quite foreign if you havent done much event driven programming but it’s really powerful. It is true that certain libraries need to be rewritten to live well in the async world, but that’s ok I think ... it’s not like the 2/3 issue.... you don’t have to go there if you don’t want to. The other thing that is not immediately obvious when learning async is that once you know what you’re doing, in many cases sync and async style can be naturally interspersed, and while that might sound like a recipe for complexity and pain, in fact it isn’t, because as mentioned earlier, async actually isn’t that hard or complex once you grasp the mental model, and interspersing both just works pretty well. You won’t know how to do that in a simple manner though till you’ve been through the grind of making it hard for yourself because at first you’ll assume that async is much harder than it really is and you’ll be looking for complexity and indeed coding complexity. I certainly understand the initial distaste that people might have for async because it’s so foreign. It’s made harder in Python because the approach has changed so there’s multiple ways to do the same thing, like “yield from” and “await” which are effectively the same thing. Different ways to do the same thing in a conceptually new and hard idea space makes the hard thing about learning async programming even harder. BUT, like most things with programming, once you’ve learned it, what at first looked really hard is in fact fairly straightforward, with just a few different rules to follow and that’s pretty much it. My advice to any python programmer is to invest the time and practice a lot and work hard to master async. It’s not going away, and it will constantly annoy you that this async thing is everywhere but you don’t know it well so you’ll feel excluded and want to disparage it. Get past that and turn it into one of your strengths and a power tool. It’ll just taste like garden snails while you’re still in that alien learning zone. Remember that it takes an expert programmer with superior knowledge to know how to do things in the simplest way.
hguhghuff 8y ago
I admire zzzeek as one of the greatest contributors to python and a programmer of the highest order. I hope one day he’ll become a real fan of async as it would be good to have all that skill directed into the async space, which frankly needs him.
- zzzeek 8y ago
  When you need to talk to 8000 database connections simultaneously, many of which are taking many seconds to respond to you, we'll talk :)

So refreshing to see a new Python web framework that isn't trying to hoist more event-based non-blocking IO on us.

by zzzeek 8y ago

So refreshing to see a new Python web framework that isn't trying to hoist more event-based non-blocking IO on us.

coolsunglasses 8y ago

foist

deleted 8y ago

[deleted]

scribu 8y ago

For those who are looking for web frameworks that use async I/O: https://github.com/huge-success/sanic https://github.com/huge-success/sanic https://gitlab.com/pgjones/quart https://gitlab.com/pgjones/quart

devbug 8y ago
I don't get the hate. I can reason about asyncio code without fuss and the throughput/latency impact is worth it. My only pain point is the derth of libraries. There are few if any palatable libraries that provide the powerful abstractions I'm used too. Things like ORMs.
- rsyring 8y ago
  Grandparents concern about async is documented here: http://techspot.zzzeek.org/2015/02/15/asynchronous-python-and-databases/ http://techspot.zzzeek.org/2015/02/15/asynchronous-python-an...

dfee 8y ago

I get that SA ORM is unfit to run on the eventloop thread, but I’m increasingly unclear if you hold your position that asyncio is bad solely because it makes SA ORM less competitive - and less of a ecosystem standard. The alternative being just using SA Core and something like aiopg, using asyncpg, or using SA in a threadpool (unfortunately, a technique that’s involved enough that the recipe for using SA isn’t a 5 min “gotcha”, re: flask). I like SA. I’ve used it for years, and you’ve helped me on the forums, so I’m appreciative for the work you’ve done. I just don’t understand why you’ve written off an entire style of programming.

zzzeek 8y ago
> but I’m increasingly unclear if you hold your position that asyncio is bad solely because it makes SA ORM less competitive - and less of a ecosystem standard. It's not that the ORM is less "competitive" because I am certainly able to make an async version of SQLAlchemy, it's just I don't have the time / interest / resources to do so, and as I've gone through the effort to write about in detail, it is a bad idea. If you want to pay me a salary to make an async SQLAlchemy I can totally do that. It would be a worthwhile effort in that it would make people happy. But also it would be a wasteful effort in terms of advancing the field of Python database programming. That's the decision I'm facing and it would be a lot easier if folks would just realize asyncio isn't getting them much overall. It's like, suppose I make bikes. But everyone wants to get to work on pogo sticks instead. Everyone who uses pogo sticks is terminally late to work, they're getting injured all the time, they are miserable, but someone told them "hey pogo sticks are FASTER than bikes!", people were just bored with bikes so much that they didn't even bother testing this assertion, and now everyone just has discussions like "oh well I have this shiny new pogo stick, so I only broke three ribs this week! hooray!" and people have just forgotten that there is nothing wrong with bikes at all and they are in fact a lot better than pogo sticks for the task of getting to work every day (you might find pogo sticks to be more interesting and fun in the moment, but as a real solution to the problem, they are not any better than bikes and probably worse). Someone who makes bikes tries to point all of this out. That puts you in the position of saying, "all your arguments about bikes being better than pogo sticks for getting to work, who cares. you are just concerned pogo sticks are a threat to your bike sales". Well, yes, they are. But the bigger issue is that it is ridiculous everyone is killing themselves trying to get to work on pogo sticks. Somehow I feel that should not be lost. > I just don’t understand why you’ve written off an entire style of programming. I've written and tweeted about this a lot so feel free to point out the reasons that might not be clear. To recap http://techspot.zzzeek.org/2015/02/15/asynchronous-python-and-databases/ http://techspot.zzzeek.org/2015/02/15/asynchronous-python-an...: asyncio is essentailly intended to provide an interface around the concept of non-blocking IO. The relational database use case, for the vast majority of uses (e.g. CRUD), gains nothing from using non-blocking IO (most databases don't even offer a non-blocking API anyway) and only complicates the application, introduces subtle bugs, and does not improve performance. The other argument made about asyncio is that replacing the OS'es task scheduler with a cooperative one defined by when IO happens to occur is "safer", because context switches are no longer implicit. The second section of the post above makes many arguments why this is not the case and especially not for modern server-side application design which is inherently multi-process.

hguhghuff 8y ago

Async programming becomes very appealing once you’ve done enough of it to really sink in. At first though, learning async programming tastes like eating a garden snail.

shrimpx 8y ago
Do you have any high level wisdom/takeaways you could share?
- hguhghuff 8y ago
  Just that my experience learning async programming really was a pain cause the concepts are so alien compared to synchronous programming. Eventually however the bell in your head rings and it all becomes second nature. At that point you have the ability to do a bunch of things in parallel in an intuitive and robust way. Also async is good for solving certain categories of problem that are harder in synch, like starting a sub process and reading and immediately responding to stderr, whislt also doing other things, without the issues that comes with multithreading. What async programming really gives you is event driven programming - the ability just to set up functions to run when certain things happen, as opposed to traditional ... do this step one then do this step two until program finishes. Again quite foreign if you havent done much event driven programming but it’s really powerful. It is true that certain libraries need to be rewritten to live well in the async world, but that’s ok I think ... it’s not like the 2/3 issue.... you don’t have to go there if you don’t want to. The other thing that is not immediately obvious when learning async is that once you know what you’re doing, in many cases sync and async style can be naturally interspersed, and while that might sound like a recipe for complexity and pain, in fact it isn’t, because as mentioned earlier, async actually isn’t that hard or complex once you grasp the mental model, and interspersing both just works pretty well. You won’t know how to do that in a simple manner though till you’ve been through the grind of making it hard for yourself because at first you’ll assume that async is much harder than it really is and you’ll be looking for complexity and indeed coding complexity. I certainly understand the initial distaste that people might have for async because it’s so foreign. It’s made harder in Python because the approach has changed so there’s multiple ways to do the same thing, like “yield from” and “await” which are effectively the same thing. Different ways to do the same thing in a conceptually new and hard idea space makes the hard thing about learning async programming even harder. BUT, like most things with programming, once you’ve learned it, what at first looked really hard is in fact fairly straightforward, with just a few different rules to follow and that’s pretty much it. My advice to any python programmer is to invest the time and practice a lot and work hard to master async. It’s not going away, and it will constantly annoy you that this async thing is everywhere but you don’t know it well so you’ll feel excluded and want to disparage it. Get past that and turn it into one of your strengths and a power tool. It’ll just taste like garden snails while you’re still in that alien learning zone. Remember that it takes an expert programmer with superior knowledge to know how to do things in the simplest way.

hguhghuff 8y ago

I admire zzzeek as one of the greatest contributors to python and a programmer of the highest order. I hope one day he’ll become a real fan of async as it would be good to have all that skill directed into the async space, which frankly needs him.

zzzeek 8y ago
When you need to talk to 8000 database connections simultaneously, many of which are taking many seconds to respond to you, we'll talk :)