7 ms·

Show HN: HTTP/2 Python-Asyncio Web Microframework

173 points by pgjones 8y ago

dharness 8y ago
Why did you choose to host this on gitlab over github?
- some_account 8y ago
  Weird question isn't it? We should be happy there are several great places to host code :)
  - pknopf 8y ago
    Who says that OP isn't happy there is competition? I am. I'm also curious on the reasoning.
- cpburns2009 8y ago
  Because everything doesn't need to be hosted on GitHub?
  - pknopf 8y ago
    I don't think that was running through the project author's mind as a factor. It is interesting to just know the reasoning. Nobody is bashing competition. It's just that GitHub has more mindshare and as such, contributors/forks are more likely, and there are more 3rd party integrations. Again, competition is good for everyone, but let's not bash someone who is just curious why a project author would come to choose GitLab.
    - luka-birsa 8y ago
      It might be that the author likes the fact that GitLab offers free private projects, which might be useful for his other work. Or likes the interface over GitHub. Or simply likes going against mainstream. Full disclosure: I use GitLab since 2015.
- pgjones 8y ago
  I like the open source nature of gitlab, and its feature set. There is a mirror on github as well, mostly as people tend to look on github first.
mcintyre1994 8y ago
This looks really nice, and Flask compatibility is great. One question, in the asyncio docs (really nice BTW, I hadn't tried that out yet and instantly grasped your example) you mention the common pitfall of `await awaitable.attribute` with missing brackets. In the Migration from Flask docs you give some examples where you need await like `await request.data` and `await request.get_json()` - do these need brackets in the same way or is `request` special? Same deal with `test_client` straight after that. BTW, do all routes have to be async here - even your quickstart that just returns 'hello'? One other thing - since you require Python 3.6 anyway it'd probably make sense to use `pipenv` instead of `venv` as your recommended install, it'd probably make your docs simpler.
- orf 8y ago
  For the common pitfall question, when you do `await expression` python will resolve `expression` and then await it. When you do `await request.json()`, python calls `request.json()` and then awaits the return value (which will return json, at some point). The same thing with `await request.data`. You can easily avoid this pitfall by writing this code: data = await resp.json() print(data['attribute']) instead of this: print((await resp.json())['attribute']) TBH I find that section of the docs more confusing than helpful, it's not an issue you'd generally run into IMO.
- pgjones 8y ago
  `request` itself isn't an awaitable, whereas the `data` property and `get_json` method both return awaitables (https://gitlab.com/pgjones/quart/blob/master/quart/wrappers/request.py#L142 https://gitlab.com/pgjones/quart/blob/master/quart/wrappers/...). I think these are probably things that become second nature through usage. The routes don't have to be async as Quart will wrap them in a coroutine anyway. However I'm not sure there is any advantage to not adding the async keyword. Thanks, I need to learn how pipenv works.
  - bb88 8y ago
    Pipenv is not bad. I switched to it in about 30 minutes. To start a new project: pipenv --three To add packages: pipenv install <package> pipenv install -d <development packages> pipenv install -r requirements.txt Then check in your Pipfile and Pipfile.lock. When someone makes a fresh checkout, they cd to where the Pipfile and Pipfile.lock are and run: pipenv install When you're ready to update your dependencies do the following: rm Pipfile.lock pipenv update (-d to load development packages) To get a shell, cd to where the Pipfile of your project is: pipenv shell To run something in the virtualenv when the virtualenv is not activated: pipenv run <command>
linsomniac 8y ago
I started dabbling with Quart a few months ago and it looks really promising. But I haven't developed any apps with it. My plan is to try building an app using it instead of Flask, I like Flask but I think this has some benefits in supporting the current future including websockets.
friend-monoid 8y ago
Cool! Might try this for some smaller project. It's a shame that asyncio is so slow though, this framework isn't going to win any speed awards I guess.
- pgjones 8y ago
  It isn't compared with other languages, however Quart sits well in the Python ecosystem, especially with uvloop. I wrote some of this up https://hackernoon.com/3x-faster-than-flask-8e89bfbe8e4f https://hackernoon.com/3x-faster-than-flask-8e89bfbe8e4f
sandGorgon 8y ago
could you add sqlalchemy integration docs as well ? because that is where I get stuck when thinking about async. How does psycogreen work with you asyncio apis ?
- dfee 8y ago
  The sqlalchemy orm and asyncio are incompatible. There are packages like aiopg that allow for sqlalchemy to be used as a functional sql layer. If you want to use the orm parts of sqlalchemy, the idea is to use thread executors and handle detaching of objects if you return them to event-loop code.
  - sandGorgon 8y ago
    Well, then how should we do database access ? Because without that part sorted out the rest of the framework is unusable. Incidentally, my personal opinion is that is the reason why we don't have massive adoption of asyncio despite lots of frameworks (like apistar). Nodejs for example has every database driver as asynchronous. It is probably worth the effort to make a asyncio compatible dB layer.
mixmastamyk 8y ago
Isn't this what sanic was supposed to be? https://github.com/channelcat/sanic https://github.com/channelcat/sanic
- abuckenheimer 8y ago
  +1 been using sanic for a couple years, it's an awesome easily grokable micro framework. Sanic is flask-like in a lot of places where a quick perusal of the quart code makes it look like they go an extra mile on the flask scale, but maybe take up some complexity in that process. Up to the consumer on which trade off they pick. I tend to think that since there is no WSGI equivalent for asyncio (something people are thinking about https://github.com/channelcat/sanic/issues/761 https://github.com/channelcat/sanic/issues/761) you want a slightly different model than flask anyway.
  - je42 8y ago
    there is ASGI https://github.com/django/asgiref/blob/master/specs/asgi.rst https://github.com/django/asgiref/blob/master/specs/asgi.rst but i am not sure how broadly as a common spec this has been accepted.
- ergo14 8y ago
  Unlike flask and OP's solution sanic seems to have a saner API without global request object.
- pgjones 8y ago
  Sanic is Flask-like, whereas Quart is the Flask-API (some Flask extensions work with Quart https://pgjones.gitlab.io/quart/flask_extensions.html#supported-extensions https://pgjones.gitlab.io/quart/flask_extensions.html#suppor...). The key part I wanted to highlight here though is that Quart serves HTTP/2, which I think sets it apart from most of the other Python frameworks. (I know Twisted also does this https://medium.com/python-pandemonium/how-to-serve-http-2-using-python-5e5bbd1e7ff1 https://medium.com/python-pandemonium/how-to-serve-http-2-us...)
cmcginty 8y ago
The author was on a recent episode of Talk Python. https://talkpython.fm/episodes/show/147/quart-flask-but-3x-faster https://talkpython.fm/episodes/show/147/quart-flask-but-3x-f...
gamesbrainiac 8y ago
I think this is all well and good, especially the compatibility with Flask. However, the biggest issue is the data access layer. Most flask apps will use an ORM like SQLAlchemy(SA) to create their data access layer. SA unfortunately, does not have a asynchronous version (its quite complex as it is). Therefore, I think it would require quite a lot of work, in order to actually get a standard flask app to work with quart. However, if you've built your data access layer directly on psycopg, then I think you're good to go.
- pgjones 8y ago
  Have you any experience with GINO (https://github.com/fantix/gino https://github.com/fantix/gino) or Peewee-Async (https://github.com/05bit/peewee-async https://github.com/05bit/peewee-async)? These look to be feasible alternatives.
  - fantix 8y ago
    Working on a Quart extension for GINO now.
- theptip 8y ago
  An interesting deep dive on why this is the case (from 2015), in case anyone wants to know more: http://techspot.zzzeek.org/2015/02/15/asynchronous-python-and-databases/ http://techspot.zzzeek.org/2015/02/15/asynchronous-python-an...
  - tomchristie 8y ago
    One thing I've never been able to reconcile is Mike Bayer's article there, set against Yury Selivanov's reported results for `asyncpg`, which at least as they're presented do indicate a significant difference in throughput. https://github.com/MagicStack/asyncpg https://github.com/MagicStack/asyncpg They're both incredibly experienced developers, and I don't have any good steer on the discrepancy. Would love to see some independent benchmarking on a typical use case to get a clearer picture on how valuable (or not) asyncio is for high-throughput database operations.
tomchristie 8y ago
Neat, good to see more `asyncio` frameworks coming along. Python's going to have a bit of an awkward time with two completely different sets of ecosystem for threaded vs. asyncio approaches, but it's necessary progress. One thing I'd be really keen to see is asyncio frameworks starting to consider adopting ASGI as a common interface. Each of quart, sanic, aiohttp currently all have their own gunicorn worker classes, http parsing, and none share the same common interface for handling the request/response interface between server and application. It's a really high barrier to new asyncio frameworks, and it means we're not able to get shared middleware, such as WSGI's whitenoise or Werkzeug debugger, or the increased robustness that shared server implementations would tend to result in. Would be interested to know what OP's position on this is?
- Fukkaudeku 8y ago
  > Python's going to have a bit of an awkward time with two completely different sets of ecosystem for threaded vs. asyncio approaches, but it's necessary progress. It's threaded vs async/await (and even then projects like https://github.com/dabeaz/curio https://github.com/dabeaz/curio bridge that gap really well), rather than threaded vs asyncio. asyncio is just one (really really poor) implementation of async/await coroutines on Python. > One thing I'd be really keen to see is asyncio frameworks starting to consider adopting ASGI as a common interface. From what I've seen of the ASGI spec, it makes it incredibly easy (like most asyncio stuff) to DoS yourself with lack of backpressure. You get your callback called with data, and like all callback systems you can't exactly not get called.
  - tomchristie 8y ago
    > It's threaded vs async/await Fair enough, sure. > lack of backpressure You’ll get new calls into the application on new requests, yes. Request bodies are pulled tho. You can perfectly well ensure that server implementations properly handle flow control, and if necessary also have a configurable number of maximum concurrent requests. Either way, those sorts of concerns would be far better addressed by having server implementations against a common interface, than by framework authors having to handle (and continually re-implement) the nitty gritty details of high/low watermarks and pausing/resuming transports.
cup-of-tea 8y ago
I've seen a few libraries like this now. Are they supposed to supersede the non asyncio versions? Is there any reason to still use flask? Will they be maintained like forks?

173 points by pgjones 8y ago

dharness 8y ago

Why did you choose to host this on gitlab over github?

some_account 8y ago
Weird question isn't it? We should be happy there are several great places to host code :)
- pknopf 8y ago
  Who says that OP isn't happy there is competition? I am. I'm also curious on the reasoning.
cpburns2009 8y ago
Because everything doesn't need to be hosted on GitHub?
- pknopf 8y ago
  I don't think that was running through the project author's mind as a factor. It is interesting to just know the reasoning. Nobody is bashing competition. It's just that GitHub has more mindshare and as such, contributors/forks are more likely, and there are more 3rd party integrations. Again, competition is good for everyone, but let's not bash someone who is just curious why a project author would come to choose GitLab.
  - luka-birsa 8y ago
    It might be that the author likes the fact that GitLab offers free private projects, which might be useful for his other work. Or likes the interface over GitHub. Or simply likes going against mainstream. Full disclosure: I use GitLab since 2015.
pgjones 8y ago
I like the open source nature of gitlab, and its feature set. There is a mirror on github as well, mostly as people tend to look on github first.

mcintyre1994 8y ago

This looks really nice, and Flask compatibility is great. One question, in the asyncio docs (really nice BTW, I hadn't tried that out yet and instantly grasped your example) you mention the common pitfall of `await awaitable.attribute` with missing brackets. In the Migration from Flask docs you give some examples where you need await like `await request.data` and `await request.get_json()` - do these need brackets in the same way or is `request` special? Same deal with `test_client` straight after that. BTW, do all routes have to be async here - even your quickstart that just returns 'hello'? One other thing - since you require Python 3.6 anyway it'd probably make sense to use `pipenv` instead of `venv` as your recommended install, it'd probably make your docs simpler.

orf 8y ago
For the common pitfall question, when you do `await expression` python will resolve `expression` and then await it. When you do `await request.json()`, python calls `request.json()` and then awaits the return value (which will return json, at some point). The same thing with `await request.data`. You can easily avoid this pitfall by writing this code: data = await resp.json() print(data['attribute']) instead of this: print((await resp.json())['attribute']) TBH I find that section of the docs more confusing than helpful, it's not an issue you'd generally run into IMO.
pgjones 8y ago
`request` itself isn't an awaitable, whereas the `data` property and `get_json` method both return awaitables (https://gitlab.com/pgjones/quart/blob/master/quart/wrappers/request.py#L142 https://gitlab.com/pgjones/quart/blob/master/quart/wrappers/...). I think these are probably things that become second nature through usage. The routes don't have to be async as Quart will wrap them in a coroutine anyway. However I'm not sure there is any advantage to not adding the async keyword. Thanks, I need to learn how pipenv works.
- bb88 8y ago
  Pipenv is not bad. I switched to it in about 30 minutes. To start a new project: pipenv --three To add packages: pipenv install <package> pipenv install -d <development packages> pipenv install -r requirements.txt Then check in your Pipfile and Pipfile.lock. When someone makes a fresh checkout, they cd to where the Pipfile and Pipfile.lock are and run: pipenv install When you're ready to update your dependencies do the following: rm Pipfile.lock pipenv update (-d to load development packages) To get a shell, cd to where the Pipfile of your project is: pipenv shell To run something in the virtualenv when the virtualenv is not activated: pipenv run <command>

linsomniac 8y ago

I started dabbling with Quart a few months ago and it looks really promising. But I haven't developed any apps with it. My plan is to try building an app using it instead of Flask, I like Flask but I think this has some benefits in supporting the current future including websockets.

friend-monoid 8y ago

Cool! Might try this for some smaller project. It's a shame that asyncio is so slow though, this framework isn't going to win any speed awards I guess.

pgjones 8y ago
It isn't compared with other languages, however Quart sits well in the Python ecosystem, especially with uvloop. I wrote some of this up https://hackernoon.com/3x-faster-than-flask-8e89bfbe8e4f https://hackernoon.com/3x-faster-than-flask-8e89bfbe8e4f

sandGorgon 8y ago

could you add sqlalchemy integration docs as well ? because that is where I get stuck when thinking about async. How does psycogreen work with you asyncio apis ?

dfee 8y ago
The sqlalchemy orm and asyncio are incompatible. There are packages like aiopg that allow for sqlalchemy to be used as a functional sql layer. If you want to use the orm parts of sqlalchemy, the idea is to use thread executors and handle detaching of objects if you return them to event-loop code.
- sandGorgon 8y ago
  Well, then how should we do database access ? Because without that part sorted out the rest of the framework is unusable. Incidentally, my personal opinion is that is the reason why we don't have massive adoption of asyncio despite lots of frameworks (like apistar). Nodejs for example has every database driver as asynchronous. It is probably worth the effort to make a asyncio compatible dB layer.

mixmastamyk 8y ago

Isn't this what sanic was supposed to be? https://github.com/channelcat/sanic https://github.com/channelcat/sanic

abuckenheimer 8y ago
+1 been using sanic for a couple years, it's an awesome easily grokable micro framework. Sanic is flask-like in a lot of places where a quick perusal of the quart code makes it look like they go an extra mile on the flask scale, but maybe take up some complexity in that process. Up to the consumer on which trade off they pick. I tend to think that since there is no WSGI equivalent for asyncio (something people are thinking about https://github.com/channelcat/sanic/issues/761 https://github.com/channelcat/sanic/issues/761) you want a slightly different model than flask anyway.
- je42 8y ago
  there is ASGI https://github.com/django/asgiref/blob/master/specs/asgi.rst https://github.com/django/asgiref/blob/master/specs/asgi.rst but i am not sure how broadly as a common spec this has been accepted.
ergo14 8y ago
Unlike flask and OP's solution sanic seems to have a saner API without global request object.
pgjones 8y ago
Sanic is Flask-like, whereas Quart is the Flask-API (some Flask extensions work with Quart https://pgjones.gitlab.io/quart/flask_extensions.html#supported-extensions https://pgjones.gitlab.io/quart/flask_extensions.html#suppor...). The key part I wanted to highlight here though is that Quart serves HTTP/2, which I think sets it apart from most of the other Python frameworks. (I know Twisted also does this https://medium.com/python-pandemonium/how-to-serve-http-2-using-python-5e5bbd1e7ff1 https://medium.com/python-pandemonium/how-to-serve-http-2-us...)

cmcginty 8y ago

The author was on a recent episode of Talk Python. https://talkpython.fm/episodes/show/147/quart-flask-but-3x-faster https://talkpython.fm/episodes/show/147/quart-flask-but-3x-f...

gamesbrainiac 8y ago

I think this is all well and good, especially the compatibility with Flask. However, the biggest issue is the data access layer. Most flask apps will use an ORM like SQLAlchemy(SA) to create their data access layer. SA unfortunately, does not have a asynchronous version (its quite complex as it is). Therefore, I think it would require quite a lot of work, in order to actually get a standard flask app to work with quart. However, if you've built your data access layer directly on psycopg, then I think you're good to go.

pgjones 8y ago
Have you any experience with GINO (https://github.com/fantix/gino https://github.com/fantix/gino) or Peewee-Async (https://github.com/05bit/peewee-async https://github.com/05bit/peewee-async)? These look to be feasible alternatives.
- fantix 8y ago
  Working on a Quart extension for GINO now.
theptip 8y ago
An interesting deep dive on why this is the case (from 2015), in case anyone wants to know more: http://techspot.zzzeek.org/2015/02/15/asynchronous-python-and-databases/ http://techspot.zzzeek.org/2015/02/15/asynchronous-python-an...
- tomchristie 8y ago
  One thing I've never been able to reconcile is Mike Bayer's article there, set against Yury Selivanov's reported results for `asyncpg`, which at least as they're presented do indicate a significant difference in throughput. https://github.com/MagicStack/asyncpg https://github.com/MagicStack/asyncpg They're both incredibly experienced developers, and I don't have any good steer on the discrepancy. Would love to see some independent benchmarking on a typical use case to get a clearer picture on how valuable (or not) asyncio is for high-throughput database operations.

tomchristie 8y ago

Neat, good to see more `asyncio` frameworks coming along. Python's going to have a bit of an awkward time with two completely different sets of ecosystem for threaded vs. asyncio approaches, but it's necessary progress. One thing I'd be really keen to see is asyncio frameworks starting to consider adopting ASGI as a common interface. Each of quart, sanic, aiohttp currently all have their own gunicorn worker classes, http parsing, and none share the same common interface for handling the request/response interface between server and application. It's a really high barrier to new asyncio frameworks, and it means we're not able to get shared middleware, such as WSGI's whitenoise or Werkzeug debugger, or the increased robustness that shared server implementations would tend to result in. Would be interested to know what OP's position on this is?

Fukkaudeku 8y ago
> Python's going to have a bit of an awkward time with two completely different sets of ecosystem for threaded vs. asyncio approaches, but it's necessary progress. It's threaded vs async/await (and even then projects like https://github.com/dabeaz/curio https://github.com/dabeaz/curio bridge that gap really well), rather than threaded vs asyncio. asyncio is just one (really really poor) implementation of async/await coroutines on Python. > One thing I'd be really keen to see is asyncio frameworks starting to consider adopting ASGI as a common interface. From what I've seen of the ASGI spec, it makes it incredibly easy (like most asyncio stuff) to DoS yourself with lack of backpressure. You get your callback called with data, and like all callback systems you can't exactly not get called.
- tomchristie 8y ago
  > It's threaded vs async/await Fair enough, sure. > lack of backpressure You’ll get new calls into the application on new requests, yes. Request bodies are pulled tho. You can perfectly well ensure that server implementations properly handle flow control, and if necessary also have a configurable number of maximum concurrent requests. Either way, those sorts of concerns would be far better addressed by having server implementations against a common interface, than by framework authors having to handle (and continually re-implement) the nitty gritty details of high/low watermarks and pausing/resuming transports.

cup-of-tea 8y ago

I've seen a few libraries like this now. Are they supposed to supersede the non asyncio versions? Is there any reason to still use flask? Will they be maintained like forks?