13 ms·
Httpx: A next-generation HTTP client for Python
- Congeec 7y agoI have been using httpx 0.95 in production. So far so good.
- daze42 7y agoDo you mean 0.9.5?
- oefrha 7y agoOne of the highlighted features is directly calling into WSGI applications[1], which uses flask as an example. Question: is there any advantage of this over flask’s builtin werkzeug test client? [1] https://www.python-httpx.org/advanced/#calling-into-python-web-apps https://www.python-httpx.org/advanced/#calling-into-python-w... [2] https://flask.palletsprojects.com/en/1.1.x/testing/ https://flask.palletsprojects.com/en/1.1.x/testing/
- grammarxcore 7y agoComparing the code samples, it looks like it's easier to spin up testing with Httpx than Flask, but that's a superficial conclusion. I don't have experience with Httpx so take this with a grain of salt. Based on the comparison, I'm going to play with Httpx for testing the next time I use Flask because the simplicity looks rad. Based on similar experience with other tools, that possibly means that Httpx is great for simple testing but if you need to go deep it's better to use the framework provided. That's an assumption, though, so I'd love to hear more from others.
- oefrha 7y agoFlask’s sample code is doing a bunch of things like spinning up db and stuff and setting up a pytest fixture, so it’s not a fair comparison. You can replace with httpx.Client(app=app) as client: ... in the httpx sample code with with app.test_client() as client: ... for flask’s builtin test client and the rest is basically the same. Now that’s a fair comparison and neither is simpler than the other. I guess one advantage of httpx is that developers might generally be more familiar with the requests response object API than the werkzeug response object API.
- dhruvkar 7y agoAfter a brief glance, I'm not sure what this offers beyond requests?
- daze42 7y agoProper async support was the big selling point for me.
- dhruvkar 7y agoah, yes that's attractive.
- mariocesar 7y agoAsync support and http2 without patching
- dhruvkar 7y agothanks.
- Felk 7y agohttpx makes a pretty good first impression on me. The homepage provides examples, key selling points, install instructions, and links to any further reading one could hope for. However, I am missing one thing: The features this offers over the popular requests library do not seem to require httpx to be a competitor, but an extension or fork. There surely must be some major incompatibilities that allowed this library to do something fundamentally different, right?
- kissgyorgy 7y ago> The features this offers over the popular requests library do not seem to require httpx to be a competitor Yes it has to be. Requests is not as great as everyone thinks it is. It's API is simple, sure, but when you need more advanced features (doesn't even support HTTP/2 AFAIK) like timeouts, proper exception handling (which you cannot do in requests)) it actually sucks. httpx is a far superior library already!
- takeda 7y agofrom comments here, two already big ones (for me) is type annotation and async support. Will need to check how it compares with aiohttp which is quite good and also has these.
- ra5 7y agoFinally a client that mixes async support and a “requests-compatible” API (for those times you don’t want to go as low as say an aiohttp or the like). Looking forward to trying it out Edit: maybe the homepage can include a very simple async example as well?
- takeda 7y agoAs an aiohttp user, what do you mean? The API od aiohttp is not low level and is very similar to requests api.
- korijn 7y agoIs that another dead butterfly? Great package though. Love the dual support for async and sync requests.
- bouke 7y agoI immediately wondered the same, and I think not. The upper wings are not at a 90 degree angle to the body, so alive and well. Apart from being a black and white drawing of course.
- daze42 7y agoI've been using httpx 0.9.3 in production now for a couple months. I switched from requests when I realized I needed async support and it has been a dream to use. The only issue I've run into has been with my attempt to reuse the same AsyncClient to make multiple concurrent requests to the same remote host. It looks like this issue may have been fixed in 0.10 or 0.11 so I'll be upgrading soon to check. Also, be sure to check out the other fantastic projects by Encode. https://github.com/encode https://github.com/encode I stumbled upon httpx after using Starlette and Uvicorn for one of our microservices and and been pleasantly surprised by how easy to set up and use they are.
- vajenetehais 7y agoI'm currently using aiohttp. There is features I don't find in butterfly docs like : limit connection, limit connection per host or per time frame. Do you know if those exist in butterfly? I'm already deep in aiohttp and it's not an easy task to learn an async client (at least in my case, but I'm no dev) so if I do switch it would be for more features (retry option on exceptions, limit requests per host and per time frame...). But that's only my opinion.
- hermitdev 7y agoI prototyped moving to aiohttp from requests/multiprocessing. The speedup was amazing. Saw something like a 60% reduction in runtime for our use case. Only reason the code hasn't gone live: we currently use requests_negotiate_sspi for authentication, which sadly isn't supported by aiohttp. Not sure if httpx supports it. Looks like it might. Planning on giving it a shot next week.
- tomchristie 7y agoWe don’t have any third party packages for that authentication style yet, but we do have an API for supporting custom auth flows... https://www.python-httpx.org/advanced/#customizing-authentication https://www.python-httpx.org/advanced/#customizing-authentic... If you’re interested in trying httpx you’d be very welcome to raise an issue an issue related to NTLM/Negotiate authentication - it’d be really helpful for us to work through that and figure out if our auth API is sufficient for implementing that, or if there’s anything we’re missing.
- bbmario 7y agoI know this is unrelated, but probably a lot of Pythonistas here and I've been wondering: what is the async web framework of choice for you guys today? As for DBs, still SQLAlchemy? What about a prettier (js) alternative?
- rtny4821 7y agoFastAPI - https://github.com/tiangolo/fastapi https://github.com/tiangolo/fastapi
- daze42 7y agoI've been using FastAPI https://github.com/tiangolo/fastapi https://github.com/tiangolo/fastapi which is built on top of Starlette as my main async framework and Uvicorn as my primary web server. Starlette and Uvicorn are both made by Encode https://github.com/encode https://github.com/encode and in my experience they consistently put out quality stuff.
- vpribish 7y agofwiw: I've moved to Trio over asyncio (I did plain old async for a couple years and Trio makes a ton of sense) Quart-trio over Flask (just to get a Trio-friendly flask-a-like server) - plain old aiohttp worked really well too. It takes a bit more roll-your-own work, but you get exactly what you want. peewee over SQLAlchemy (less committed to this change, but peewee has been fine so far and is much more streamlined) I'm mostly just using SQLite. the async version of the ORM looks pretty new, i'm not using it yet.
- workthrowaway 7y agosanic seems nice. also, sqlalchemy is an over-engineered system imo. i only go for it when i have no other choices. otherwise i use a database client directly.
- oefrha 7y ago> async web framework Not enough experience with async to comment. > DB peewee is good enough and more ergonomic compared to SQLA for a lot of use cases. > formatter black. To be clear it often produces truly horrendous code, but at least there’s no arguing and no fussing over options or details.
- pbreit 7y agoI don't understand why this is such a complicated category and that many platforms do not have solid http clients in standard library. On every single project I do, it's just a bunch of posting JSON and getting a response synchronously. Over and over.
- neurostimulant 7y agoPlenty of reason why it's hard to ship in standard library. Here's some off top of my mind: - Should the library includes its own CA store, or use the system's CA store? These kind of library often include their own CA store (since they changes often), and httpx seem to use 3rd party lib to handle that (certifi). This is hard to do in a standard library for variety of reasons (users rarely update their python installation, system CA store is not always available/up to date, etc). - While the http protocol itself is pretty stable, some part of it are still changing overtime. Things like compression types (brotli is gaining traction these days, and we might get a new compression types in the future), new http headers added, etc. Security issue also show up all the time. The user will want tighter release schedule than python's so they can get these stuff sooner. The situation is even worse for users that stuck in a particular version of python for some reason since they now won't have access to these new update ever.
- sciurus 7y agoHTTP/2 was only standardized 4 years ago. HTTP/3 is being actively developed. That's actually not really stable on a language's standard library timespan, IMHO.
- dragonwriter 7y ago> Should the library includes its own CA store, or use the system's CA store? The CA store should be a configurable option, and one of the supported options should be the system CA store. > The user will want tighter release schedule than python's so they can get these stuff sooner. Ruby is moving stdlib to default and bundled gems, which addresses this. There's no reason that “delivered with the interpreter” needs to mean “frozen with the interpreter”.
- yegle 7y ago> Fully type annotated. This is a huge win compare to requests. AFAIKT requests is too flexible (read: easier to misuse) and difficult to add type annotation now. The author of requests gave a horrible type annotation example here [0]. IMO at this time when you evaluate a new Python library before adopting, "having type annotation" should be as important as "having decent unit test coverage". [0]: https://lwn.net/Articles/643399/ https://lwn.net/Articles/643399/
- jordic 7y agoThe huge win against requests is that Https is fully async.. you can download 20 files in parallel without not too much effort. Throughout in python asyncio is amazing, something similar to node or perhaps better... That's the main point.
- westurner 7y agoFWIW, requests3 has "Type-annotations for all public-facing APIs", asyncio, HTTP/2, connection pooling, timeouts, etc https://github.com/kennethreitz/requests3 https://github.com/kennethreitz/requests3
- ehsankia 7y agoRelevant blog and discussion: https://vorpus.org/blog/why-im-not-collaborating-with-kenneth-reitz/ https://vorpus.org/blog/why-im-not-collaborating-with-kennet... https://news.ycombinator.com/item?id=19826680 https://news.ycombinator.com/item?id=19826680
- jordic 7y agoSorry, have you checked the source? Are these features there or only announced? Has requests added a timeout by default finally?
- westurner 7y agoIt looks like requests is now owned by PSF. https://github.com/psf/requests https://github.com/psf/requests But IDK why requests3 wasn't transferred as well, and why issues appear to be disabled on the repo now. The docs reference a timeout arg (that appears to default to the socket default timeout) for connect and/or read https://3.python-requests.org/user/advanced/#timeouts https://3.python-requests.org/user/advanced/#timeouts And the tests reference a timeout argument. If that doesn't work, I wonder how much work it would be to send a PR (instead of just talking s to Ken and not contributing any code)
- tiangolo 7y agoFastAPI creator here... if you use FastAPI, HTTPX would probably be the best match for sending requests, just saying... :D
- keshab 7y agoHey, FastAPI is great. Thank you.
- tiangolo 7y ago:D
- Svenstaro 7y agoWriting FastAPI code as I procrastinate right now!
- tiangolo 7y agoHa! I'm glad it's enjoyable enough as to do it as "procrastination" :p
- bransonf 7y agoSometimes I find it hard to justify all the time I spend on HN. But discovering FastAPI is probably going to compensate for plenty of this time. This is a really awesome library, thanks!
- tiangolo 7y agoThank you! :D
- carapace 7y agohttps://fastapi.tiangolo.com/ https://fastapi.tiangolo.com/ Looks great!
- prox 7y ago
- sheerun 7y agoThere is also quite nice httpx library for Ruby: https://gitlab.com/honeyryderchuck/httpx https://gitlab.com/honeyryderchuck/httpx
- neuland 7y agoLooks like this does not include certifi [0] and loads system certificates by default. This is a breath of fresh air to see, because so many packages want to use their own certs and have a custom system to override it to use system certs. Edit: Well, looks like it does use certifi. But my grumble still stands, I don't understand why does everyone want to mess with your certs. [0] https://pypi.org/project/certifi/ https://pypi.org/project/certifi/
- Twirrim 7y agoIt uses certifi by default, and it's down in the dependencies: https://www.python-httpx.org/advanced/#changing-the-verification-defaults https://www.python-httpx.org/advanced/#changing-the-verifica...
- neuland 7y agoBummer. Luckily, it doesn't invent a new way to override, `SSL_CERT_FILE` is mentioned in the environmental variables.
- greglindahl 7y agoI want to mess with certs in Python so that my web crawler can actually access the whole web. If you don't talk to a wide variety of hosts, you probably haven't noticed that it's broken for 1%.
- dragonwriter 7y agoIf you work in a corporate environment, you would probably notice that systems that insist on bundling their own certs without an easy to activate option of using system cert store are broken. (And even if the library has an easy to use option, if it's easy to not expose it, much software built on the library will still be broken.) People should be empowered to substitute cert stores, but the system store should be the default.
- greglindahl 7y ago
- hannse 7y agoI'm really impressed by the work done by Encode and Tom Christie. IMO it feels like httpx is set to be the go-to http client in the near future. A friend wrote respx https://github.com/lundberg/respx https://github.com/lundberg/respx which is a utility to mock HTTP requests made by httpx. It works similar to the requests mocking library responses https://github.com/getsentry/responses https://github.com/getsentry/responses
- Rapzid 7y agoWhat are the selling points for people already using aiohttp?
- lilydjwg 7y agoMaybe support for trio and HTTP/2? Also I ran into hard-to-debug issues when there were lots of requests in the past. I'll check again with httpx soon.
- julienfr112 7y agoI use intensively requests-cache, that add local caching (in sqlite) for python requests and is configurable (should it cache non 200 response, should it cache POST ...) and it's amazing when scrapping. I didn"t find a similar companion package for httpx.
- ryanmccullagh 7y agoI like async support, but always, in async examples, doesn't the await keyword just block anyways? I like to see the usage of async inside of a main loop, where other things can be processed while waiting for the async response to come in. Further, not all applications have the idea of an event loop, so async may not be needed, but it's useful to have the option. I use async operations to multiplex operations in a queue. The linux scheduler can handle the execution time slices for me, I'm not going to build a scheduler, but the queue controller's role is to accept jobs and handle timeouts and results of each async operation.
- _bohm 7y ago> doesn't the await keyword just block anyways? The await keyword only blocks the execution of the coroutine in which it is used. It releases the event loop so that other coroutines can continue processing while the result of an awaitable is being fetched.
- teruakohatu 7y agoWith all the alleged drama surrounding the requests module and the promised async support, I am pleased there is a drama-free alternative. https://vorpus.org/blog/why-im-not-collaborating-with-kenneth-reitz/ https://vorpus.org/blog/why-im-not-collaborating-with-kennet...
- deleted 7y ago[deleted]
- jsmeaton 7y agoOne thing requests still has going for it is that it’s used under the covers by a lot of client libraries with a dynamic import, allowing you to pass in a session object to control things like connection pooling and custom headers. I do look forward to httpx becoming the new “standard” though. Tom is a great developer, and his ecosystem of tools are going to have a really big impact in python web dev over the coming years.
- bproven 7y agolooks nice, but appears to have a quite a few third party deps that requests doesnt?
- toyg 7y agoRequests simply vendors what it needs, afaik. Which is sub-optimal. At least this gives you visibility into those deps.
- tus88 7y ago> urllib3 - Sync client support. Well that's going to work wonders for async now isn't it?
- Too 7y agoToo bad they carried over the biggest nuisance from requests, having to call raise_for_status() after every single request. One extra line everywhere and another thing that can be forgotten and cause strange errors when you least expect it.
- deleted 7y ago[deleted]
- florimondmanca 7y agoThoughts on this one [0]? :-) [0]: https://github.com/encode/httpx/issues/752 https://github.com/encode/httpx/issues/752
- hypewatch 7y agoI’ve used httpx to test server load issues for our most sensitive system and it works exactly how you’d expect it would. Highly recommend it.
- theon144 7y agoGreat! Sometimes you don't need huge magic libraries with tons of features, but solid, good, plumbing. Will definitely be checking out and potentially replacing requests and aiohttp.
- speedplane 7y agoMy project has a dozen or so libraries, many of which require different and conflicting requests, urllib3, etc., libraries. I have to setup the packages very carefully to ensure the right module gets the right version of its HTTP library and it has generally turned me off to Python HTTP libraries altogether. For my own code, I stick to the basic built-in libraries, regardless of how difficult to use they are.
- mproud 7y agoThat’s a dead butterfly. https://emilydamstra.com/news/please-enough-dead-butterflies/ https://emilydamstra.com/news/please-enough-dead-butterflies...