17 ms·
Python Requests: HTTP for Humans
Requests is an ISC Licensed HTTP library, written in Python, for human beings.
Requests allow you to send HEAD, GET, POST, PUT, PATCH, and DELETE HTTP requests. You can add headers, form data, multipart files, and parameters with simple Python dictionaries, and access the response data in the same way. It’s powered by urllib2, but it does all the hard work and crazy hacks for you.
- leonderijke 15y agoI'm using it in a project I'm working on. It's really easy to use!
- pchivers 15y agoLink to previous discussion on HN: http://news.ycombinator.com/item?id=2882301 http://news.ycombinator.com/item?id=2882301
- d0vs 15y agoUnfortunately, no Python 3 support
- kenneth_reitz 15y agoI can assure you that it's directly around the corner :)
- sneak 15y agoWhy is that unfortunate? Why would anyone run python > 2.7?
- baq 15y agowhy not?
- briancurtin 15y agoDo you have a good reason why they shouldn't? Apparently 113,798 people found a reason to download the 3.2.2 Windows installers (32 and 64 bit) from python.org through only the first 10 days of the month of October. http://www.python.org/webstats/usage_201110.html http://www.python.org/webstats/usage_201110.html (doesn't include mention of the 3.2 source tarball downloads, which are under 5,000, most likely since people get that from their distro anyway)
- megaman821 15y agoThis looks so nice. Is there any upcoming support for HTTP caching? I know it is not the easiest thing to do because you will probably need to make several cache backends (file, db, memcached, redis, etc.) to keep everyone happy. Also I couldn't tell if there is keep-alive support, is there?
- kenneth_reitz 15y agoKeep-alive is 100% supported in the upcoming v0.7.0 release, along with async i/o :) Caching: I think that will be best be served by a hook/helper module, but we'll see.
- samps 15y agoCool! Is there anything I can look at to see how async I/O will work? Does all the async support depend on gevent monkey patching (as requests/async.py would suggest)?
- kenneth_reitz 15y agoIt's unreleased, and therefore completely undocumented at the moment. But here's the API: https://github.com/kennethreitz/requests/issues/98 https://github.com/kennethreitz/requests/issues/98
- st3fan 15y agoMmm async. Does that also mean that it will have better support for timeouts? I'm not interested in socket timeouts, more in a 'finish this in N seconds or just give up' thing. For that reason I am using pycurl now for some stuff where I had previously used the requests library. Which has a really really horrible API :-/
- simonw 15y agoThe most interesting thing about caching is that you also get support for 304/Etags/Last-Modified - which is amazingly useful for any application that might hit the same URL more than once. I'd vote for that logic in requests core, with caching as a key/value interface (ship an in-memory or file-based default, but make it easy to hook in memcached/redis/etc ala Django).
- dreamdu5t 15y agoWhat's with the "for human beings" phrase? Is that a way of saying the software is easy?
- RegEx 15y ago"Requests is an ISC Licensed HTTP library, written in Python, for human beings. Most existing Python modules for sending HTTP requests are extremely verbose and cumbersome. Python’s builtin urllib2 module provides most of the HTTP capabilities you should need, but the api is thoroughly broken. It requires an enormous amount of work (even method overrides) to perform the simplest of tasks."
- jabwork 15y agoIt would appear so. Based off of testimonials and briefly looking at the API I'd say this is primarily aimed at a simple API + fewer "gotchas" E.g., urllib/urllib2 both support urlopen(url, [data]) if `data` is supplied the request is executed as a POST, if not it is executed as a GET. Issuing a GET w/ a querystring requires mutating the `url` and omitting `data`. This surprised me at first, and probably a number of others
- scott_s 15y agoSome APIs are appropriate for programmers to target. Others are more appropriate for compilers or higher-abstraction libraries to target.
- JonnieCache 15y agoIt generally means it has an API which lends itself to writing readable, intuitive code.
- bascule 15y agoI just started working on a similar library for Ruby: https://github.com/tarcieri/http https://github.com/tarcieri/http
- voyvf 15y agoI love this API, and it looks like the dev version is getting even better. r.iter_content is exactly what I needed! :D
- old-gregg 15y agoAbsolutely my favourite recent development in the Python world. I won't be mistaken if I say that many of us have built half-arse versions of this on top of urllib(2) multiple times in the past. :-) The support for keep-alive is particularly welcome, huge kudos to the author. Question to users of other languages: do equivalents exist for PHP, Perl, C# or Java? Ruby folks rave about restclient, but is there anything else?
- bad_user 15y agoIn the FAQ it says that support for keep-alive is on the way. I do have a half-baked implementation on urllib2 that I hate. This is really cool.
- kenneth_reitz 15y agoKeep-alive has first-class support in the soon-to-be-released version!
- Kliment 15y agoIs it just me or does the FAQ break the back button?
- davnola 15y agoFor Ruby there's httparty, https://github.com/jnunemaker/httparty https://github.com/jnunemaker/httparty, curb, and a couple of others that escape me right now.
- bascule 15y agoHere's the new one I'm working on: https://github.com/tarcieri/http https://github.com/tarcieri/http httparty does a few of the same tricks, but I still find its API a bit obtrusive. At least it automatically parses JSON! (my library does this too, if an appropriate library is loaded) In addition to a humane interface, it will soon be backed by a Ragel-generated HTTP parser: https://github.com/tarcieri/http/tree/master/parser https://github.com/tarcieri/http/tree/master/parser
- jacoblyles 15y agoFinally, an easy way to make POST requests from Python
- mp3tricord 15y agoLooks like Python got something similar to Perl's LWP https://metacpan.org/module/LWP https://metacpan.org/module/LWP
- draegtun 15y agoI think LWP is more equivalent to Python's urllib2 (ie. both big beasts!). So this i believe is closer to what LWP::UserAgent (https://metacpan.org/module/LWP::UserAgent https://metacpan.org/module/LWP::UserAgent) does. Also check out Mojo::UserAgent (https://metacpan.org/module/Mojo::UserAgent https://metacpan.org/module/Mojo::UserAgent) for a nice(er) alternative. And don't forget that Perl now comes a nice lightweight alternative as standard: https://metacpan.org/module/HTTP::Tiny https://metacpan.org/module/HTTP::Tiny
- mp3tricord 15y agoYes gp, LWP::UserAgent specifically. Never knew about HTTP::Tiny thanks.. I'll check it out.
- DanielRibeiro 15y agoDiscussion from the two month old submission: http://news.ycombinator.com/item?id=2882301 http://news.ycombinator.com/item?id=2882301
- vidar 15y agoWhat is the timeline for v0.7? Desperately need the asynchronous stuff!
- kenneth_reitz 15y agoAs soon as I can push it out. Aiming for Friday (if not sooner)!
- vidar 15y agoThat is great news! I really appreciate your work, thanks!
- jrydberg 15y agoIt would be nice with something similar for Twisted.
- efsavage 15y agoI don't mean to badmouth Requests or Python in general, it certainly looks like an improvement, but can anyone enlighten me as to why an API that wraps an HTTP request with concise methods is #1 on HN, twice? This looks more like a good answer to SO question than "news". This looks like all the other wrapper methods you spend 5 minutes doing, once, when you start a new codebase?
- mcav 15y agoIt's #1 because people vote it up; people vote it up because it interests them or they'd like to see more of it. That's the pattern for every link here. We may agree or disagree with the consensus of the masses, but the masses make the front page.
- tomp 15y agoThe design of this library reminds me of Apple's design... While a lot of Python is more like Google, or Microsoft - they just don't get it. (Admittedly, parts of Python are very well-designed, but I still see a lot of room for improvement...)
- toyg 15y agoMaybe because doing the same wrappers over and over again, as short as they might be, is boring?
- lukeschlather 15y agoThe philosophy of Python is that those wrapper methods shouldn't be necessary. I'm glad when language quirks and libraries get linked, it's good for discussion. (And more informative than a good third of the stuff that gets linked.)
- drgath 15y agoI came here with the same question. Considering I come from PHP & JavaScript where the native APIs are awful, wrapping is a necessary step for sanity. Interesting that Python didn't have this until now.
- cjus 15y agoGiven that there are a great deal of service APIs in the wild this is going to lower the barrier of entry for many less experienced developers looking to prototype their ideas.
- seanmcq 15y agoI note that you call gevent.monkey.patch_all() in async. I'm still fairly new to gevent and I'm curious if others consider this good style for a library?
- MostAwesomeDude 15y agoIf you're using Twisted (or Tornado, ugh) then there are builtin HTTP clients which are async in all the right ways and provide similar high-level APIs for making requests. If you're using gevent, then this is what you want. If you're using eventlet, you should switch to gevent or learn Twisted. So yeah, this is suboptimal, but not the worst thing in the world. It should really be documented though.
- artlogic 15y agoForgive me if I've missed this, but does this library have support for multipart/form-data POSTing? That's a capability in the python library that's sorely missing - especially in 2.X.
- d0mine 15y agoyes, it does
- pbreit 15y agoIf Python is going to include urllib or urllib2 in the standard distro, why not make it good? Http is foundational to (web) programming. I am shocked at how poor the standard httpclient implementations are in most environments.
- abraham 15y agoThere is also an addon for Requests that lets you make OAuth requests: https://github.com/maraujop/requests-oauth-hook https://github.com/maraujop/requests-oauth-hook
- 16s 15y agoHow does this compare (feature wise) to the poster library? Are they about the same? I've used poster some, but not requests... I'll have to try it now. Just curious how it compares to poster.
- d0mine 15y ago`requests` used `poster` at least in the past (if I recall correctly)
- kenneth_reitz 15y agoThe current stable release actually has poster inside (and has for a while), but it's being taken out soon.
- josephturnip 15y agoI love Requests, but it would be really nice to have some Python libraries with cert-verified SSL support. I've hacked SSL support on top of urllib2 for my projects at work (https://github.com/josephturnerjr/urllib2.VerifiedHTTPS https://github.com/josephturnerjr/urllib2.VerifiedHTTPS), but it'd be really nice if libraries started prioritizing secure communications as a first-order requirement.
- kenneth_reitz 15y agoThis will be in requests very soon. :)
- josephturnip 15y agoThen I will be porting my projects over to Requests very soon :) It's pretty easy to implement, as it already works at the lower (socket) level with the ssl module, it just hasn't made it up the stack into urllib2 (or Requests..yet!).
- nemoniac 15y agoExcellent library but they need to fix this piece of misinformation: "A project that is released as GPL cannot be used in any commercial product without the product itself also being offered as open source." This is false. A project released as GPL can also be released on another (commercial) license.
- kenneth_reitz 15y agoYes, it has to also be offered as open source.
- dannyobrien 15y agoOh, what I would give for a "mechanize for humans" built on this! I love mechanize, but I dream of not having to go back to its one page documentation and the source code every time I want to do something.
- objectified 15y agoAlthough Requests looks very nice, I've wondered why most people wouldn't simply use pycurl - http://pycurl.sourceforge.net/ http://pycurl.sourceforge.net/ ? It's stable and you get all the power that cURL has (which is a lot), and the API is fairly simple - although maybe not as pythonic as the Requests API is. On a side note: last time I checked urllib2 (which actually drives Requests) could not do SSL requests over a proxy, which gave me the feeling that this implementation still isn't very mature.
- kenneth_reitz 15y agoI'm moving off of urllib2 as we speak, to sit directly on top of httplib. One thing at a time :)
- jsilence 15y agoTrying to use the requests module yields: File "/usr/lib/python2.6/cookielib.py", line 38, in <module> from calendar import timegm ImportError: cannot import name timegm for me. On two different machines, one OSX SL with Python 2.6 from MacPorts and the other Ubuntu 10.04.3 LTS with Python 2.6 via aptitude. Can not find any solution for this via Google and its siblings. I'm stuck :( -jsl