17 ms·

A Go “clone” of the great and famous Requests library

92 points by agonzalezro 11y ago

gghh 11y ago
forgive my ignorance, is the original library http://python-requests.org http://python-requests.org ?
- ionforce 11y ago
  If true, I don't think it's ignorance. Not everyone is a Pythonista. Exactly how famous is this library?
  - deleted 11y ago
    [deleted]
  - mtrn 11y ago
    The Python standard library is overall very solid. However, it has a few darker corners - and, in the past, HTTP handling was not as easy as it could be. In the age of APIs, the requests library filled this void with a compellingly simple solution and became one of the most popular libraries (with millions of downloads every month [1]). [1] https://pypi.python.org/pypi/requests https://pypi.python.org/pypi/requests
  - coldtea 11y ago
    In the Python world it's as famous as Joda or Hibernate would be for Javaists...
  - sauere 11y ago
    > Exactly how famous is this library? It is the de-facto standard for anything that has to interact with a API, or just HTTP in general. The old way to do these sort of things was the urllib2 module (part of the Python standard library). urllib2 has some design flaws and can be a pain to work with.
  - acdha 11y ago
    Extremely popular - almost every HTTP client library I use has switched to requests over the last couple years. As a proxy, check out the download counts on this tracker using data from the PyPI, where the vast majority of Python packages are installed from: https://python3wos.appspot.com/ https://python3wos.appspot.com/ Looking at the top few packages: * simplejson is the upstream for the json module in stdlib which people install since it gets performance updates first * requests * six: the most common library used to bridge the Python 2/3 transition * virtualenv: near-ubiquitous development tool (it allows you to maintain a separate Python environment for each project to avoid cross-talk, and is also used by popular testing tools like tox which run your tests under a variety of Python versions) * distribute: a few years back, the stdlib setuptools module was forked for a major overhaul. That's since been merged back in but many packages still reference it, particularly in older releases * boto: AWS client library, used by the official awscli tool * pip: Python package installer, now bundled with Python but updates & older versions of Python use the PyPI version
  - Twirrim 11y ago
    About as famous as they come in the python community. It is the perfect example of a library done right. I've yet to meet a developer who have used it that dislike it. It handles all of the low level stuff for you, such as sessions, cookies, gzip, form encoding POSTs, composing URLs with arguments, thread safety etc. etc. without you even having to be aware of them, all in a terse and more readable format. It does it without taking the ability away to control those if you really want to, not that you're likely to need to anyway. Take a look at the two examples here, one using the standard library for http, urllib2, and the other using requests: https://gist.github.com/kennethreitz/973705 https://gist.github.com/kennethreitz/973705 I want requests to exist in pretty much every language. The API is clean and simple and perfectly expressive.
    - IndianAstronaut 11y ago
      Kenneth Reitz needs to do something with datetime in Python.
  - zellyn 11y ago
    Not only is it famous for actually performing HTTP requests, but it is widely considered a seminal example of excellent API design. You'll see quite a few Python libraries that aspire to emulate this quality of API design, often adopting Requests' tagline: "XXXX for humans"
- deleted 11y ago
  [deleted]
- deleted 11y ago
  [deleted]
- IznastY 11y ago
  I'm guessing this is based off of https://github.com/request/request https://github.com/request/request though I'm not 100%.
  - zorked 11y ago
    It's not, and I think it's older than Node.js.
- levigross 11y ago
  yes
jkldotio 11y ago
Very poorly named given Requests has a branch using Gevent that's already called grequests. https://github.com/kennethreitz/grequests https://github.com/kennethreitz/grequests
- Bedon292 11y ago
  I didn't even notice that part, very true. I think it should probably be gorequests. Sounds better than grequests anyways.
  - mangeletti 11y ago
    I agree, gorequests is the obvious choice, and changing the name before it becomes too popular would be a good idea.
    - pertsix 11y ago
      sounds like a great FPS title from the mid-90's. GoreQuests, Rated M for Mature.
  - mavroprovato 11y ago
    Gore Quests? Are you sure? :-)
    - Bedon292 11y ago
      Yeah, I definitely just reread that post and realized what it said. Maybe go-requests or something...
  - vhost- 11y ago
    There are so many packages with "go" in the name. Why not be a little more creative and avoid that? I don't put the word Go in any of my package names and it's been fun.
- levigross 11y ago
  I am not very good at naming things and figured to just slap a 'g' on requests => grequests. I figured that I would write the code and decide on a name later...
shawnps 11y ago
Whenever I'm reading through Go code and come across something that uses this sort of abstraction library, I usually just close it and move on. The stdlib is really nice and I find code that stays as "vanilla" Go as possible to be the easiest to read.
- Rezo 11y ago
  It's not the first time I've heard this sentiment around Go. I find it a fascinating difference in mindset to for example Node.js where in my experience even the smallest application will pull in dozens of dependencies and there's a package for absolutely everything imaginable. Because of the way dependencies are hierarchically scoped, there's no version conflicts in Node and a project may end up with 3 different Requests modules with only disk space as the cost. What do you consider to be the downsides to liberally using libraries that reduce the amount of code by wrapping the standard lib or that build on top of other libraries? Personally, and I may be wrong as someone still new to Go, it seems to me that the single monolithic repository model that Google uses permeates the whole community and has made adding an external dependency a Big Deal and something you have to carefully consider instead of something you instinctively do.
  - pcwalton 11y ago
    > What do you consider to be the downsides to liberally using libraries that reduce the amount of code by wrapping the standard lib or that build on top of other libraries? Personally, and I may be wrong as someone still new to Go, it seems to me that the single monolithic repository model that Google uses permeates the whole community and has made adding an external dependency a Big Deal and something you have to carefully consider instead of something you instinctively do. Because there's no equivalent of npm in Golang, culturally speaking.
  - tedsuo 11y ago
    Simply put: dependencies are code, and code comes at a maintenance cost. Thin abstractions, such as a this library, can be replaced by a function or two, which is a lower maintenance overhead because there is less code, you wrote the code, and all the code is being used. IMHO, dependencies should be pulled in for things that would take you weeks to do correctly. Avoid pulling in a dependency for something you can do yourself in 5 minutes. So I would pull in an http library if it actually did the mechanics of http better than the one in the stdlib, but not one that just wrapped it up for me a bit. For me, that's not a Go thing, that's just an I'm Older Now thing. Also, snarkily, dependency management is a mess in Go right now; none of us have any dependencies because we can't figure out how to do it. :P That is definitely a hole right now (though it's starting to get plugged in 1.5 with the /vendor directory) and is definitely exacerbated by the core team not being able to use any solution they could provide for the community.
domrdy 11y ago
Would be cool if the examples showcased fetching a list of URL's in parallel or something else that is a bit more complex.
- levigross 11y ago
  I agree! The README is really sparse right now (I didn't expect so much exposure right). It is just a matter of time before the README has more information.
iqandjoke 11y ago
After reading https://github.com/mozillazg/request https://github.com/mozillazg/request, I found yours looks a little bit better.
- levigross 11y ago
  Thanks – I wrote this library because I like using requests when writing in Python and wanted something similar when writing in Go.
spenczar5 11y ago
These ports from Python and Ruby (like Martini, for another example) are well-intentioned, but I don't think they're a good idea to use. The original libraries get a lot of their expressive power from polymorphism - Requests gets a tremendous amount of mileage out of letting you pass in keyword arguments to the `get` method, for example. Go prefers that you be more explicit and use more function declarations, even if it means repetition in the library code. When developers try to get around this, you see awkward constructs like the pointer-to-a-config-struct used here.
- evmar 11y ago
  I immediately noticed this as well. This API has flags for "asynchronous" requests, but Go style is to provide a blocking API, and let the caller decide to use a goroutine if they need concurrency.
  - XorNot 11y ago
    Constructing a channel (as it does though) is just pragmatic though and definitely not any less go-like. You get a channel - you can read from it until it's closed which is always safe. Go-routines are cheap so you don't care about the background details. For simplifying HTTP, this is great.
  - levigross 11y ago
    Originally the API was completely asynchronous (every "request" function returned a channel). I got a lot of feedback that I shouldn't do that, so I moved the asynchronous requests to specific functions e.g GetAsync and made the standard API synchronous.
    - sagichmal 11y ago
      You should go further, removing the async methods altogether. In Go, making sync APIs async is trivial and well-understood.
- coldtea 11y ago
  I personally welcome those kind of ports and hate blind insistent on "idiomatic" code. Idiomatic Java, Perl and other languages were shit for a long time, they only improved when some brave soulds fed up with it and explorer non-idiomatic solutions (like the Play framework). And some things I see in "idiomatic" Go make my hair crawl...
tptacek 11y ago
What's a use case in Go for an HTTP request library interface that returns a new channel for each request? There's no good way to select on a slice of channels, is there?
- chrj 11y ago
  Not without involving the reflect package. So no good way.
  - ble 11y ago
    The alternatives: Using http://golang.org/pkg/reflect/#Select http://golang.org/pkg/reflect/#Select to build what you need; Or building something specific to your use case without the above, which seems likely to be... goofy and unmanageable. Creating objects that create new channels to do `select`s over some fixed number of cases and composing those. Do you have any experience using `reflect.Select`? While not exactly ergonomic, it seemed a... pretty okay tradeoff, inconvenience for power, and at the very least more power for approximately the same amount of pain as the rest of the `reflect` package.
    - tptacek 11y ago
      One problem with using "reflect" this way, apart from the fact that it's (a) counter to idiom and (b) takes your code in a direction of being counter to idiom in general, is that "reflect" is pretty slow. You can do it, but since the only reason to do it is to avoid spawning goroutines, I'm not sure why you'd bother.
- jerf 11y ago
  No good way, no. I consider this a Go anti-pattern. Without a good reason, do not try to provide "asynchronousness" in a library. Go natively supports goroutines and channels, and it's considered baseline skill in the language to be able to fire something off in a goroutine (it's literally a two-character keyword) and receive something on a channel, if you want to. It would be not only adequate but preferable to implement this library as a fully synchronous request system and expect the user of the library to implement what asynchronousness they may require. Your hardwiring of exactly how the "asynchronousness" works may conflict with my own needs. All the "asynchronousness" you need to provide is already hard-wired into the Go runtime itself; what certain language communities have trained you to think is "synchronous" code already isn't. You don't have to super-duper-extra make it even more asynchronous. I won't quite call exposing channels in a library API a code smell, it's a little too useful for that, but it's still something where you ought to pause for a moment and really think about what it means, especially if you created it rather than receiving it. Oh, and let me be clear: Levigross, I'm seriously suggesting that you change this library wholesale to be fully synchronous, and the fact that this will be an API change is a feature, not a bug. You'll find it also simplifies the API significantly, also a feature.
nlake44 11y ago
Go is great, its open source. I wish Hacker News open sourced their code so that we knew what they are doing. Scared of having folks game the system? Why not have the community make it the best algorithm possible. Why keep pushing YC all the time? Do they not have enough already? Painters and hackers! ha!
- coldtea 11y ago
  It's not an abstract community, much less one with some FOSS mission statement or anything, it's a YC combinator site.
fishnchips 11y ago
As much as I love to see new things written in Go I am not a big fan of directly porting libraries written in different languages. It's not even that they're not idiomatic - I can live with that just fine. But the real power of Go comes from interfaces - two great examples being `io.Reader` and `io.Writer`. Once you can structure your library code so that it reasonably implements these standard interfaces you can do complex stuff in a trivial way. Say I want to get a file via HTTP, encrypt it while getting a hash of plaintext and upload it to say Google Cloud. I can pretty trivially do it by putting one `io.Writer` (Google Cloud) in another (encryption) and another (hashing, via `io.MultiWriter`) and then perform the whole thing using a simple `io.Copy`. This sort of expressive power can be achieved iff all libraries you're using adhere to the same philosophy - which is not the case with ports.
- levigross 11y ago
  I agree with this sentiment and therefore my Response method is also an `io.ReadCloser` (exposing the http.Response.Body io.ReadCloser that Go has to offer).
hobarrera 11y ago
Until go fixes it's issues with IPv6 (noticeably: it won't work on IPv6 out-of-the-box), I can't really respect it or consider it "production ready" for anything network related. No, I'm not trolling. #8453 got closed, and immediately, #11081 was opened since it re-broke this. Original issue reported by myself: https://github.com/golang/go/issues/8124 https://github.com/golang/go/issues/8124

92 points by agonzalezro 11y ago

gghh 11y ago

forgive my ignorance, is the original library http://python-requests.org http://python-requests.org ?

ionforce 11y ago
If true, I don't think it's ignorance. Not everyone is a Pythonista. Exactly how famous is this library?
- deleted 11y ago
  [deleted]
- mtrn 11y ago
  The Python standard library is overall very solid. However, it has a few darker corners - and, in the past, HTTP handling was not as easy as it could be. In the age of APIs, the requests library filled this void with a compellingly simple solution and became one of the most popular libraries (with millions of downloads every month [1]). [1] https://pypi.python.org/pypi/requests https://pypi.python.org/pypi/requests
- coldtea 11y ago
  In the Python world it's as famous as Joda or Hibernate would be for Javaists...
- sauere 11y ago
  > Exactly how famous is this library? It is the de-facto standard for anything that has to interact with a API, or just HTTP in general. The old way to do these sort of things was the urllib2 module (part of the Python standard library). urllib2 has some design flaws and can be a pain to work with.
- acdha 11y ago
  Extremely popular - almost every HTTP client library I use has switched to requests over the last couple years. As a proxy, check out the download counts on this tracker using data from the PyPI, where the vast majority of Python packages are installed from: https://python3wos.appspot.com/ https://python3wos.appspot.com/ Looking at the top few packages: * simplejson is the upstream for the json module in stdlib which people install since it gets performance updates first * requests * six: the most common library used to bridge the Python 2/3 transition * virtualenv: near-ubiquitous development tool (it allows you to maintain a separate Python environment for each project to avoid cross-talk, and is also used by popular testing tools like tox which run your tests under a variety of Python versions) * distribute: a few years back, the stdlib setuptools module was forked for a major overhaul. That's since been merged back in but many packages still reference it, particularly in older releases * boto: AWS client library, used by the official awscli tool * pip: Python package installer, now bundled with Python but updates & older versions of Python use the PyPI version
- Twirrim 11y ago
  About as famous as they come in the python community. It is the perfect example of a library done right. I've yet to meet a developer who have used it that dislike it. It handles all of the low level stuff for you, such as sessions, cookies, gzip, form encoding POSTs, composing URLs with arguments, thread safety etc. etc. without you even having to be aware of them, all in a terse and more readable format. It does it without taking the ability away to control those if you really want to, not that you're likely to need to anyway. Take a look at the two examples here, one using the standard library for http, urllib2, and the other using requests: https://gist.github.com/kennethreitz/973705 https://gist.github.com/kennethreitz/973705 I want requests to exist in pretty much every language. The API is clean and simple and perfectly expressive.
  - IndianAstronaut 11y ago
    Kenneth Reitz needs to do something with datetime in Python.
- zellyn 11y ago
  Not only is it famous for actually performing HTTP requests, but it is widely considered a seminal example of excellent API design. You'll see quite a few Python libraries that aspire to emulate this quality of API design, often adopting Requests' tagline: "XXXX for humans"
deleted 11y ago
[deleted]
deleted 11y ago
[deleted]
IznastY 11y ago
I'm guessing this is based off of https://github.com/request/request https://github.com/request/request though I'm not 100%.
- zorked 11y ago
  It's not, and I think it's older than Node.js.
levigross 11y ago
yes

jkldotio 11y ago

Very poorly named given Requests has a branch using Gevent that's already called grequests. https://github.com/kennethreitz/grequests https://github.com/kennethreitz/grequests

Bedon292 11y ago
I didn't even notice that part, very true. I think it should probably be gorequests. Sounds better than grequests anyways.
- mangeletti 11y ago
  I agree, gorequests is the obvious choice, and changing the name before it becomes too popular would be a good idea.
  - pertsix 11y ago
    sounds like a great FPS title from the mid-90's. GoreQuests, Rated M for Mature.
- mavroprovato 11y ago
  Gore Quests? Are you sure? :-)
  - Bedon292 11y ago
    Yeah, I definitely just reread that post and realized what it said. Maybe go-requests or something...
- vhost- 11y ago
  There are so many packages with "go" in the name. Why not be a little more creative and avoid that? I don't put the word Go in any of my package names and it's been fun.
levigross 11y ago
I am not very good at naming things and figured to just slap a 'g' on requests => grequests. I figured that I would write the code and decide on a name later...

shawnps 11y ago

Whenever I'm reading through Go code and come across something that uses this sort of abstraction library, I usually just close it and move on. The stdlib is really nice and I find code that stays as "vanilla" Go as possible to be the easiest to read.

Rezo 11y ago
It's not the first time I've heard this sentiment around Go. I find it a fascinating difference in mindset to for example Node.js where in my experience even the smallest application will pull in dozens of dependencies and there's a package for absolutely everything imaginable. Because of the way dependencies are hierarchically scoped, there's no version conflicts in Node and a project may end up with 3 different Requests modules with only disk space as the cost. What do you consider to be the downsides to liberally using libraries that reduce the amount of code by wrapping the standard lib or that build on top of other libraries? Personally, and I may be wrong as someone still new to Go, it seems to me that the single monolithic repository model that Google uses permeates the whole community and has made adding an external dependency a Big Deal and something you have to carefully consider instead of something you instinctively do.
- pcwalton 11y ago
  > What do you consider to be the downsides to liberally using libraries that reduce the amount of code by wrapping the standard lib or that build on top of other libraries? Personally, and I may be wrong as someone still new to Go, it seems to me that the single monolithic repository model that Google uses permeates the whole community and has made adding an external dependency a Big Deal and something you have to carefully consider instead of something you instinctively do. Because there's no equivalent of npm in Golang, culturally speaking.
- tedsuo 11y ago
  Simply put: dependencies are code, and code comes at a maintenance cost. Thin abstractions, such as a this library, can be replaced by a function or two, which is a lower maintenance overhead because there is less code, you wrote the code, and all the code is being used. IMHO, dependencies should be pulled in for things that would take you weeks to do correctly. Avoid pulling in a dependency for something you can do yourself in 5 minutes. So I would pull in an http library if it actually did the mechanics of http better than the one in the stdlib, but not one that just wrapped it up for me a bit. For me, that's not a Go thing, that's just an I'm Older Now thing. Also, snarkily, dependency management is a mess in Go right now; none of us have any dependencies because we can't figure out how to do it. :P That is definitely a hole right now (though it's starting to get plugged in 1.5 with the /vendor directory) and is definitely exacerbated by the core team not being able to use any solution they could provide for the community.

domrdy 11y ago

Would be cool if the examples showcased fetching a list of URL's in parallel or something else that is a bit more complex.

levigross 11y ago
I agree! The README is really sparse right now (I didn't expect so much exposure right). It is just a matter of time before the README has more information.

iqandjoke 11y ago

After reading https://github.com/mozillazg/request https://github.com/mozillazg/request, I found yours looks a little bit better.

levigross 11y ago
Thanks – I wrote this library because I like using requests when writing in Python and wanted something similar when writing in Go.

spenczar5 11y ago

These ports from Python and Ruby (like Martini, for another example) are well-intentioned, but I don't think they're a good idea to use. The original libraries get a lot of their expressive power from polymorphism - Requests gets a tremendous amount of mileage out of letting you pass in keyword arguments to the `get` method, for example. Go prefers that you be more explicit and use more function declarations, even if it means repetition in the library code. When developers try to get around this, you see awkward constructs like the pointer-to-a-config-struct used here.

evmar 11y ago
I immediately noticed this as well. This API has flags for "asynchronous" requests, but Go style is to provide a blocking API, and let the caller decide to use a goroutine if they need concurrency.
- XorNot 11y ago
  Constructing a channel (as it does though) is just pragmatic though and definitely not any less go-like. You get a channel - you can read from it until it's closed which is always safe. Go-routines are cheap so you don't care about the background details. For simplifying HTTP, this is great.
- levigross 11y ago
  Originally the API was completely asynchronous (every "request" function returned a channel). I got a lot of feedback that I shouldn't do that, so I moved the asynchronous requests to specific functions e.g GetAsync and made the standard API synchronous.
  - sagichmal 11y ago
    You should go further, removing the async methods altogether. In Go, making sync APIs async is trivial and well-understood.
coldtea 11y ago
I personally welcome those kind of ports and hate blind insistent on "idiomatic" code. Idiomatic Java, Perl and other languages were shit for a long time, they only improved when some brave soulds fed up with it and explorer non-idiomatic solutions (like the Play framework). And some things I see in "idiomatic" Go make my hair crawl...

tptacek 11y ago

What's a use case in Go for an HTTP request library interface that returns a new channel for each request? There's no good way to select on a slice of channels, is there?

chrj 11y ago
Not without involving the reflect package. So no good way.
- ble 11y ago
  The alternatives: Using http://golang.org/pkg/reflect/#Select http://golang.org/pkg/reflect/#Select to build what you need; Or building something specific to your use case without the above, which seems likely to be... goofy and unmanageable. Creating objects that create new channels to do `select`s over some fixed number of cases and composing those. Do you have any experience using `reflect.Select`? While not exactly ergonomic, it seemed a... pretty okay tradeoff, inconvenience for power, and at the very least more power for approximately the same amount of pain as the rest of the `reflect` package.
  - tptacek 11y ago
    One problem with using "reflect" this way, apart from the fact that it's (a) counter to idiom and (b) takes your code in a direction of being counter to idiom in general, is that "reflect" is pretty slow. You can do it, but since the only reason to do it is to avoid spawning goroutines, I'm not sure why you'd bother.
jerf 11y ago
No good way, no. I consider this a Go anti-pattern. Without a good reason, do not try to provide "asynchronousness" in a library. Go natively supports goroutines and channels, and it's considered baseline skill in the language to be able to fire something off in a goroutine (it's literally a two-character keyword) and receive something on a channel, if you want to. It would be not only adequate but preferable to implement this library as a fully synchronous request system and expect the user of the library to implement what asynchronousness they may require. Your hardwiring of exactly how the "asynchronousness" works may conflict with my own needs. All the "asynchronousness" you need to provide is already hard-wired into the Go runtime itself; what certain language communities have trained you to think is "synchronous" code already isn't. You don't have to super-duper-extra make it even more asynchronous. I won't quite call exposing channels in a library API a code smell, it's a little too useful for that, but it's still something where you ought to pause for a moment and really think about what it means, especially if you created it rather than receiving it. Oh, and let me be clear: Levigross, I'm seriously suggesting that you change this library wholesale to be fully synchronous, and the fact that this will be an API change is a feature, not a bug. You'll find it also simplifies the API significantly, also a feature.

nlake44 11y ago

Go is great, its open source. I wish Hacker News open sourced their code so that we knew what they are doing. Scared of having folks game the system? Why not have the community make it the best algorithm possible. Why keep pushing YC all the time? Do they not have enough already? Painters and hackers! ha!

coldtea 11y ago
It's not an abstract community, much less one with some FOSS mission statement or anything, it's a YC combinator site.

fishnchips 11y ago

As much as I love to see new things written in Go I am not a big fan of directly porting libraries written in different languages. It's not even that they're not idiomatic - I can live with that just fine. But the real power of Go comes from interfaces - two great examples being `io.Reader` and `io.Writer`. Once you can structure your library code so that it reasonably implements these standard interfaces you can do complex stuff in a trivial way. Say I want to get a file via HTTP, encrypt it while getting a hash of plaintext and upload it to say Google Cloud. I can pretty trivially do it by putting one `io.Writer` (Google Cloud) in another (encryption) and another (hashing, via `io.MultiWriter`) and then perform the whole thing using a simple `io.Copy`. This sort of expressive power can be achieved iff all libraries you're using adhere to the same philosophy - which is not the case with ports.

levigross 11y ago
I agree with this sentiment and therefore my Response method is also an `io.ReadCloser` (exposing the http.Response.Body io.ReadCloser that Go has to offer).

hobarrera 11y ago

Until go fixes it's issues with IPv6 (noticeably: it won't work on IPv6 out-of-the-box), I can't really respect it or consider it "production ready" for anything network related. No, I'm not trolling. #8453 got closed, and immediately, #11081 was opened since it re-broke this. Original issue reported by myself: https://github.com/golang/go/issues/8124 https://github.com/golang/go/issues/8124