14 ms·
Httpx: A Ruby HTTP library
- thunderbong 7y agoI've been using httparty, since forever. Am I right to assume the primary difference with this library is concurrent connections? https://github.com/jnunemaker/httparty https://github.com/jnunemaker/httparty
- ykevinator 7y agoMe too but I am intrrigued.
- chucke 7y agohttparty is a dsl wrapper around net/http, so it's also limited by it. For instance, it doesn't come with keep-alive support (I'm using httparty in my dayjob legacy code). httpx is more than concurrent connections. It's actually about concurrent requests done right, advanced http features (it can do connection coalescing, alt-svc), and it comes with a baked-in plugin system which makes it easy to extend. Go through the wiki and give it a try when you can.
- BerislavLopac 7y agoHmmm... https://www.encode.io/httpx/ https://www.encode.io/httpx/
- chucke 7y agoThat one seems more recent. Would be cool to know if ruby's httpx was the inspiration behind it (I can only judge looking at the name :) ).
- florimondmanca 7y agoPython HTTPX co-maintainer here: AFAIK, Ruby's HTTPX did not inspire Python's HTTPX. Eg. the name came from a totally unrelated discussion, and research did not yield results about Ruby's counterpart back then [0]. In fact, Python HTTPX started with requests-async [1], which was then spun up into httpcore, which was then renamed to http3, and finally HTTPX. Still very funny to me that these two projects (Python HTTPX and Ruby HTTPX) have so much in common: HTTP/2 support, API compatibility with the current de-facto HTTP library of the language, etc. [0]: https://github.com/python-http/discussions/issues/1#issuecomment-510014839 https://github.com/python-http/discussions/issues/1#issuecom... [1]: https://github.com/encode/requests-async https://github.com/encode/requests-async
- uranusjr 7y agoLOL I was almost posted a ref to HTTPX until I saw this thread :p Thank you for HTTPX. I only learnt about it recently, but it’s now my go-to tool moving forward. Excellent work!
- chucke 7y agoTrue, there are indeed a lot of common goals, and even features. Even our debug flag (HTTPX_DEBUG) is the same. Is that a python pattern? Good luck in our work!
- lorensi 7y agoBefore we start, I would like to say a few words about Net::HTTP. Net::HTTP is the HTTP client that comes with the Ruby standard library, and it's ... https://10peeps.com/best-fm-transmitter-for-car/ https://10peeps.com/best-fm-transmitter-for-car/
- 0xFACEFEED 7y agoNever understood why people prefer APIs like this: page1, page2, ... = HTTPX.get("/page1", "/page2", ...) Why encode parallelism into a single polymorphic function? Why not use composition instead? Now you're stuck with this API forever. And if you want to customize parallel fetches (eg: batching) you'll need another different looking API.
- cranej 7y agoAnd I guess it won't return unless all pages are fetched? even worse...
- chucke 7y agoThere are timeouts involved, that is, you'll get error responses. Feel free to try the example scripts.
- cranej 7y ago> ...while baking in support for concurrent requests... Timeout does not help here - if the api is for concurrent requests, it's bad that caller cannot do anything before `(the slowest request finished || timeout exceeded)`.
- chucke 7y agoHi, I maintain httpx. I'll try to answer your concerns. The API was not a question of preference. If you look at most http lib APIs (python requests included), a single call returns a response object. My goal was to keep compatibility with this API to ease the migration effort, while baking in support for concurrent requests, which was the whole point of the lib. There is no parallelism in that call btw. It's in the Readme. If it's http/2, it'll multiplex requests. If it's http/1, it'll try to pipeline, and if doesn't work, it'll fetch them one by one (with or without keep-alive). If they're different origins, the requests will be managed inside an event loop. > Why not use composition instead? You'll have to explain with an example of your own. However, there's already a ticket for implementing an event-based request handling, I just never prioritized it because no one asked for it yet, no one has shown interest in contributing, and frankly, I haven't needed it yet. > Now you're stuck with this API forever. It's still early 0.x days, so I might still remove APIs I don't think are future proof. That said, a lot of libs have lived with similar APIs all these years, and I don't see any reason for removing it.
- fouc 7y agoRandom comment - I find it interesting that I have such a strong association of open source = github, so when something is on gitlab, I automatically deduct mental points, even though that's not fair at all. I think gitlab's sidebar is probably not helping with my association, it's too similar to bitbucket or other "enterprise-y" type websites.
- midgetjones 7y agoPersonally I find the UI and terminology so willfully - almost obstinately - different to github's, that it's an absolute pain to navigate. I'm sure I could get used to it, but just now I just wanted to jump to the code, and it took roughly 30 seconds to find it hidden inside the 'repository' tab.
- manojlds 7y agoThe link is already in a page similar to the main page of repo on Github. Open and link and scroll up?
- midgetjones 7y agoUgh, it's a link to the `README`, with a fixed sidebar. I partially take back my original comment, but I'd still argue that it's not the most helpful UI.
- sytse 7y agoHow can we make it more helpful?
- shantly 7y agoYeah, some of it is just familiarity but I do think GH truly nailed UI to quickly evaluate a project while also being good for working in long-term, at least on desktop—I have complaints about their mobile UI, but when I'm using that it's usually because I'm just curious and messing around anyway, so it's not a big deal. It's also nice and snappy. [EDIT] I also assume a higher likelihood that key persons on the project are the contrarian-for-contrary's-sake sort if they're not on GH. Not that it's certain, mind, but that it's a bit more likely.
- save_ferris 7y agoWhy would I choose this over Faraday?
- davefp 7y agoFaraday is a wrapper around lower-level libraries like this one. Faraday would likely need some modification to support the concurrency aspect, but eventually you could be using Faraday _with_ httpx, rather than instead of.
- save_ferris 7y agoGood to know, thanks!
- conorh 7y agoIt wasn't until I read a comment here that there was a wiki that I found most of your documentation. I'd link this higher in your README as it is what I was looking for when I went to the project (lots of examples): https://gitlab.com/honeyryderchuck/httpx/-/wiki_pages/home https://gitlab.com/honeyryderchuck/httpx/-/wiki_pages/home
- kawsper 7y agoDoes anyone have any plans/thoughts/visions for how we should deal with SRV-records in our standard libraries in different languages? Requesting a DNS-server for a SRV-record gives you a prioritised list of IPs and ports, and it is up to the implementor to pick the right one. As it works currently you would have to do a lookup, pick a endpoint, and then feed it into your HTTP library or TCP library. This is fine in code that you control, but there are so many libraries out there that expects a hostname or an IP-address with a port number, and in a highly containerised world endpoints are more short lived than ever, so we need retry mechanics as well: Try the next server:port pair in the prioritized list.
- acejam 7y agoThis looks interesting, thanks for sharing. I've always had good success with httprb[1], but I'll give this a look over too. [1] https://github.com/httprb/http https://github.com/httprb/http
- hartator 7y agoWe are using http.rb at https://serpapi.com https://serpapi.com It’s interesting to see that the Ruby ecosystem doesn’t seem to find its “requests” library. There is so many very good competitive libraries: native net/http, HttpClient by Nahi, http.rb, HttpParty, RestHttp, RestClient, excon, em-http, faraday, curb, etc., and now Httpx!
- Exuma 7y agoDon't forget Typhoeus. That's literally my Goto "nuke the API with 10,000 concurrent requests" library. It's also based on Curl so very stable. https://github.com/typhoeus/typhoeus https://github.com/typhoeus/typhoeus hydra = Typhoeus::Hydra.new 10.times do request = Typhoeus::Request.new("www.example.com", followlocation: true) request.on_complete do |response| #do_something_with response end hydra.queue(request) end hydra.run
- hartator 7y agoTrue. Typhoeus is awesome as well.
- VectorLock 7y agoIts almost emblematic of Ruby's mindset vs. Python's "there is one right way to do something" philosophy.
- hartator 7y agoExcept for the language itself: Python 2.x vs 3.x are still battling choices after 10 years :)
- pritambaral 7y agoThere is no battle. Python 2 updates have been unsupported for 10 years, and full support dies with this year.
- dragonwriter 7y ago
- obahareth 7y agoThis looks like a pretty awesome library and very well-written and documented library. I'll definitely try it out in some of my Ruby projects and I would love to see an adapter for Faraday. Kudos to the author(s).
- chucke 7y agoThank you. And there's already one :)
- damir 7y agoI found this gem (pun intended) only after having an issue with original awesome http.rb gem not being able to talk http2. Happy httpx user ever since. Ping me if you decide to start accepting donations/support payments.
- pbreit 7y agoIt's astounding to me that so many platforms/languages still have such mediocre http libraries. Surely basic http capabilities should be built-in or in standard libraries?
- jsnk 7y agoIt is part of core Ruby. https://ruby-doc.org/stdlib-2.6.5/libdoc/net/http/rdoc/Net/HTTP.html https://ruby-doc.org/stdlib-2.6.5/libdoc/net/http/rdoc/Net/H... People develop other HTTP packages usually for ergonomic reasons, but the HTTP from core Ruby works fine.
- sandstrom 7y agoGreat library! Although choice and experimentation certainly has its value, I'm curious to hear if there was any attempt to build the missing features into any of the existing libraries? (for example excon or http.rb which this is based on)
- damir 7y agoAlso, rubyweekly #474 [1] has HTTPX gem listed on top of the issue. Congrats to HTTPX author on well deserved attention. 1: https://rubyweekly.com/issues/474 https://rubyweekly.com/issues/474