4 ms·

Noob question: what's the purpose of having so many request libraries? In JavaScript it's crazy, but in Python it's starting to feel the same: requests, httpx,

by BoumTAC 1mo ago

Noob question: what's the purpose of having so many request libraries? In JavaScript it's crazy, but in Python it's starting to feel the same: requests, httpx, httpx2... From my perspective, HTTP requests have been mastered for years (decades?). What's the point of switching libraries? What's so extraordinary about the new HTTP library that makes requests look crap? I understand that new protocols appear from time to time (like HTTP/2), but that's very rare. And if there are performance differences between two libraries, are they really that huge? Even if they are, 99% of people don't need that boost. What prevents someone from just improving the existing library's performance?

t098i3 1mo ago
`requests` has a friendlier API than the standard library. However, it does not support newer versions of HTTP, nor does it support `async`. The maintainer considers it complete, feature-frozen software, IIRC - they'd rather keep it stable for existing users and let other libraries succeed it. `aiohttp` supports `async` but was never popular for non-async `httpx` supported newer HTTP versions and both async and non-async. However the maintainer decided to close all outside submissions, issues and discussions for personal reasons. `httpx2` is a maintained fork of `httpx`
- BoumTAC 1mo ago
  Thanks for the explanation !
- lofaszvanitt 1mo ago
  And why don't people use pycurl? Curl is the king of the hill...
  - t098i3 1mo ago
    There are tradeoffs to the non-Python HTTP libraries like pycurl and pyreqwests, namely: - Non-portability to alternative Python implementations or environments without the supporting C libraries - Less visibility into the client internals (for debugging etc)
    - lofaszvanitt 1mo ago
      Thx!
- niltecedu 1mo ago
  Also niquests as well which is a dropin replacement and imo wayyy better than httpx
  - JackSlateur 1mo ago
    niquest is something like a one-man-show, which is an issue-per-se Of course, it is clearly superior to everything else (being authored by a french guy !). But sadly, some people do not accept this simple fact :(
    - fmajid 1mo ago
      “Niquer” means “to f*ck” in French slang
- giov4 1mo ago
  thanks for the summary. >`httpx` supported newer HTTP versions and both async and non-async. However the maintainer decided to close all outside submissions, issues and discussions for personal reasons. how can something like this happen? and which positive consequence there can be out of that?
  - t098i3 1mo ago
    > how can something like this happen? https://xkcd.com/2347 https://xkcd.com/2347 Specifically in this case, the consensus seems to be the maintainer is having some mental health struggles and chose to close out outside contribution.
    - giov4 1mo ago
      ohh thanks for that xkcd! totally right! only problem I have and that's probably why I didn't associate it directly myself is that I see the example very valid for many other FOSS cases. but in this case it feels like the part at the bottom is made of multiple random persons, not a single one, and one of those random persons decided to exclude all the other random persons. so maybe the lower layer shall be represented as made of multiple pieces that fight each other, beside the all modern shit on top of it.
- RantyDave 1mo ago
  'urllib3' completely saved me once. We had to authenticate with the border proxy using a certificate, then make a request to a web service ... that needed authenticating with a certificate. Urllib3 was the only one that could do it.
  - anakaine 1mo ago
    I've had to do similar in the past, and also wound up on urllib3.
deleted 1mo ago
[deleted]
globular-toast 1mo ago
There can be different APIs or different technical implementations with certain performance characteristics. In this case, there isn't any purpose, though. It's just that development has slowed or stopped on the previous libraries and forking/renaming is easier than trying to continue the previous projects, for various reasons.
lucideer 1mo ago
HTTP is a pretty complex space once you get into the nitty-gritty of use-cases, common request handling patterns & conditional/dynamic header-definitions. In general this leads to: - (a) built-ins (& some libraries) being very verbose to use because they tend to take a "fundamentals of HTTP" approach & not implement shortcuts/common patterns - (b) libraries commonly falling into the trap of going the opposite direction & ending up with massive scope-creep and/or oversimplified, un-expressive APIs. This leads to a churn of libraries due to developers yearning for a Goldilocks implementation in between. Javascript tends to be ahead of other ecosystems in churn due to ease of library publishing, Python's catching up, but overall it's the same pattern in both. --- Also, one big additional factor is async: - NodeJS built-ins implemented a callback pattern just before promises took off, & have struggled to upgrade to promises cleanly without breaking backward compat. - Python's got a similar story with various generations of http library being sync, or relying on different async approaches in confusing ways.
deleted 1mo ago
[deleted]
fulafel 1mo ago
Async ruins everything, as usual.
EugeneV13 1mo ago
[flagged]

by BoumTAC 1mo ago

t098i3 1mo ago

`requests` has a friendlier API than the standard library. However, it does not support newer versions of HTTP, nor does it support `async`. The maintainer considers it complete, feature-frozen software, IIRC - they'd rather keep it stable for existing users and let other libraries succeed it. `aiohttp` supports `async` but was never popular for non-async `httpx` supported newer HTTP versions and both async and non-async. However the maintainer decided to close all outside submissions, issues and discussions for personal reasons. `httpx2` is a maintained fork of `httpx`

BoumTAC 1mo ago
Thanks for the explanation !
lofaszvanitt 1mo ago
And why don't people use pycurl? Curl is the king of the hill...
- t098i3 1mo ago
  There are tradeoffs to the non-Python HTTP libraries like pycurl and pyreqwests, namely: - Non-portability to alternative Python implementations or environments without the supporting C libraries - Less visibility into the client internals (for debugging etc)
  - lofaszvanitt 1mo ago
    Thx!
niltecedu 1mo ago
Also niquests as well which is a dropin replacement and imo wayyy better than httpx
- JackSlateur 1mo ago
  niquest is something like a one-man-show, which is an issue-per-se Of course, it is clearly superior to everything else (being authored by a french guy !). But sadly, some people do not accept this simple fact :(
  - fmajid 1mo ago
    “Niquer” means “to f*ck” in French slang
giov4 1mo ago
thanks for the summary. >`httpx` supported newer HTTP versions and both async and non-async. However the maintainer decided to close all outside submissions, issues and discussions for personal reasons. how can something like this happen? and which positive consequence there can be out of that?
- t098i3 1mo ago
  > how can something like this happen? https://xkcd.com/2347 https://xkcd.com/2347 Specifically in this case, the consensus seems to be the maintainer is having some mental health struggles and chose to close out outside contribution.
  - giov4 1mo ago
    ohh thanks for that xkcd! totally right! only problem I have and that's probably why I didn't associate it directly myself is that I see the example very valid for many other FOSS cases. but in this case it feels like the part at the bottom is made of multiple random persons, not a single one, and one of those random persons decided to exclude all the other random persons. so maybe the lower layer shall be represented as made of multiple pieces that fight each other, beside the all modern shit on top of it.
RantyDave 1mo ago
'urllib3' completely saved me once. We had to authenticate with the border proxy using a certificate, then make a request to a web service ... that needed authenticating with a certificate. Urllib3 was the only one that could do it.
- anakaine 1mo ago
  I've had to do similar in the past, and also wound up on urllib3.

deleted 1mo ago

[deleted]

globular-toast 1mo ago

There can be different APIs or different technical implementations with certain performance characteristics. In this case, there isn't any purpose, though. It's just that development has slowed or stopped on the previous libraries and forking/renaming is easier than trying to continue the previous projects, for various reasons.

lucideer 1mo ago

HTTP is a pretty complex space once you get into the nitty-gritty of use-cases, common request handling patterns & conditional/dynamic header-definitions. In general this leads to: - (a) built-ins (& some libraries) being very verbose to use because they tend to take a "fundamentals of HTTP" approach & not implement shortcuts/common patterns - (b) libraries commonly falling into the trap of going the opposite direction & ending up with massive scope-creep and/or oversimplified, un-expressive APIs. This leads to a churn of libraries due to developers yearning for a Goldilocks implementation in between. Javascript tends to be ahead of other ecosystems in churn due to ease of library publishing, Python's catching up, but overall it's the same pattern in both. --- Also, one big additional factor is async: - NodeJS built-ins implemented a callback pattern just before promises took off, & have struggled to upgrade to promises cleanly without breaking backward compat. - Python's got a similar story with various generations of http library being sync, or relying on different async approaches in confusing ways.

deleted 1mo ago

[deleted]

fulafel 1mo ago

Async ruins everything, as usual.

EugeneV13 1mo ago

[flagged]