7 ms·

A retrospective on Requests

125 points by zdw 3y ago

dplarson 3y ago
For folks that don't recognize the author (Ian Stapleton Cordasco), he's one of the core maintainers for python-requests (sigmavirus24 on github).
raziel2p 3y ago
Yikes. I always had a weird feeling about Requests looking from the outside, but it's worse than I thought. If you're looking for alternatives: https://www.python-httpx.org/ https://www.python-httpx.org/
- zdw 3y ago
  Or just the built-in urllib is pretty usable as-is at this point.
  - sjwhevvvvvsj 3y ago
    My main issue with urllib is it requires too much boilerplate and it ends up being harder to read the code at a later date in my experience. In general I prefer the standard library, but it’s a bit of a toss up in this case.
- argulane 3y ago
  Plain urllib3 staring from v2 is also pretty nice to use.
- globular-toast 3y ago
  I use and have contributed to httpx but I'm scared of moths and had to block the image in my browser. Looks like they updated the image to be an even more detailed one :(
  - aobdev 3y ago
    I’ve seen comments here on HN about people having similarly visceral reactions to the name CockroachDB and swear they will never use it for that reason alone. I always laughed at the absurdity but now I actually have some sympathy towards this type of reaction. If I opened the docs for an unsuspecting python package and was greeted with a detailed, half-screen photo of any arachnid I would probably throw whatever device I was using.
  - oasisaimlessly 3y ago
    For reference, it's a butterfly, not a moth. Source: https://github.com/encode/httpx/issues/834 https://github.com/encode/httpx/issues/834
noelwelsh 3y ago
It's good to see an API critique. I don't think enough attention is paid to their design. The problematic design is something I see fairly frequently from people who aren't familiar with sum types, aka modelling data that has logical ors. It's very common in the Javascript world as well, for example, and in Go it's baked in with functions returning four possible values (success and error, success and nil, nil and error, nil and nil) when only two are valid. A better design uses a combinator libraries / builders / fluent API. These are little finite state machines and the best libraries make it so invalid transitions cannot be compiled (which requires a type system).
lenkite 3y ago
They should just go ahead and make breaking changes after a big announcement so that everyone is clear that will no backward compatibility at the next major release. Otherwise the majority of design, development and mental energy will be spent in compatibility issues leading to full-burnout with little to no feature gain. You only need to look at C++ papers where superhumanly smart and driven IQ 200+ people desperately try with all their will to define a language or stdlib feature/improvement that preserves ABI backward compatibility and then finally give up and throw the towel after two dozen iterations and rejections. And also suffer from severe burnout in the process.
- EchoChamberMan 3y ago
  Pretty sure Deno for Node has similar problems. Sometimes you just gotta let the past go.
  - LunaSea 3y ago
    Only if the delta of the improvements is big enough which isn't the case for Deno
    - EchoChamberMan 3y ago
      Sorry what? I'm saying Deno suffers due to Node backward compatibility. A write up about it: https://www.baldurbjarnason.com/2024/disillusioned-with-deno/ https://www.baldurbjarnason.com/2024/disillusioned-with-deno...
- zokier 3y ago
  Or just make new project and let community deal with requests if they want.
  - meitham 3y ago
    I don’t see a market need for a new one. Python’s own stdlib urllib is pretty good for most cases, except async, and aiohttp is covering that gap.
zokier 3y ago
I feel this is bit of a recurring theme in Python world where nice simple things are dragged into serious (enterprise) production world and then they accumulate gazillion bits of cruft, hiding the originally nice simple thing. It's quite fundamental problem trying to please both the ad-hoc small script and X million line production system users; Python straddling the gap imho ends up pleasing neither. My preference would be to focus on excelling on the small-scale stuff and leave the complex stuff for ecosystems better aligned with that; basically let Java be Java and Python be Python.
- game_the0ry 3y ago
  > My preference would be to focus on excelling on the small-scale stuff and leave the complex stuff for ecosystems better aligned with that; basically let Java be Java and Python be Python. Agreed 100%. Strongly prefer python for the simple scripting and conceptually simple mental model, and others for more complex stuff. Also, don't use tools that were not designed for the job from the beginning and then complain that it does not fit your use case.
  - math_dandy 3y ago
    Don’t you think that ship has sailed? Python powers software of incredible complexity these days including back-ends for large companies and all of AI (yes, I know the “hard parts” are written in C++, but still).
    - game_the0ry 3y ago
      > Don’t you think that ship has sailed? No, those ships are setting sail with anchors down, and complaining why they're moving so slow.
      - itsoktocry 3y ago
        >No, those ships are setting sail with anchors down, and complaining why they're moving so slow. It's 2024 and you still think the choice of language in which you build your product is what makes or breaks a company?
blibble 3y ago
> I gave it but realized that most people only know that Requests does the right things for them for TLS as long as you don't work somewhere that has its own internal CA infrastructure requests decides to use its own built-in CA store instead of the carefully tweaked OpenSSL default (the location of which you may not know declaratively) meanwhile urllib works fine (yes I read further down where this point is sort of mentioned)
bryik 3y ago
Interesting post on a library I've used a lot. > Requests has always had modules intended for its use and only its use. At some point, we tried to document them for ourselves and that led to people using them externally and then filing bugs against them. This is a perfect example of 3 adults who did not consent explicitly to the use, but Python says that we must have because we didn't hide them well enough. I don't understand the problem? It's bad that users reported bugs in internal modules?
- aunderscored 3y ago
  It's bad that users used modules marked explicitly as not got public use, because now those modules can't be changed without breaking user code.

125 points by zdw 3y ago

For folks that don't recognize the author (Ian Stapleton Cordasco), he's one of the core maintainers for python-requests (sigmavirus24 on github).

raziel2p 3y ago

Yikes. I always had a weird feeling about Requests looking from the outside, but it's worse than I thought. If you're looking for alternatives: https://www.python-httpx.org/ https://www.python-httpx.org/

zdw 3y ago
Or just the built-in urllib is pretty usable as-is at this point.
- sjwhevvvvvsj 3y ago
  My main issue with urllib is it requires too much boilerplate and it ends up being harder to read the code at a later date in my experience. In general I prefer the standard library, but it’s a bit of a toss up in this case.
argulane 3y ago
Plain urllib3 staring from v2 is also pretty nice to use.
globular-toast 3y ago
I use and have contributed to httpx but I'm scared of moths and had to block the image in my browser. Looks like they updated the image to be an even more detailed one :(
- aobdev 3y ago
  I’ve seen comments here on HN about people having similarly visceral reactions to the name CockroachDB and swear they will never use it for that reason alone. I always laughed at the absurdity but now I actually have some sympathy towards this type of reaction. If I opened the docs for an unsuspecting python package and was greeted with a detailed, half-screen photo of any arachnid I would probably throw whatever device I was using.
- oasisaimlessly 3y ago
  For reference, it's a butterfly, not a moth. Source: https://github.com/encode/httpx/issues/834 https://github.com/encode/httpx/issues/834

noelwelsh 3y ago

It's good to see an API critique. I don't think enough attention is paid to their design. The problematic design is something I see fairly frequently from people who aren't familiar with sum types, aka modelling data that has logical ors. It's very common in the Javascript world as well, for example, and in Go it's baked in with functions returning four possible values (success and error, success and nil, nil and error, nil and nil) when only two are valid. A better design uses a combinator libraries / builders / fluent API. These are little finite state machines and the best libraries make it so invalid transitions cannot be compiled (which requires a type system).

lenkite 3y ago

They should just go ahead and make breaking changes after a big announcement so that everyone is clear that will no backward compatibility at the next major release. Otherwise the majority of design, development and mental energy will be spent in compatibility issues leading to full-burnout with little to no feature gain. You only need to look at C++ papers where superhumanly smart and driven IQ 200+ people desperately try with all their will to define a language or stdlib feature/improvement that preserves ABI backward compatibility and then finally give up and throw the towel after two dozen iterations and rejections. And also suffer from severe burnout in the process.

EchoChamberMan 3y ago
Pretty sure Deno for Node has similar problems. Sometimes you just gotta let the past go.
- LunaSea 3y ago
  Only if the delta of the improvements is big enough which isn't the case for Deno
  - EchoChamberMan 3y ago
    Sorry what? I'm saying Deno suffers due to Node backward compatibility. A write up about it: https://www.baldurbjarnason.com/2024/disillusioned-with-deno/ https://www.baldurbjarnason.com/2024/disillusioned-with-deno...
zokier 3y ago
Or just make new project and let community deal with requests if they want.
- meitham 3y ago
  I don’t see a market need for a new one. Python’s own stdlib urllib is pretty good for most cases, except async, and aiohttp is covering that gap.

zokier 3y ago

I feel this is bit of a recurring theme in Python world where nice simple things are dragged into serious (enterprise) production world and then they accumulate gazillion bits of cruft, hiding the originally nice simple thing. It's quite fundamental problem trying to please both the ad-hoc small script and X million line production system users; Python straddling the gap imho ends up pleasing neither. My preference would be to focus on excelling on the small-scale stuff and leave the complex stuff for ecosystems better aligned with that; basically let Java be Java and Python be Python.

game_the0ry 3y ago
> My preference would be to focus on excelling on the small-scale stuff and leave the complex stuff for ecosystems better aligned with that; basically let Java be Java and Python be Python. Agreed 100%. Strongly prefer python for the simple scripting and conceptually simple mental model, and others for more complex stuff. Also, don't use tools that were not designed for the job from the beginning and then complain that it does not fit your use case.
- math_dandy 3y ago
  Don’t you think that ship has sailed? Python powers software of incredible complexity these days including back-ends for large companies and all of AI (yes, I know the “hard parts” are written in C++, but still).
  - game_the0ry 3y ago
    > Don’t you think that ship has sailed? No, those ships are setting sail with anchors down, and complaining why they're moving so slow.
    - itsoktocry 3y ago
      >No, those ships are setting sail with anchors down, and complaining why they're moving so slow. It's 2024 and you still think the choice of language in which you build your product is what makes or breaks a company?

blibble 3y ago

> I gave it but realized that most people only know that Requests does the right things for them for TLS as long as you don't work somewhere that has its own internal CA infrastructure requests decides to use its own built-in CA store instead of the carefully tweaked OpenSSL default (the location of which you may not know declaratively) meanwhile urllib works fine (yes I read further down where this point is sort of mentioned)

bryik 3y ago

Interesting post on a library I've used a lot. > Requests has always had modules intended for its use and only its use. At some point, we tried to document them for ourselves and that led to people using them externally and then filing bugs against them. This is a perfect example of 3 adults who did not consent explicitly to the use, but Python says that we must have because we didn't hide them well enough. I don't understand the problem? It's bad that users reported bugs in internal modules?

aunderscored 3y ago
It's bad that users used modules marked explicitly as not got public use, because now those modules can't be changed without breaking user code.