4 ms·

Slightly off-topic, but the fact that this script even needs a package manager in a language with a standard library as large as Python is pretty shocking. Maki

by ratorx 2y ago

Slightly off-topic, but the fact that this script even needs a package manager in a language with a standard library as large as Python is pretty shocking. Making an HTTP request js pretty basic stuff for a scripting language, you shouldn’t need or want a library for it. And I’m not blaming the author, the standard library docs even recommend using a third party library (albeit not the one the author is using) on the closest equivalent (urllib.request)! > The Requests package is recommended for a higher-level HTTP client interface. Especially for a language that has not cared too much about backwards compatibility historically, having an ergonomic HTTP client seems like table stakes.

gorgoiler 2y ago
It has two! — http.client and urllib.request — and they are really usable. Lots of people just like requests though as an alternative, or for historical reasons, or because of some particular aspect of its ergonomics, or to have a feature they’d rather have implemented for them than have to write in their own calling code. At this stage it’s like using jQuery just to find an element by css selector (instead of just using document.querySelector.)
- ratorx 2y ago
  Sure historical popularity is a good reason for people who are already familiar with it to keep using it. That is not really an excuse for why the standard library docs for the clients you mentioned link to requests though (especially if they were actually good, rather than just being legacy). If they really were good, why would the standard library itself suggest something else?
- WesolyKubeczek 2y ago
  Requests used to have good PR back in the day and ended up entrenched as a transitive dependency for a lot of things. Because it’s for humans, right? But recently I had to do something using raw urllib3, and you know what? It’s just as ergonomic.
  - masklinn 2y ago
    That’s pretty much irrelevant given urllib3 is a third party dependency as well.
imtringued 2y ago
They could have used a database driver for msql, postgresql or mongodb for a more realistic example (very common for sysadmin type scripts that are only used once and then thrown away) and your complaint would be invalid, but then you'd have to set up the database and the example would no longer be fit for a quick blog post that gives you the opportunity to just copy paste the code and run it for yourself.
- ratorx 2y ago
  Well, the “this script needs package manager part”. The rest of my comment about the state of the HTTP client in Python would still be valid (but I probably wouldn’t have discovered it).
mkesper 2y ago
The standard library does not give you a possibility to do async HTTP requests, that's what httpx does. As Python still heavily relies on async this is really a bummer.
- masklinn 2y ago
  There’s absolutely no need for async http here. The script does one http request at the top of main. And a trivial one too (just a simple GET). response = urlopen(url) return json.load(response) is what they’re saving themselves from.
  - fullstop 2y ago
    In that case, sure, but if you have an entire async framework you don't want that blocking call. For as much as Python is embracing async / coroutines, I'm surprised that their http functions do not support it yet.
    - masklinn 2y ago
      > In that case, sure, but if you have an entire async framework you don't want that blocking call. What “entire async framework”, do you mean asyncio or some other third party library? In the former case, are you using it just to feel cool like TFA? > For as much as Python is embracing async / coroutines, I'm surprised that their http functions do not support it yet. asyncio doesn’t even support async file io.
      - fullstop 2y ago
        > What “entire async framework”, do you mean asyncio or some other third party library? In the former case, are you using it just to feel cool like TFA? I was thinking more along the lines of a project like Home Assistant. For my personal stuff I have been using AnyIO. > asyncio doesn’t even support async file io. many operating systems do not support async file io to begin with.
diggan 2y ago
> Making an HTTP request js pretty basic stuff for a scripting language, you shouldn’t need or want a library for it. Sometimes languages/runtimes move slowly :) Speaking as a JS developer, this is how we made requests for a long time (before .fetch), inside the browser which is basically made for making requests: var xhr = new XMLHttpRequest(); xhr.open('POST', 'https://example.com', true); xhr.setRequestHeader('Content-type', 'application/x-www-form-urlencoded'); xhr.onload = function () { console.log(this.responseText); }; xhr.send('param=add_comment'); Of course, we quickly wanted a library for it, most of us ended up using jQuery.get() et al before it wasn't comfortable up until .fetch appeared (or various npm libraries, if you were an early nodejs adopter)
- ratorx 2y ago
  I don’t think it’s just slowness or stability. The original release of requests was in 2011 and the standard library module (urllib.request) was added in Python 3.3 in 2012.
  - masklinn 2y ago
    It’s way older than that. It used to live in urllib2, which dates back to at least Python 2.1, released in April 2001.
- d4mi3n 2y ago
  This takes me back. I'm glad `fetch` has become the canonical way to do this. XHR was a new capability at the time, but back then we were just starting to learn about all the nasty things people could do by maliciously issuing XHR requests and/or loading random executables onto the page. Clickjacking was all the rage and nothing equivalent to Content Security Policy existed at the time.
masklinn 2y ago
requests is really useful for non-trivial http requests (especially as urllib has terrible defaults around REST style interactions). But here all the script is going is a trivial GET, that’s urllib.request.urlopen(url)
jgalt212 2y ago
I agree. Requests is an embarrassment and indictment of the Python standard library. And so are dataclasses. They just should have subsumed attrs.
deleted 2y ago
[deleted]
zahlman 2y ago
>And I’m not blaming the author, the standard library docs even recommend using a third party library (albeit not the one the author is using) on the closest equivalent (urllib.request)! For perspective: urllib has existed since at least as 1.4 (released in 1996), as long as python.org's archive goes back (https://docs.python.org/release/1.4/lib/node113.html#SECTION001220000000000000000 https://docs.python.org/release/1.4/lib/node113.html#SECTION...). Requests dates to 2011. httpx (the author's choice) has a 0.0.1 release from 2015, but effectively didn't exist until 2019 and is still zerover after a failed 1.0.0 prerelease in 2021. Python can't be sanely compared to the modern package-manager-based upstarts because it's literally not from that generation. When Python came out, the idea of versioning the language (not referring to a year some standards document was published) was, as far as I can tell, kinda novel. Python is older than Java, Applescript, and VB; over twice as old as Go; and over three times as old as Swift. >Especially for a language that has not cared too much about backwards compatibility historically It's always confused me that people actually see things this way. In my view, excessive concern for compatibility has severely inhibited Python (and especially packaging, if you want to include that despite being technically third-party) from fixing real problems. People switching over to 3.x should have been much faster; the breaking changes were unambiguously for the better and could not have been done in non-breaking ways. There are tons of things the developers refuse to remove from the standard library that they would never even remotely consider adding today if they weren't already there - typically citing "maintenance burden" for even the simplest things. Trying to get anything added is a nightmare: even if you convince everyone it looks like a good idea, you'll invariably asked to prove interest by implementing it yourself (who's to say all the good ideas come from programmers?) and putting it on PyPI. (I was once told this myself even though I was proposing a method on a builtin. Incidentally, I learned those can be patched in CPython, thanks to a hack involving the GC implementation.) Then, even if you somehow manage to get people to notice you, and everyone likes it, now there is suddenly no reason to add it; after all, you're in a better position to maintain it externally, since it can be versioned separately. If I were remaking Python today, the standard library would be quite minimal, although it would integrate bare necessities for packaging - APIs, not applications. (And the few things that really need to be in the standard library for a REPL to be functional and aware of the platform, would be in a namespace. They're a honking great idea. Let's do more of those.)
gabrielsroka 2y ago
I talked about that in my readme https://github.com/gabrielsroka/r https://github.com/gabrielsroka/r
rat87 2y ago
Python has historically cared about backwards compatibility. Nowadays they're finally dropping some old libraries that probably shouldn't have been in the stdlib. They're not likely to add more. Especially now that you can add dependencies to scripts so easily

by ratorx 2y ago

gorgoiler 2y ago

It has two! — http.client and urllib.request — and they are really usable. Lots of people just like requests though as an alternative, or for historical reasons, or because of some particular aspect of its ergonomics, or to have a feature they’d rather have implemented for them than have to write in their own calling code. At this stage it’s like using jQuery just to find an element by css selector (instead of just using document.querySelector.)

ratorx 2y ago
Sure historical popularity is a good reason for people who are already familiar with it to keep using it. That is not really an excuse for why the standard library docs for the clients you mentioned link to requests though (especially if they were actually good, rather than just being legacy). If they really were good, why would the standard library itself suggest something else?
WesolyKubeczek 2y ago
Requests used to have good PR back in the day and ended up entrenched as a transitive dependency for a lot of things. Because it’s for humans, right? But recently I had to do something using raw urllib3, and you know what? It’s just as ergonomic.
- masklinn 2y ago
  That’s pretty much irrelevant given urllib3 is a third party dependency as well.

imtringued 2y ago

They could have used a database driver for msql, postgresql or mongodb for a more realistic example (very common for sysadmin type scripts that are only used once and then thrown away) and your complaint would be invalid, but then you'd have to set up the database and the example would no longer be fit for a quick blog post that gives you the opportunity to just copy paste the code and run it for yourself.

ratorx 2y ago
Well, the “this script needs package manager part”. The rest of my comment about the state of the HTTP client in Python would still be valid (but I probably wouldn’t have discovered it).

mkesper 2y ago

The standard library does not give you a possibility to do async HTTP requests, that's what httpx does. As Python still heavily relies on async this is really a bummer.

masklinn 2y ago
There’s absolutely no need for async http here. The script does one http request at the top of main. And a trivial one too (just a simple GET). response = urlopen(url) return json.load(response) is what they’re saving themselves from.
- fullstop 2y ago
  In that case, sure, but if you have an entire async framework you don't want that blocking call. For as much as Python is embracing async / coroutines, I'm surprised that their http functions do not support it yet.
  - masklinn 2y ago
    > In that case, sure, but if you have an entire async framework you don't want that blocking call. What “entire async framework”, do you mean asyncio or some other third party library? In the former case, are you using it just to feel cool like TFA? > For as much as Python is embracing async / coroutines, I'm surprised that their http functions do not support it yet. asyncio doesn’t even support async file io.
    - fullstop 2y ago
      > What “entire async framework”, do you mean asyncio or some other third party library? In the former case, are you using it just to feel cool like TFA? I was thinking more along the lines of a project like Home Assistant. For my personal stuff I have been using AnyIO. > asyncio doesn’t even support async file io. many operating systems do not support async file io to begin with.

diggan 2y ago

> Making an HTTP request js pretty basic stuff for a scripting language, you shouldn’t need or want a library for it. Sometimes languages/runtimes move slowly :) Speaking as a JS developer, this is how we made requests for a long time (before .fetch), inside the browser which is basically made for making requests: var xhr = new XMLHttpRequest(); xhr.open('POST', 'https://example.com', true); xhr.setRequestHeader('Content-type', 'application/x-www-form-urlencoded'); xhr.onload = function () { console.log(this.responseText); }; xhr.send('param=add_comment'); Of course, we quickly wanted a library for it, most of us ended up using jQuery.get() et al before it wasn't comfortable up until .fetch appeared (or various npm libraries, if you were an early nodejs adopter)

ratorx 2y ago
I don’t think it’s just slowness or stability. The original release of requests was in 2011 and the standard library module (urllib.request) was added in Python 3.3 in 2012.
- masklinn 2y ago
  It’s way older than that. It used to live in urllib2, which dates back to at least Python 2.1, released in April 2001.
d4mi3n 2y ago
This takes me back. I'm glad `fetch` has become the canonical way to do this. XHR was a new capability at the time, but back then we were just starting to learn about all the nasty things people could do by maliciously issuing XHR requests and/or loading random executables onto the page. Clickjacking was all the rage and nothing equivalent to Content Security Policy existed at the time.

masklinn 2y ago

requests is really useful for non-trivial http requests (especially as urllib has terrible defaults around REST style interactions). But here all the script is going is a trivial GET, that’s urllib.request.urlopen(url)

jgalt212 2y ago

I agree. Requests is an embarrassment and indictment of the Python standard library. And so are dataclasses. They just should have subsumed attrs.

deleted 2y ago

[deleted]

zahlman 2y ago

>And I’m not blaming the author, the standard library docs even recommend using a third party library (albeit not the one the author is using) on the closest equivalent (urllib.request)! For perspective: urllib has existed since at least as 1.4 (released in 1996), as long as python.org's archive goes back (https://docs.python.org/release/1.4/lib/node113.html#SECTION001220000000000000000 https://docs.python.org/release/1.4/lib/node113.html#SECTION...). Requests dates to 2011. httpx (the author's choice) has a 0.0.1 release from 2015, but effectively didn't exist until 2019 and is still zerover after a failed 1.0.0 prerelease in 2021. Python can't be sanely compared to the modern package-manager-based upstarts because it's literally not from that generation. When Python came out, the idea of versioning the language (not referring to a year some standards document was published) was, as far as I can tell, kinda novel. Python is older than Java, Applescript, and VB; over twice as old as Go; and over three times as old as Swift. >Especially for a language that has not cared too much about backwards compatibility historically It's always confused me that people actually see things this way. In my view, excessive concern for compatibility has severely inhibited Python (and especially packaging, if you want to include that despite being technically third-party) from fixing real problems. People switching over to 3.x should have been much faster; the breaking changes were unambiguously for the better and could not have been done in non-breaking ways. There are tons of things the developers refuse to remove from the standard library that they would never even remotely consider adding today if they weren't already there - typically citing "maintenance burden" for even the simplest things. Trying to get anything added is a nightmare: even if you convince everyone it looks like a good idea, you'll invariably asked to prove interest by implementing it yourself (who's to say all the good ideas come from programmers?) and putting it on PyPI. (I was once told this myself even though I was proposing a method on a builtin. Incidentally, I learned those can be patched in CPython, thanks to a hack involving the GC implementation.) Then, even if you somehow manage to get people to notice you, and everyone likes it, now there is suddenly no reason to add it; after all, you're in a better position to maintain it externally, since it can be versioned separately. If I were remaking Python today, the standard library would be quite minimal, although it would integrate bare necessities for packaging - APIs, not applications. (And the few things that really need to be in the standard library for a REPL to be functional and aware of the platform, would be in a namespace. They're a honking great idea. Let's do more of those.)

gabrielsroka 2y ago

I talked about that in my readme https://github.com/gabrielsroka/r https://github.com/gabrielsroka/r

rat87 2y ago

Python has historically cared about backwards compatibility. Nowadays they're finally dropping some old libraries that probably shouldn't have been in the stdlib. They're not likely to add more. Especially now that you can add dependencies to scripts so easily