7 ms·

That fails pretty hard at providing a rationale. Basically it says that CGI is an inefficient interface because it involves creating a new process! Even if th

by kragen 1y ago

That fails pretty hard at providing a rationale. Basically it says that CGI is an inefficient interface because it involves creating a new process! Even if that were true, "You shouldn't want to do such an inefficient thing" is very, very rarely a reasonable answer to a technical question like "How do I write a CGI script in Python?" or "How do I parse a CSV file in Python?" There are certainly some suboptimal design choices in the cgi module's calling interface, things you did a much better job of in Django, but what made them "near-impossible to fix" was that at the time everyone reading and writing PEPs considered backwards compatibility to be not a bad thing, or even a mildly good thing, but an essential thing that was worth putting up with pain for. Fixing a badly designed interface is easy if you know what it should look like and aren't constrained by backwards compatibility.

simonw 1y ago
The main rationale is earlier in the PEP: https://peps.python.org/pep-0594/#rationale https://peps.python.org/pep-0594/#rationale
- deleted 1y ago
  [deleted]
- kragen 1y ago
  Right. Isn't that insane? If I hadn't found it by means of having modules removed that my code depended on, I would have thought it was satire. That policy, and the heinous character assassination the PSF carried out against Tim Peters, mean I can no longer recommend in good conscience that anyone adopt Python.
  - hot_gril 1y ago
    Idk about the internal affairs, I just really don't like Python for web backend kind of things. It's taking them way too long to sort out parallelism and packaging, while NodeJS got both right from the start and gracefully upgraded (no 2->3 mess). Also I used Python way before JS, and I still like JS's syntax better. Especially not using whitespace for scope, which makes even less sense in a scripting language since it's hard to type that into a REPL.
    - kragen 1y ago
      Yeah, generally I feel like the indentation sensitivity was the right idea (the alternative evidently being worse compiler error messages, bugs like the `goto fail` vulnerability, and greater verbosity) but it causes real difficulties with the REPL, as well as with shell one-liners. Jupyter fixes the REPL problem, and it's a major advance in REPLs in a number of other ways, but it has real problems of its own.
    - eurleif 1y ago
      Node.js actually had no parallelism at the start, other than the ability to manually spawn new processes. Worker threads were only added in 2018 with v10.5.0, and only stabilized in 2019 with v12. What Node.js had from the start was concurrency via asynchronous IO. And before Node.js was around to be JavaScript's async IO framework, there was a robust async IO framework for Python called Twisted. Node.js was influenced by Twisted[0], and this is particularly evident in the design of its Promise abstraction (which you had to use directly when it was first released, because JavaScript didn't add the async/await keywords until years later). [0] https://nodejs.org/en/about https://nodejs.org/en/about
      - frollogaston 1y ago
        I was referring to the async io from the start, not worker threads. Other langs had their own frameworks for this, including Twisted for Python, but it really makes a difference having that stuff built-in and default.
        eurleif 1y ago
        Async IO is concurrency, not parallelism. And Node.JS is simply a framework for JavaScript, like Twisted is a framework for Python. If you compare a framework to a language, then of course the framework has more stuff built in; but that's hardly a fair comparison.
        frollogaston 1y ago
        NodeJS is a separate runtime. It's not really the same language either since import syntax differs from web JS and the standard libs are different. Or in practical terms, you can't copy a lot of browser JS code and expect it to work in NodeJS as-is. But that's beside the point. Performant web backends are way easier to deal with in NodeJS than in Python. I'm not comparing to Twisted because, even though it looks good, every Python backend I've ever seen was either plain Python or Django, which was also a mess compared to Express.
  - vodou 1y ago
    I really understand your frustration. Everyone developing in Python for a long time has felt it a bit too often when breaking changes (even between minor version updates) once again ruins the day. But I also understand that the world is not perfect. We all need to prioritize all the time. As they write in the rationale: "The team has limited resources, reduced maintenance cost frees development time for other improvements". And the cgi module is apparently even unmaintained. I guess a "batteries included" philosophy sooner or later is caught up by reality. What do you mean by "character assassination" carried out against Tim Peters? Not anything in the linked article I presume?
    - kragen 1y ago
      No, it was an unrelated scandal. I don't have my bookmarks handy at the moment, so hopefully you can find a link. As for prioritizing, I think the right choice is to deprioritize Python.
      - sitkack 1y ago
        Python is for everyone, not just the PSF Cabal. Like the Democratic party, there is a huge need for new leadership. We have all seen what a little brigading can do.
    - Kwpolska 1y ago
      He was banned for 3 months for opposing a change to the PSF bylaws that would allow the board to remove members with a simple majority vote. https://www.theregister.com/2024/08/09/core_python_developer_suspended_coc/ https://www.theregister.com/2024/08/09/core_python_developer... https://tim-one.github.io/psf/ban https://tim-one.github.io/psf/ban https://chrismcdonough.substack.com/p/the-shameful-defenestration-of-tim https://chrismcdonough.substack.com/p/the-shameful-defenestr...
      - vodou 1y ago
        Alright. Another case when "code of conducts" trumps manners or actually being a grownup. It really is a shame. Happened to a friend of mine on a rather big technical mailing list just for arguing for something that some people disagreed to. It would be nice to get back to a system based on manners and respect. That system worked for years.
  - jiggawatts 1y ago
    I don't even use Python, and even I've read Tim Peters' works and think highly of him! To have him so unceremoniously booted for upsetting a committee is absolutely insane. This is a bit like Apple firing Steve Jobs for wearing sneakers to work because it violates some dress code.
  - peter422 1y ago
    If you rely on ‘cgi’ in your Python application, you are probably fine using 3.12 until mid-2028 when it stops being maintained (and probably beyond). You guys are all really getting worked up over very little.
  - RandomThoughts3 1y ago
    [dead]
  - const_cast 1y ago
    I wouldn't recommend Python for new projects regardless. New scripts maybe, sure, but not new projects. Python has a lot of problems and it's just not really worth it because the experience it provides is not unique.
    - kragen 1y ago
      That's where I'm ending up, but I don't even want to use it for new scripts! What are its most important problems, as you see it?
      - const_cast 1y ago
        Performance is a big one. GIL is still a thorn in Python's side, although somewhat ironically CGI side-steps that. Then there's the environment configuration, which is just one big footgun. And then there's error handling. I find python scripts with ticking time bombs all the time. The maintenance burden of Python projects is just so much higher than it has any right to be. The language is neat, but not that neat. I think too often we think of performance as a sort of "tradeoff" for having a bad, unergonomic language, but that's not necessarily true. Plenty of languages have poor performance and are also a pain in the ass. We no longer live in a world where our options are C++ or scripting languages. We have mature environments with fantastic tooling. We have fast compilers with amazing error messages. We have great runtimes with more than adequate performance.
        kragen 1y ago
        That makes sense to me. Which alternatives are you favoring, especially for the kind of prototyping stuff that is Python's strong point? I do think there are some inherent tradeoffs in the space.
        const_cast 1y ago
        Go is a great choice these days pretty much exclusively due to the tooling. Turns out having a good compiler that's really, really fast is a big deal. My main issue with prototyping as a concept is that it doesn't exist in most workplaces. Prototypes quickly devolve into applications. Discarding code is risky. Your best bet IMO is choosing a language that's ergonomic in the long run, because odds are you're in for the long run. dotnet is another great choice because of the tooling and batteries included, although you do have to deal with a fairly slow compiler. Java is okay too, but Java is very restrictive and high-friction, which might not lend itself to prototyping. In the world of scripting languages, ironically PHP is a decent choice. It has better progressive typing than Python and it's reasonably safe these days. We've sort of come full circle on PHP. The downside is that PHP programmers tend to throw everything in an array, especially when going fast. That hurts readability and the IDE a lot. And then, of course, typescript and node. I don't like typescript. There's something about scripting languages with build steps that pisses me off. But, it's got a wide developer pool and it's not the worst language ever. Although there's a bit too much teeth-pulling IMO with typescript.
- stefan_ 1y ago
  It seems Python has firmly reached its "Wikipedia notability" era, with busybodies that code little but discuss much dominating and ruining all progress. They make up stuff that reads insane to anyone doing actual work, like the "maintenance burden" of the cgi module: https://github.com/python/cpython/commits/3.12/Lib/cgi.py https://github.com/python/cpython/commits/3.12/Lib/cgi.py Turns out most maintenance this thing received is the various attempts of removing it.
riedel 1y ago
Moving stuff out of the standard library seems like a reason. However, I think this all is a weird mix of arguments. IMHO new process spawning is a feature and not a bug in the use cases where CGI is used. Most of the stuff is low traffic config interfaces or remote invocable scripts. There was this trend to move stuff to fcgi. We had tons of cases of memory leaks in long running but really seldomly used stuff like mailing list servers. To me cgi is the poor man's alternative to serverless. However, I also do not really completely understand why a standard library has to support it. I have bash scripts running using the Apache CGI mod.
- kragen 1y ago
  I would have phrased it, "serverless is a marketing term for CGI scripts." I have bash CGI scripts too, though Shellshock and bash's general bug-proneness make me doubt that this was wise. There are some advantages of having the CGI protocol implemented in a library. There are common input-handling bugs the library can avoid, it means that simple CGI programs can be really simple, and it lets you switch away from CGI when desired. That said, XSS was a huge problem with really simple CGI programs, and an HTML output library that avoids that by default is more important than the input parsing—another thing absent from Python's standard library but done right by Django.
- chgs 1y ago
  I used bash cgi scripts all the time. Haven’t used a python cgi module but the main benefit of perl’s cgi module (also removed) is the query parameter parsing.
  - kragen 1y ago
    I think CGI.pm is still in CPAN? It was never in the Perl standard library, was it? As mentioned elsewhere in the thread, the query parameter parsing is still in the Python standard library, just invoked differently.
    - hwj 1y ago
      Yes, it's still in CPAN, but previously it was in the standard library.
pjmlp 1y ago
Not to mention that if efficiency is a goal, probably Python isn't the language as well, so it is a very strange argument from Python developers.
- kragen 1y ago
  It would have been a less strange argument 25 years ago, before the manycore era, when using Python involved less of a performance sacrifice. And there are still cases where Python is acceptably performant. However, the argument is from only 6 years ago, which makes it ridiculous.
  - jwatte 1y ago
    Python is still a 10x or more performance sacrifice for anything that's actually CPU throughput limited. Or, alternatively, your VM hosting cost will be 10x larger on Python, than something top of the line, if your workload is CPU throughput limited. Whether you're actually CPU limited, and whether VM hosting costs is your largest cost, is a totally different question :-)
    - kragen 1y ago
      Typically, computation you do in Python code costs you about 40× as much CPU as if you did it in C. But with Numpy I usually see only about a 4× single-core slowdown after a little optimization work. Many database-backed web services are bottlenecked on the database or text template instantiation, neither of which are really related to Python's CPU efficiency. As a side note, the most popular databases are getting only a tiny fraction of the available performance on current hardware. I wrote a couple of comments with more details about this a week ago: https://news.ycombinator.com/item?id=44408654 https://news.ycombinator.com/item?id=44408654 In the manycore world, Python's GIL makes some approaches to scaling across cores unavailable, though that is changing. But I don't think those are usually relevant to web server throughput, just (potentially) latency.
      - IgorPartola 1y ago

by kragen 1y ago

simonw 1y ago

The main rationale is earlier in the PEP: https://peps.python.org/pep-0594/#rationale https://peps.python.org/pep-0594/#rationale

deleted 1y ago
[deleted]
kragen 1y ago
Right. Isn't that insane? If I hadn't found it by means of having modules removed that my code depended on, I would have thought it was satire. That policy, and the heinous character assassination the PSF carried out against Tim Peters, mean I can no longer recommend in good conscience that anyone adopt Python.
- hot_gril 1y ago
  Idk about the internal affairs, I just really don't like Python for web backend kind of things. It's taking them way too long to sort out parallelism and packaging, while NodeJS got both right from the start and gracefully upgraded (no 2->3 mess). Also I used Python way before JS, and I still like JS's syntax better. Especially not using whitespace for scope, which makes even less sense in a scripting language since it's hard to type that into a REPL.
  - kragen 1y ago
    Yeah, generally I feel like the indentation sensitivity was the right idea (the alternative evidently being worse compiler error messages, bugs like the `goto fail` vulnerability, and greater verbosity) but it causes real difficulties with the REPL, as well as with shell one-liners. Jupyter fixes the REPL problem, and it's a major advance in REPLs in a number of other ways, but it has real problems of its own.
  - eurleif 1y ago
    Node.js actually had no parallelism at the start, other than the ability to manually spawn new processes. Worker threads were only added in 2018 with v10.5.0, and only stabilized in 2019 with v12. What Node.js had from the start was concurrency via asynchronous IO. And before Node.js was around to be JavaScript's async IO framework, there was a robust async IO framework for Python called Twisted. Node.js was influenced by Twisted[0], and this is particularly evident in the design of its Promise abstraction (which you had to use directly when it was first released, because JavaScript didn't add the async/await keywords until years later). [0] https://nodejs.org/en/about https://nodejs.org/en/about
    - frollogaston 1y ago
      I was referring to the async io from the start, not worker threads. Other langs had their own frameworks for this, including Twisted for Python, but it really makes a difference having that stuff built-in and default.
      - eurleif 1y ago
        Async IO is concurrency, not parallelism. And Node.JS is simply a framework for JavaScript, like Twisted is a framework for Python. If you compare a framework to a language, then of course the framework has more stuff built in; but that's hardly a fair comparison.
        frollogaston 1y ago
        NodeJS is a separate runtime. It's not really the same language either since import syntax differs from web JS and the standard libs are different. Or in practical terms, you can't copy a lot of browser JS code and expect it to work in NodeJS as-is. But that's beside the point. Performant web backends are way easier to deal with in NodeJS than in Python. I'm not comparing to Twisted because, even though it looks good, every Python backend I've ever seen was either plain Python or Django, which was also a mess compared to Express.
- vodou 1y ago
  I really understand your frustration. Everyone developing in Python for a long time has felt it a bit too often when breaking changes (even between minor version updates) once again ruins the day. But I also understand that the world is not perfect. We all need to prioritize all the time. As they write in the rationale: "The team has limited resources, reduced maintenance cost frees development time for other improvements". And the cgi module is apparently even unmaintained. I guess a "batteries included" philosophy sooner or later is caught up by reality. What do you mean by "character assassination" carried out against Tim Peters? Not anything in the linked article I presume?
  - kragen 1y ago
    No, it was an unrelated scandal. I don't have my bookmarks handy at the moment, so hopefully you can find a link. As for prioritizing, I think the right choice is to deprioritize Python.
    - sitkack 1y ago
      Python is for everyone, not just the PSF Cabal. Like the Democratic party, there is a huge need for new leadership. We have all seen what a little brigading can do.
  - Kwpolska 1y ago
    He was banned for 3 months for opposing a change to the PSF bylaws that would allow the board to remove members with a simple majority vote. https://www.theregister.com/2024/08/09/core_python_developer_suspended_coc/ https://www.theregister.com/2024/08/09/core_python_developer... https://tim-one.github.io/psf/ban https://tim-one.github.io/psf/ban https://chrismcdonough.substack.com/p/the-shameful-defenestration-of-tim https://chrismcdonough.substack.com/p/the-shameful-defenestr...
    - vodou 1y ago
      Alright. Another case when "code of conducts" trumps manners or actually being a grownup. It really is a shame. Happened to a friend of mine on a rather big technical mailing list just for arguing for something that some people disagreed to. It would be nice to get back to a system based on manners and respect. That system worked for years.
- jiggawatts 1y ago
  I don't even use Python, and even I've read Tim Peters' works and think highly of him! To have him so unceremoniously booted for upsetting a committee is absolutely insane. This is a bit like Apple firing Steve Jobs for wearing sneakers to work because it violates some dress code.
- peter422 1y ago
  If you rely on ‘cgi’ in your Python application, you are probably fine using 3.12 until mid-2028 when it stops being maintained (and probably beyond). You guys are all really getting worked up over very little.
- RandomThoughts3 1y ago
  [dead]
- const_cast 1y ago
  I wouldn't recommend Python for new projects regardless. New scripts maybe, sure, but not new projects. Python has a lot of problems and it's just not really worth it because the experience it provides is not unique.
  - kragen 1y ago
    That's where I'm ending up, but I don't even want to use it for new scripts! What are its most important problems, as you see it?
    - const_cast 1y ago
      Performance is a big one. GIL is still a thorn in Python's side, although somewhat ironically CGI side-steps that. Then there's the environment configuration, which is just one big footgun. And then there's error handling. I find python scripts with ticking time bombs all the time. The maintenance burden of Python projects is just so much higher than it has any right to be. The language is neat, but not that neat. I think too often we think of performance as a sort of "tradeoff" for having a bad, unergonomic language, but that's not necessarily true. Plenty of languages have poor performance and are also a pain in the ass. We no longer live in a world where our options are C++ or scripting languages. We have mature environments with fantastic tooling. We have fast compilers with amazing error messages. We have great runtimes with more than adequate performance.
      - kragen 1y ago
        That makes sense to me. Which alternatives are you favoring, especially for the kind of prototyping stuff that is Python's strong point? I do think there are some inherent tradeoffs in the space.
        const_cast 1y ago
        Go is a great choice these days pretty much exclusively due to the tooling. Turns out having a good compiler that's really, really fast is a big deal. My main issue with prototyping as a concept is that it doesn't exist in most workplaces. Prototypes quickly devolve into applications. Discarding code is risky. Your best bet IMO is choosing a language that's ergonomic in the long run, because odds are you're in for the long run. dotnet is another great choice because of the tooling and batteries included, although you do have to deal with a fairly slow compiler. Java is okay too, but Java is very restrictive and high-friction, which might not lend itself to prototyping. In the world of scripting languages, ironically PHP is a decent choice. It has better progressive typing than Python and it's reasonably safe these days. We've sort of come full circle on PHP. The downside is that PHP programmers tend to throw everything in an array, especially when going fast. That hurts readability and the IDE a lot. And then, of course, typescript and node. I don't like typescript. There's something about scripting languages with build steps that pisses me off. But, it's got a wide developer pool and it's not the worst language ever. Although there's a bit too much teeth-pulling IMO with typescript.
stefan_ 1y ago
It seems Python has firmly reached its "Wikipedia notability" era, with busybodies that code little but discuss much dominating and ruining all progress. They make up stuff that reads insane to anyone doing actual work, like the "maintenance burden" of the cgi module: https://github.com/python/cpython/commits/3.12/Lib/cgi.py https://github.com/python/cpython/commits/3.12/Lib/cgi.py Turns out most maintenance this thing received is the various attempts of removing it.

riedel 1y ago

Moving stuff out of the standard library seems like a reason. However, I think this all is a weird mix of arguments. IMHO new process spawning is a feature and not a bug in the use cases where CGI is used. Most of the stuff is low traffic config interfaces or remote invocable scripts. There was this trend to move stuff to fcgi. We had tons of cases of memory leaks in long running but really seldomly used stuff like mailing list servers. To me cgi is the poor man's alternative to serverless. However, I also do not really completely understand why a standard library has to support it. I have bash scripts running using the Apache CGI mod.

kragen 1y ago
I would have phrased it, "serverless is a marketing term for CGI scripts." I have bash CGI scripts too, though Shellshock and bash's general bug-proneness make me doubt that this was wise. There are some advantages of having the CGI protocol implemented in a library. There are common input-handling bugs the library can avoid, it means that simple CGI programs can be really simple, and it lets you switch away from CGI when desired. That said, XSS was a huge problem with really simple CGI programs, and an HTML output library that avoids that by default is more important than the input parsing—another thing absent from Python's standard library but done right by Django.
chgs 1y ago
I used bash cgi scripts all the time. Haven’t used a python cgi module but the main benefit of perl’s cgi module (also removed) is the query parameter parsing.
- kragen 1y ago
  I think CGI.pm is still in CPAN? It was never in the Perl standard library, was it? As mentioned elsewhere in the thread, the query parameter parsing is still in the Python standard library, just invoked differently.
  - hwj 1y ago
    Yes, it's still in CPAN, but previously it was in the standard library.

pjmlp 1y ago

Not to mention that if efficiency is a goal, probably Python isn't the language as well, so it is a very strange argument from Python developers.

kragen 1y ago
It would have been a less strange argument 25 years ago, before the manycore era, when using Python involved less of a performance sacrifice. And there are still cases where Python is acceptably performant. However, the argument is from only 6 years ago, which makes it ridiculous.
- jwatte 1y ago
  Python is still a 10x or more performance sacrifice for anything that's actually CPU throughput limited. Or, alternatively, your VM hosting cost will be 10x larger on Python, than something top of the line, if your workload is CPU throughput limited. Whether you're actually CPU limited, and whether VM hosting costs is your largest cost, is a totally different question :-)
  - kragen 1y ago
    Typically, computation you do in Python code costs you about 40× as much CPU as if you did it in C. But with Numpy I usually see only about a 4× single-core slowdown after a little optimization work. Many database-backed web services are bottlenecked on the database or text template instantiation, neither of which are really related to Python's CPU efficiency. As a side note, the most popular databases are getting only a tiny fraction of the available performance on current hardware. I wrote a couple of comments with more details about this a week ago: https://news.ycombinator.com/item?id=44408654 https://news.ycombinator.com/item?id=44408654 In the manycore world, Python's GIL makes some approaches to scaling across cores unavailable, though that is changing. But I don't think those are usually relevant to web server throughput, just (potentially) latency.
    - IgorPartola 1y ago