4 ms·

As an outsider to python, I never got how a language who got popular for being simple, elegant and readable could end up with perhaps the most complex tooling s

As an outsider to python, I never got how a language who got popular for being simple, elegant and readable could end up with perhaps the most complex tooling situation (dependencies, envs, etc). Any time I glance at the community there seems to be a new way of doing things. What caused python to go through these issues? Is there any fundamental design flaw ?

dgfitz 1y ago
People. People happened. Ideologies and strong opinions.
lvl155 1y ago
I call it the JS-syndrome.
- javchz 1y ago
  As many flaws as the npm/yarn/pnpm ecosystem has, its interoperability is waaaay better than the whole juggling act between pip, ven, poetry, Anaconda, Miniforge, and uv across projects. UV it's a step in the right direction, but legacy projects without Dockerfile can be tricky to start.
- morshu9001 1y ago
  JS did this right, in fact uv is kinda replicating the npm way. And there are other JS things I'd like Py to follow suit on.
nomel 1y ago
Seems like the flaw is that it was never a first class citizen of the language. easy_install never even made it to 1.0 Still, not bad for a bunch of mostly unpaid volunteers.
simonw 1y ago
It's mostly about age. Python has been around for 35 years now. The first version of a Python package directory was the cheeseshop (Monthy Python reference) in 2003. The earliest version of a pip-like tool was "easy_install" which - I kid you not - worked by scraping the HTML listing page of the cheeseshop and downloading zip files linked from that! More recent languages like Node.js and Rust and Go all got to create their packaging ecosystems learning from the experiences of Perl and Python before them. There is one part of Python that I consider a design flaw when it comes to packaging: the sys.modules global dictionary means it's not at all easy in Python to install two versions of the same package at the same time. This makes it really tricky if you have dependency A and dependency B both of which themselves require different versions of dependency C.
- falcor84 1y ago
  On a tangent, the somewhat related issue of Python 3 not being able to import Python 2 packages famously led Zed Shaw of "Learn Python the Hard Way" to write a rant about how Python is not Turing Complete. I checked again and apparently he removed that rant and only has a disclaimer in its place mentioning that he was obviously being hyperbolic [0]. [0] https://learnpythonthehardway.org/book/nopython3.html#the-python-3-vm-is-poorly-designed https://learnpythonthehardway.org/book/nopython3.html#the-py...
  - zahlman 1y ago
    Zed Shaw seems to have some very interesting beliefs about the 2->3 migration in general. I think it's fair to call some of it conspiratorial.
    - falcor84 1y ago
      Indeed that was a weird time, but he did eventually relent and release a version for Python 3 - https://learncodethehardway.com/client/#/product/learn-python-the-hard-way-5e-2023/ https://learncodethehardway.com/client/#/product/learn-pytho...
  - testdelacc1 1y ago
    I don’t think anyone takes Zed Shaw seriously. He’s like that uncle you see at family gatherings whom you nod along politely to.
- lelandbatey 1y ago
  I think it's also from trying to keep with the old paradigm of "libraries are installed and managed globally, potentially as linkable object files." All the languages of today gain all their improvements from: 1. Nothing should be global, but if it is it's only a cache (and caches are safe to delete since they're only used as a performance optimization) 2. You have to have extremely explicit artifact versioning, which means everything needs checksums, which means mostly reproducible builds 3. The "blessed way" is to distribute the source (or a mostly-source dist) and compile things in; the happy path is not distributing pre-computed binaries Now, everything I just said above is also wrong in many aspects or there's support for breaking any and all of the rules I just outlined, but in general, everything's built to adhere to those 3 rules nowadays. And what's crazy is that for many decades, those three rules above were considered absolutely impossible, or anti-patterns, or annoying, or a waste, etc (not without reason, but still we couldn't do it). That's what made package managers and package management so awful. That's why it was even possible to break things with `sudo pip install` vs `apt install`. Now that we've abandoned the old ways in e.g. JS/Rust/Go and adopted the three rules, all kinds of delightful side effects fall out. Tools now which re-build a full dependency tree on-disk in the project directory are the norm (it's done automatically! No annoying bits! No special flags! No manual venv!). Getting serious about checksums for artifacts means we can do proper versioning, which means we can do aggressive caching of dependencies across different projects safely, which means we don't have to _actually_ have 20 copies of every dependency, one for each repo. It all comes from the slow distributed Gentoo/FreeBSD-ification of everything and it's great!
WD-42 1y ago
If you read the initial bbs post by Guido introducing Python he describes it mostly as an alternative to bash. Basically a really nice scripting language with a decent standard library. I don’t think it was designed from the start to end up where it has. He created a genius syntax that people love.
mikepurvis 1y ago
Python 2.0 was released in October 2000. The Python ecosystem has witnessed several significant shifts in expectation as far as how software is built and delivered, from Slackware-style source builds to vendor packages to containers to uv just downloading a standalone binary archive. And the deadsnakes ppa and venvs, plus the ongoing awkwardness about whether pip should be writing stuff into usr/local or ~/.local or something else. All of this alongside the rise of GitHub and free CI builders, it being trivial to depend on lots of other packages of unknown provenance, stdlib packages being completely sidelined by stuff like requests. It’s really only in the last ten years or so that there’s been the clarity of what is a build backend vs frontend, what a lock file is and how workspace management fits into the whole picture. Distutils and setuptools are in there too. Basically, Python’s packaging has been a mess for a long time, but uv getting almost everything right all of a sudden isn’t an accident; it’s an abrupt gelling of ideas that have been in progress for two decades.
- zahlman 1y ago
  > the deadsnakes ppa Please don't use this. You need to be careful about how you place any secondary installation of Python on Ubuntu. Meanwhile, it's easy to build from source on Ubuntu and you can easily control its destination this way (by setting a prefix when you ./configure, and using make altinstall) and keep it out of Apt's way. > and venvs, plus the ongoing awkwardness about whether pip should be writing stuff into usr/local or ~/.local or something else. There is not really anything like this. You just use venvs now, which should have already been the rule since 3.3. If you need to put the package in the system environment, use an Apt package for that. If there isn't an Apt package for what you want, it shouldn't live in the system environment and also shouldn't live in your "user" site-packages — because that can still cause problems for system tools written in Python, including Apt. You only need to think about venvs as the destination, and venvs are easy to understand (and are also fundamental to how uv works). Start with https://chriswarrick.com/blog/2018/09/04/python-virtual-environments/ https://chriswarrick.com/blog/2018/09/04/python-virtual-envi... . > It’s really only in the last ten years or so that there’s been the clarity of what is a build backend vs frontend Well no; it's in that time that the idea of separating a backend and frontend emerged. Before that, it was assumed that Setuptools could just do everything. But it really couldn't, and it also led to people distributing source packages for pure-Python projects, resulting in installation doing a ton of ultimately useless work. And now that Setuptools is supposed to be focused on providing a build backend, it's mostly dead code in that workflow, but they still can't get rid of it for backwards compatibility reasons. (Incidentally, uv's provided backend only supports pure Python — they're currently recommending heavyweight tools like maturin and scikit-build-core if you need to compile something. Although in principle you can use Setuptools if you want.)
zahlman 1y ago
1. Age; there are absurd amounts of legacy cruft. Every time you have a better idea about how to do things, you have to agonize over whether you'll be allowed to remove the old way. And then using the old ways ends up indirectly causing problems for people using the new ways. 2. There is tons of code in the Python ecosystem not written in Python. One of the most popular packages, NumPy, depends on dozens of megabytes of statically compiled C and Fortran code. 3. Age again; things were designed in an era before the modern conception of a "software ecosystem", so there was nobody imagining that one day you'd be automatically fetching all the transitive dependencies and trying to build them locally, perhaps using build systems that you'd also fetch automatically. 4. GvR didn't seem to appreciate the problem fully in the early 2010s, which is where Conda came from. 5. Age again. Old designs overlooked some security issues and bootstrapping issues (this ties into all the previous points); in particular, it was (and still is) accepted that because you can include code in any language and all sorts of weird build processes, the "build the package locally" machinery needs to run arbitrary code. But that same system was then considered acceptable for pure-Python packages for many years, and the arbitrary code was even used to define metadata. And in that code, you were expected to be able to use some functionality provided by a build system written in Python, e.g. in order to locate and operate a compiler. Which then caused bootstrapping problems, because you couldn't assume that your users had a compatible version of the main build system (Setuptools) installed, and it had to be installed in the same environment as the target for package installation. So you also didn't get build isolation, etc. It was a giant mess. 5a. So they invented a system (using pyproject.toml) that would address all those problems, and also allow for competition from other build back-ends. But the other build back-end authors mostly wanted to make all-in-one tools (like Poetry, and now, er, uv); and meanwhile it was important to keep compatibility, so a bunch of defaults were chosen that enabled legacy behaviour — and ended up giving old packages little to no reason to fix anything. Oh, and also they released the specification for the "choose the build back-end system" and "here's how installers and build back-ends communicate" years before the specification for "human-friendly input for the package metadata system".
pansa2 1y ago
Python is not simple, it's a very complex language hiding behind friendly syntax. Given that, plus the breadth and complexity of its ecosystem, it makes sense that its tooling is also complex.
cgearhart 1y ago
BDFL left a long time ago. It’s not opinionated anymore. The language went from being small enough to fit in that guy’s head to a language controlled by committee that’s trying to please everyone.
- swyx 1y ago
  poor answer. guido had very little impact on the packaging mess.
  - zahlman 1y ago
    Many would say that's the problem; i.e. that he should have had more impact. Check out the history of Conda.
    - swyx 1y ago
      right but "BDFL left" is clearly the wrong thing to blame when "BDFL never cared enough" so it doesnt matter if he left
      - kstrauser 1y ago
        That’s right. And we switched from eggs to wheel’s on Guido’s watch, but that was from him being a good leader and letting other smart people do clever things on their own.
morshu9001 1y ago
It was an intentional design decision to separate package installation and management. I think that created the mess we have now. Funny thing is that decision was for modularity, but uv didn't even reuse pip.
- zahlman 1y ago
  > Funny thing is that decision was for modularity, but uv didn't even reuse pip. To be fair, that's justified by pip's overall lack of good design. Which in turn is justified by its long, organic development (I'm not trying to slight the maintainers here). But I'm making modular pieces that I hope will showcase the original idea properly. Starting with an installer, PAPER, and build backend, bbbb. These work together with `build` and `twine` (already provided by PyPA) to do the important core tasks of packaging and distribution. I'm not trying to make a "project manager", but I do plan to support PEP 751 lockfiles.
davesque 1y ago
Dependency management has always felt complicated. However, environment management I think is actually way simpler than people realize. Python basically just walks up directories trying to find its packages dir. A python "env" is just a copy of the python binary in its own directory. That's pretty much it. Basically all difficulties I've ever had with Python environments have been straightened out by going back to that basic understanding. I feel like the narrative about virtualenvs has always seemed scary but the reality really isn't.

dgfitz 1y ago

People. People happened. Ideologies and strong opinions.

lvl155 1y ago

I call it the JS-syndrome.

javchz 1y ago
As many flaws as the npm/yarn/pnpm ecosystem has, its interoperability is waaaay better than the whole juggling act between pip, ven, poetry, Anaconda, Miniforge, and uv across projects. UV it's a step in the right direction, but legacy projects without Dockerfile can be tricky to start.
morshu9001 1y ago
JS did this right, in fact uv is kinda replicating the npm way. And there are other JS things I'd like Py to follow suit on.

nomel 1y ago

Seems like the flaw is that it was never a first class citizen of the language. easy_install never even made it to 1.0 Still, not bad for a bunch of mostly unpaid volunteers.

simonw 1y ago

It's mostly about age. Python has been around for 35 years now. The first version of a Python package directory was the cheeseshop (Monthy Python reference) in 2003. The earliest version of a pip-like tool was "easy_install" which - I kid you not - worked by scraping the HTML listing page of the cheeseshop and downloading zip files linked from that! More recent languages like Node.js and Rust and Go all got to create their packaging ecosystems learning from the experiences of Perl and Python before them. There is one part of Python that I consider a design flaw when it comes to packaging: the sys.modules global dictionary means it's not at all easy in Python to install two versions of the same package at the same time. This makes it really tricky if you have dependency A and dependency B both of which themselves require different versions of dependency C.

falcor84 1y ago
On a tangent, the somewhat related issue of Python 3 not being able to import Python 2 packages famously led Zed Shaw of "Learn Python the Hard Way" to write a rant about how Python is not Turing Complete. I checked again and apparently he removed that rant and only has a disclaimer in its place mentioning that he was obviously being hyperbolic [0]. [0] https://learnpythonthehardway.org/book/nopython3.html#the-python-3-vm-is-poorly-designed https://learnpythonthehardway.org/book/nopython3.html#the-py...
- zahlman 1y ago
  Zed Shaw seems to have some very interesting beliefs about the 2->3 migration in general. I think it's fair to call some of it conspiratorial.
  - falcor84 1y ago
    Indeed that was a weird time, but he did eventually relent and release a version for Python 3 - https://learncodethehardway.com/client/#/product/learn-python-the-hard-way-5e-2023/ https://learncodethehardway.com/client/#/product/learn-pytho...
- testdelacc1 1y ago
  I don’t think anyone takes Zed Shaw seriously. He’s like that uncle you see at family gatherings whom you nod along politely to.
lelandbatey 1y ago
I think it's also from trying to keep with the old paradigm of "libraries are installed and managed globally, potentially as linkable object files." All the languages of today gain all their improvements from: 1. Nothing should be global, but if it is it's only a cache (and caches are safe to delete since they're only used as a performance optimization) 2. You have to have extremely explicit artifact versioning, which means everything needs checksums, which means mostly reproducible builds 3. The "blessed way" is to distribute the source (or a mostly-source dist) and compile things in; the happy path is not distributing pre-computed binaries Now, everything I just said above is also wrong in many aspects or there's support for breaking any and all of the rules I just outlined, but in general, everything's built to adhere to those 3 rules nowadays. And what's crazy is that for many decades, those three rules above were considered absolutely impossible, or anti-patterns, or annoying, or a waste, etc (not without reason, but still we couldn't do it). That's what made package managers and package management so awful. That's why it was even possible to break things with `sudo pip install` vs `apt install`. Now that we've abandoned the old ways in e.g. JS/Rust/Go and adopted the three rules, all kinds of delightful side effects fall out. Tools now which re-build a full dependency tree on-disk in the project directory are the norm (it's done automatically! No annoying bits! No special flags! No manual venv!). Getting serious about checksums for artifacts means we can do proper versioning, which means we can do aggressive caching of dependencies across different projects safely, which means we don't have to _actually_ have 20 copies of every dependency, one for each repo. It all comes from the slow distributed Gentoo/FreeBSD-ification of everything and it's great!

WD-42 1y ago

If you read the initial bbs post by Guido introducing Python he describes it mostly as an alternative to bash. Basically a really nice scripting language with a decent standard library. I don’t think it was designed from the start to end up where it has. He created a genius syntax that people love.

mikepurvis 1y ago

Python 2.0 was released in October 2000. The Python ecosystem has witnessed several significant shifts in expectation as far as how software is built and delivered, from Slackware-style source builds to vendor packages to containers to uv just downloading a standalone binary archive. And the deadsnakes ppa and venvs, plus the ongoing awkwardness about whether pip should be writing stuff into usr/local or ~/.local or something else. All of this alongside the rise of GitHub and free CI builders, it being trivial to depend on lots of other packages of unknown provenance, stdlib packages being completely sidelined by stuff like requests. It’s really only in the last ten years or so that there’s been the clarity of what is a build backend vs frontend, what a lock file is and how workspace management fits into the whole picture. Distutils and setuptools are in there too. Basically, Python’s packaging has been a mess for a long time, but uv getting almost everything right all of a sudden isn’t an accident; it’s an abrupt gelling of ideas that have been in progress for two decades.

zahlman 1y ago
> the deadsnakes ppa Please don't use this. You need to be careful about how you place any secondary installation of Python on Ubuntu. Meanwhile, it's easy to build from source on Ubuntu and you can easily control its destination this way (by setting a prefix when you ./configure, and using make altinstall) and keep it out of Apt's way. > and venvs, plus the ongoing awkwardness about whether pip should be writing stuff into usr/local or ~/.local or something else. There is not really anything like this. You just use venvs now, which should have already been the rule since 3.3. If you need to put the package in the system environment, use an Apt package for that. If there isn't an Apt package for what you want, it shouldn't live in the system environment and also shouldn't live in your "user" site-packages — because that can still cause problems for system tools written in Python, including Apt. You only need to think about venvs as the destination, and venvs are easy to understand (and are also fundamental to how uv works). Start with https://chriswarrick.com/blog/2018/09/04/python-virtual-environments/ https://chriswarrick.com/blog/2018/09/04/python-virtual-envi... . > It’s really only in the last ten years or so that there’s been the clarity of what is a build backend vs frontend Well no; it's in that time that the idea of separating a backend and frontend emerged. Before that, it was assumed that Setuptools could just do everything. But it really couldn't, and it also led to people distributing source packages for pure-Python projects, resulting in installation doing a ton of ultimately useless work. And now that Setuptools is supposed to be focused on providing a build backend, it's mostly dead code in that workflow, but they still can't get rid of it for backwards compatibility reasons. (Incidentally, uv's provided backend only supports pure Python — they're currently recommending heavyweight tools like maturin and scikit-build-core if you need to compile something. Although in principle you can use Setuptools if you want.)

zahlman 1y ago

1. Age; there are absurd amounts of legacy cruft. Every time you have a better idea about how to do things, you have to agonize over whether you'll be allowed to remove the old way. And then using the old ways ends up indirectly causing problems for people using the new ways. 2. There is tons of code in the Python ecosystem not written in Python. One of the most popular packages, NumPy, depends on dozens of megabytes of statically compiled C and Fortran code. 3. Age again; things were designed in an era before the modern conception of a "software ecosystem", so there was nobody imagining that one day you'd be automatically fetching all the transitive dependencies and trying to build them locally, perhaps using build systems that you'd also fetch automatically. 4. GvR didn't seem to appreciate the problem fully in the early 2010s, which is where Conda came from. 5. Age again. Old designs overlooked some security issues and bootstrapping issues (this ties into all the previous points); in particular, it was (and still is) accepted that because you can include code in any language and all sorts of weird build processes, the "build the package locally" machinery needs to run arbitrary code. But that same system was then considered acceptable for pure-Python packages for many years, and the arbitrary code was even used to define metadata. And in that code, you were expected to be able to use some functionality provided by a build system written in Python, e.g. in order to locate and operate a compiler. Which then caused bootstrapping problems, because you couldn't assume that your users had a compatible version of the main build system (Setuptools) installed, and it had to be installed in the same environment as the target for package installation. So you also didn't get build isolation, etc. It was a giant mess. 5a. So they invented a system (using pyproject.toml) that would address all those problems, and also allow for competition from other build back-ends. But the other build back-end authors mostly wanted to make all-in-one tools (like Poetry, and now, er, uv); and meanwhile it was important to keep compatibility, so a bunch of defaults were chosen that enabled legacy behaviour — and ended up giving old packages little to no reason to fix anything. Oh, and also they released the specification for the "choose the build back-end system" and "here's how installers and build back-ends communicate" years before the specification for "human-friendly input for the package metadata system".

pansa2 1y ago

Python is not simple, it's a very complex language hiding behind friendly syntax. Given that, plus the breadth and complexity of its ecosystem, it makes sense that its tooling is also complex.

cgearhart 1y ago

BDFL left a long time ago. It’s not opinionated anymore. The language went from being small enough to fit in that guy’s head to a language controlled by committee that’s trying to please everyone.

swyx 1y ago
poor answer. guido had very little impact on the packaging mess.
- zahlman 1y ago
  Many would say that's the problem; i.e. that he should have had more impact. Check out the history of Conda.
  - swyx 1y ago
    right but "BDFL left" is clearly the wrong thing to blame when "BDFL never cared enough" so it doesnt matter if he left
    - kstrauser 1y ago
      That’s right. And we switched from eggs to wheel’s on Guido’s watch, but that was from him being a good leader and letting other smart people do clever things on their own.

morshu9001 1y ago

It was an intentional design decision to separate package installation and management. I think that created the mess we have now. Funny thing is that decision was for modularity, but uv didn't even reuse pip.

zahlman 1y ago
> Funny thing is that decision was for modularity, but uv didn't even reuse pip. To be fair, that's justified by pip's overall lack of good design. Which in turn is justified by its long, organic development (I'm not trying to slight the maintainers here). But I'm making modular pieces that I hope will showcase the original idea properly. Starting with an installer, PAPER, and build backend, bbbb. These work together with `build` and `twine` (already provided by PyPA) to do the important core tasks of packaging and distribution. I'm not trying to make a "project manager", but I do plan to support PEP 751 lockfiles.

davesque 1y ago

Dependency management has always felt complicated. However, environment management I think is actually way simpler than people realize. Python basically just walks up directories trying to find its packages dir. A python "env" is just a copy of the python binary in its own directory. That's pretty much it. Basically all difficulties I've ever had with Python environments have been straightened out by going back to that basic understanding. I feel like the narrative about virtualenvs has always seemed scary but the reality really isn't.