4 ms·

Thinking about how python is the only ecosystem where you need to deal with any of this, and how many thousands of developer-hours are spent on it

arturkane7 4y ago
Good point. I am wondering what are the specific reasons say Java or JS don’t suffer from this, and hoping for rant-free responses that don’t turn into language wars.
- IgorPartola 4y ago
  In part because Python is often used as an OS language (Debian has it in the base system and a bunch of OS subsystems use it) so if you upgrade some library at the OS level it affects more than just your own code. In part it is because Python has modules, not libraries. Modules can have executables bundled with them. In part, it’s because Python modules packages don’t exactly care about backwards compatibility. By and large, people who put out packages will keep a changelog but not keep the same API going for years. So slightly different version of the same package can give you different results. Lastly it is because Python does not have a clear definition of a project. If at the language level I could say “this is the root of my project. Do not use any OS-level libraries. Here are my library dependencies and they are to be stores in $PROJECT_ROOT/libs/“ then none of this would be required. I have rarely found that the version of Python itself was a big problem for me, but having the ability for multiple versions to be installed at the OS level combined with the above mechanism would entirely eliminate the need for virtual envs.
- nonethewiser 4y ago
  For JS, which I'm intepreti as node because I think that's the real parallel, the answer is NVN and NPM. Or alternatives such as yarn or pnpm. But even the default package manager is great compared to python. I havent tried it but my understanding is poetry is a bit more like npm.
- Kwpolska 4y ago
  I would say that one important factor is the tooling. Instead of having one or two standard tools, as seen in Java (maven/gradle), Node (npm/yarn), .NET (dotnet CLI and MSBuild/maybe FAKE), Python has a plethora of opinionated tools. venv+pip is built-in, but doesn’t handle any of the “where to put the venv” parts, and doesn’t handle making packages installable (and the classic solution is setuptools). There are many competing projects, like pipenv (which does not work at all for libraries), poetry (which uses a non-standard way of specifying metadata and dependencies, and isn’t too friendly with the rest of the packaging world), flit, hatch, etc. Why are there so many competing tools? Why can’t the community just pick one and make it a good tool for all use-cases? There’s an organization called “Python Packaging Authority”, and it maintains almost all tools mentioned in the previous paragraph (except the stdlib venv, and poetry). If they’re an authority, they should just say “___ is the way to go, we’ll add the missing use-cases to it, here’s a tool to migrate everything else to ___”. Instead, they have done things that support the proliferation of tools, like the PEP 517 (which is a standard API for package managers to talk to build tools). Another deficiency in the Python package management system is the fact that you need virtual environments to get things done, and that they are often finicky. Some of the modern tools hide the venv from you, but this gets less practical if you’re trying to use things from within scripts, or trying to point your WSGI server at the venv, or in multi-user scenarios. Some of the modern tools put it in .venv and manage it for you, but venvs can randomly break in cases the tool might not be aware of (eg. in some system package managers, upgrading Python to a new minor version would cause venvs to break, because symlink targets moved.) Node solves this by having a `node_modules` directory. .NET does stuff at build-time that most people don’t need to care about, and then just puts the .dll files for your dependencies next to the .dll with your thing, and the loader looks in the directory your code came from. Python has a proposal for `__pypackages__` [0], but it’s been there since May 2018 with not much progress. [0] https://peps.python.org/pep-0582/ https://peps.python.org/pep-0582/
  - KyeRussell 4y ago
    I’m not exactly some hip and happening ‘on the inside’ developer, but I’m aware enough of the various Big Personalities in the Python community to be quite confident in my belief that politics played a role in PyPA not taking a more consistent stance. Which is…not at all justification, certainly.
- brundolf 4y ago
  A key thing for Node is that NPM (in addition to being first-party and having a standard manifest format) installs all dependencies in the project directory, under node_modules People complain about node_modules, but the benefit is that every Node project on a system is isolated automatically and can re-download all of its dependencies trivially (after cleaning them, or after being checked out onto a new machine) A Node project's system-wide dependencies are: 1. A new-enough Node installation, and ...that's it The same is true for Go and Rust and other modern languages. Python is the odd one out.
  - melony 4y ago
    Go, Rust, and JS are unique in having package management solutions that prevents the diamond dependency problem. Most other languages suffer from the same problems as python in that they cannot have more than one version of the same package globally. The dependency resolution algorithms for Ruby/Dart/Julia are all NP class and require constraint solving which often fails to find a solution if your dependencies are complex.
    - brundolf 4y ago
      You can face the diamond problem in NPM, it just deals with it internally (usually by installing both versions under node_modules). But I don't see that as related to whether dependencies are project-specific or system-wide
    - DougBTX 4y ago
      For what it’s worth, the Poetry version solver works in the same way as Dart’s: https://github.com/python-poetry/poetry/blob/286f4ddb70394ddcf30015e924b70895616f1074/src/poetry/mixology/version_solver.py#L76 https://github.com/python-poetry/poetry/blob/286f4ddb70394dd...
- 63 4y ago
  To be fair, trying to figure out Maven and proper Java IDE integration (which is essential) can also be pretty miserable. Even though there aren't as many tool choices to deal with, the ones that exist can get really complicated and have poor documentation.
chazeon 4y ago
Well python is also the only language have a full set of numerical / scientific / data-science software that is well-maintained. Seeing the Gigs of storage used by Rust / node without building any of more complex libraries, I cannot imagine how much more they will use when doing scientific / data stuff...
maliker 4y ago
It's extremely annoying. The reality is, python can deploy or isolate any way you'd like, but because there isn't one canonical way to do it, everyone comes up with their own system. My best guess about why this is the case is because python is over 30 years old, so they've reinvented the packaging wheel (ha, .whl) four of five times, leading to massive confusion.
pauljurczak 4y ago
> python is the only ecosystem where you need to deal with any of this Coming from decades spent in C/C++ ecosystem, this is definitely not true. I would say Python users have it rather easy. A lot of Python packaging problems stems from dependencies on messy C/C++ ecosystem.
- brundolf 4y ago
  Sure, fair. I guess in my head I meant "only modern ecosystem" and/or "only major ecosystem that gets used in web development"

arturkane7 4y ago

Good point. I am wondering what are the specific reasons say Java or JS don’t suffer from this, and hoping for rant-free responses that don’t turn into language wars.

IgorPartola 4y ago
In part because Python is often used as an OS language (Debian has it in the base system and a bunch of OS subsystems use it) so if you upgrade some library at the OS level it affects more than just your own code. In part it is because Python has modules, not libraries. Modules can have executables bundled with them. In part, it’s because Python modules packages don’t exactly care about backwards compatibility. By and large, people who put out packages will keep a changelog but not keep the same API going for years. So slightly different version of the same package can give you different results. Lastly it is because Python does not have a clear definition of a project. If at the language level I could say “this is the root of my project. Do not use any OS-level libraries. Here are my library dependencies and they are to be stores in $PROJECT_ROOT/libs/“ then none of this would be required. I have rarely found that the version of Python itself was a big problem for me, but having the ability for multiple versions to be installed at the OS level combined with the above mechanism would entirely eliminate the need for virtual envs.
nonethewiser 4y ago
For JS, which I'm intepreti as node because I think that's the real parallel, the answer is NVN and NPM. Or alternatives such as yarn or pnpm. But even the default package manager is great compared to python. I havent tried it but my understanding is poetry is a bit more like npm.
Kwpolska 4y ago
I would say that one important factor is the tooling. Instead of having one or two standard tools, as seen in Java (maven/gradle), Node (npm/yarn), .NET (dotnet CLI and MSBuild/maybe FAKE), Python has a plethora of opinionated tools. venv+pip is built-in, but doesn’t handle any of the “where to put the venv” parts, and doesn’t handle making packages installable (and the classic solution is setuptools). There are many competing projects, like pipenv (which does not work at all for libraries), poetry (which uses a non-standard way of specifying metadata and dependencies, and isn’t too friendly with the rest of the packaging world), flit, hatch, etc. Why are there so many competing tools? Why can’t the community just pick one and make it a good tool for all use-cases? There’s an organization called “Python Packaging Authority”, and it maintains almost all tools mentioned in the previous paragraph (except the stdlib venv, and poetry). If they’re an authority, they should just say “___ is the way to go, we’ll add the missing use-cases to it, here’s a tool to migrate everything else to ___”. Instead, they have done things that support the proliferation of tools, like the PEP 517 (which is a standard API for package managers to talk to build tools). Another deficiency in the Python package management system is the fact that you need virtual environments to get things done, and that they are often finicky. Some of the modern tools hide the venv from you, but this gets less practical if you’re trying to use things from within scripts, or trying to point your WSGI server at the venv, or in multi-user scenarios. Some of the modern tools put it in .venv and manage it for you, but venvs can randomly break in cases the tool might not be aware of (eg. in some system package managers, upgrading Python to a new minor version would cause venvs to break, because symlink targets moved.) Node solves this by having a `node_modules` directory. .NET does stuff at build-time that most people don’t need to care about, and then just puts the .dll files for your dependencies next to the .dll with your thing, and the loader looks in the directory your code came from. Python has a proposal for `__pypackages__` [0], but it’s been there since May 2018 with not much progress. [0] https://peps.python.org/pep-0582/ https://peps.python.org/pep-0582/
- KyeRussell 4y ago
  I’m not exactly some hip and happening ‘on the inside’ developer, but I’m aware enough of the various Big Personalities in the Python community to be quite confident in my belief that politics played a role in PyPA not taking a more consistent stance. Which is…not at all justification, certainly.
brundolf 4y ago
A key thing for Node is that NPM (in addition to being first-party and having a standard manifest format) installs all dependencies in the project directory, under node_modules People complain about node_modules, but the benefit is that every Node project on a system is isolated automatically and can re-download all of its dependencies trivially (after cleaning them, or after being checked out onto a new machine) A Node project's system-wide dependencies are: 1. A new-enough Node installation, and ...that's it The same is true for Go and Rust and other modern languages. Python is the odd one out.
- melony 4y ago
  Go, Rust, and JS are unique in having package management solutions that prevents the diamond dependency problem. Most other languages suffer from the same problems as python in that they cannot have more than one version of the same package globally. The dependency resolution algorithms for Ruby/Dart/Julia are all NP class and require constraint solving which often fails to find a solution if your dependencies are complex.
  - brundolf 4y ago
    You can face the diamond problem in NPM, it just deals with it internally (usually by installing both versions under node_modules). But I don't see that as related to whether dependencies are project-specific or system-wide
  - DougBTX 4y ago
    For what it’s worth, the Poetry version solver works in the same way as Dart’s: https://github.com/python-poetry/poetry/blob/286f4ddb70394ddcf30015e924b70895616f1074/src/poetry/mixology/version_solver.py#L76 https://github.com/python-poetry/poetry/blob/286f4ddb70394dd...
63 4y ago
To be fair, trying to figure out Maven and proper Java IDE integration (which is essential) can also be pretty miserable. Even though there aren't as many tool choices to deal with, the ones that exist can get really complicated and have poor documentation.

chazeon 4y ago

Well python is also the only language have a full set of numerical / scientific / data-science software that is well-maintained. Seeing the Gigs of storage used by Rust / node without building any of more complex libraries, I cannot imagine how much more they will use when doing scientific / data stuff...

maliker 4y ago

It's extremely annoying. The reality is, python can deploy or isolate any way you'd like, but because there isn't one canonical way to do it, everyone comes up with their own system. My best guess about why this is the case is because python is over 30 years old, so they've reinvented the packaging wheel (ha, .whl) four of five times, leading to massive confusion.

pauljurczak 4y ago

> python is the only ecosystem where you need to deal with any of this Coming from decades spent in C/C++ ecosystem, this is definitely not true. I would say Python users have it rather easy. A lot of Python packaging problems stems from dependencies on messy C/C++ ecosystem.

brundolf 4y ago
Sure, fair. I guess in my head I meant "only modern ecosystem" and/or "only major ecosystem that gets used in web development"