3 ms·

Honest/naive question - why has it been so hard to create a good package management system for Python? Couldn't someone do a straight port of one that has been

Honest/naive question - why has it been so hard to create a good package management system for Python? Couldn't someone do a straight port of one that has been proven to work well from a different language?

toyg 8y ago
A bit of politics, a bit of genuinely different requirements across the ecosystem. Unlike most other languages with a single packaging tool, Python serves very different niches at the same time: webdev, sysadmin, embedded, datascience, fat clients, libraries, bindings, etc etc. They all have different usecases and concepts of “packaging tool”. When a new tool emerges, it usually serves one niche better than others; the other elements of the ecosystem push back, and a new tool is built... repeat ad infinitum. This does not happen with languages where a single niche accounts for most of the use, so a broad consensus is easier to achieve. The closest to Python is Java, but that is traditionally steered top-down (but still has competing toolchains, e.g. ant/maven/gradle). Pipenv in particular has some additional problems that were completely self-inflicted (namely, a famous developer abusing his popularity to push an incomplete implementation as blessed).
cwp 8y ago
The problem is that (just about) all python packages express their dependencies with a script called setup.py. To figure out the full transitive closure of dependencies you have to walk the graph, downloading and executing setup.py scripts. First of all, that's slow. But worse, setup.py is a python script with access to the full power of python. It can crash. It can be slow. It can loop forever. It can have dependencies of its own. It can require specific versions of python. It can download stuff from the internet. It can do anything. If you wanted to implement something like npm for python, you'd have to convince all the python package maintainers to write and test package.json files for all their packages. Even if they were willing and enthusiastic about that, you'd have a chicken and egg problem because nobody could test a package.json until all their dependencies had them. Python has a such a plethora of packaging tools because people keep trying to solve the problem by writing better code. But the real problem is lack metadata. Python will always have a lousy packaging ecosystem because it relies on setup.py.

toyg 8y ago

A bit of politics, a bit of genuinely different requirements across the ecosystem. Unlike most other languages with a single packaging tool, Python serves very different niches at the same time: webdev, sysadmin, embedded, datascience, fat clients, libraries, bindings, etc etc. They all have different usecases and concepts of “packaging tool”. When a new tool emerges, it usually serves one niche better than others; the other elements of the ecosystem push back, and a new tool is built... repeat ad infinitum. This does not happen with languages where a single niche accounts for most of the use, so a broad consensus is easier to achieve. The closest to Python is Java, but that is traditionally steered top-down (but still has competing toolchains, e.g. ant/maven/gradle). Pipenv in particular has some additional problems that were completely self-inflicted (namely, a famous developer abusing his popularity to push an incomplete implementation as blessed).

cwp 8y ago

The problem is that (just about) all python packages express their dependencies with a script called setup.py. To figure out the full transitive closure of dependencies you have to walk the graph, downloading and executing setup.py scripts. First of all, that's slow. But worse, setup.py is a python script with access to the full power of python. It can crash. It can be slow. It can loop forever. It can have dependencies of its own. It can require specific versions of python. It can download stuff from the internet. It can do anything. If you wanted to implement something like npm for python, you'd have to convince all the python package maintainers to write and test package.json files for all their packages. Even if they were willing and enthusiastic about that, you'd have a chicken and egg problem because nobody could test a package.json until all their dependencies had them. Python has a such a plethora of packaging tools because people keep trying to solve the problem by writing better code. But the real problem is lack metadata. Python will always have a lousy packaging ecosystem because it relies on setup.py.