7 ms·

I only know a bit about Python - in what sense is pip not a package manager?

by roel_v 6y ago

andi999 6y ago
in the functioning sense.
- estomagordo 6y ago
  care to elaborate?
nickjj 6y ago
> in what sense is pip not a package manager? It is a package manager but it lacks features that many other package managers have in Ruby, Node, Elixir, and other languages. For example there's no concept of a separate lock file with pip. Sure you can pip freeze your dependencies out to a file but this includes dependencies of dependencies, not just your app's top level dependencies. The frozen file is good to replicate versions across builds but it's really bad for human readability. Ideally we should have a file made for humans to define their top level dependencies (with version pinning support) and a lock file that has every dependency with exact pinned versions.
- musingsole 6y ago
  Isn't that effectively the result of a typical `setup.py -> pip compile -> requirements.txt` flow? The setup.py file contains a human readable designation of requirements and then `pip compile` generates a requirements.txt with all deps' (and deps of deps') versions specified.
- xapata 6y ago
  > Sure you can pip freeze your dependencies out to a file but this includes dependencies of dependencies, not just your app's top level dependencies. Isn't that a good thing? > no concept of a separate lock file with pip. setup.py/.cfg vs requirements.txt, no?
  - baq 6y ago
    yes, but don't underestimate the power of convention. if you make pip run 'pip freeze > requirements.txt.lock' after every 'pip install whatever', you almost solve that particular problem if setup.py is configured to parse that (it isn't by default and there's no easy way to do that!)
    - xapata 6y ago
      That's the whole point of distinguishing between logical dependencies and reproducibility dependencies. I use setup.cfg to describe the logical dependencies, I supply a requirements.txt (or environment.yml, or a Dockerfile) to provide the tools necessary to create a deployable build.
  - nickjj 6y ago
    > Isn't that a good thing? Yes, a very good thing. > setup.py/.cfg vs requirements.txt, no? A lot of web applications aren't proper packages in the sense that you pip install them. They end up being applications you run inside of a Python interpreter that happen to have dependencies and you kick things off by running a web app server like gunicorn or uwsgi. For a Python analogy vs what other languages do, you would end up having a requirements.txt file with your top level dependencies and when you run a pip install, it would auto-generate a separate requirements.lock file with all deps pinned to their exact versions. Then you'd commit both files to version control, but you would only ever modify your requirements.txt by hand. If a lock file is present that gets used during a pip install, otherwise it would use your requirements.txt file. The above work flow is how Ruby, Elixir and Node's package managers operate out of the box. It seems to work pretty well in practice for ensuring your top level deps are readable and your builds are deterministic. Currently there's no sane way to replicate that behavior using pip. That's partly why other Python package managers have come into existence over the years.
    - xapata 6y ago
      I don't understand the distinction you're making. Are you pip-installing or not? If not, why not? My method for deploying a web application is to have a Dockerfile which pip-installs the Python package, but I could see someone using a Makefile to pip-install from requirements.txt instead. In fact, I use `make` to run the commands in my Dockerfile.
- vlovich123 6y ago
  FWIW I had a lot of success using https://github.com/jazzband/pip-tools https://github.com/jazzband/pip-tools to have dependencies automatically managed in a virtualenv. * Basically I would have a single bash script that every `.py` entrypoint links to. * Beside that symlink is a `requirements.in` file that just lists the top-level dependencies I know about. * There's a `requirements.txt` file generated via pip-tools that lists all the dependencies with explicit version numbers. * The bash script then makes sure there's a virtual environment in that folder & the installed package list matches exactly the `requirements.txt` file (i.e. any extra packages are uninstalled, any missing/mismatched version packages are installed correctly). This was great because during development if you want to add a new dependency or change the installed version (i.e. pip-compile -U to update the dependency set), it didn't matter what the build server had & could test any diff independently & inexpensively. When developers pulled a new revision, they didn't have to muck about with the virtualenv - they could just launch the script without thinking about python dependencies. Finally, unrelated pieces of code would have their own dependency chains so there wasn't even a global project-wide set of dependencies (e.g. if 1 tool depends on component A, the other tools don't need to). I viewed the lack of `setup.py` as a good thing - deploying new versions of tools was a git push away rather than relying on chef or having users install new versions manually. This was the smoothest setup I've ever used for running python from source without adopting something like Bazel/BUCK (which add a lot of complexity for ingesting new dependencies as you can't leverage pip & they don't support running the python scripts in-place).
snazz 6y ago
It does a bad job of dealing with versioning conflicts and multiple projects, so Python developers resort to hacks like virtual environments to get work done. Compared to Cargo or even Go modules, it's not a great solution. It's also missing lots of features that are standard in other package managers.
- vmchale 6y ago
  > so Python developers resort to hacks like virtual environments to get work done It's really hard with many deps, it's why cabal (for instance) moved away from a global model.
- deleted 6y ago
  [deleted]
- orf 6y ago
  The need for a virtualenv has nothing to do with pip. Python only has “global” dependencies due to the way its import system works.
  - jrochkind1 6y ago
    Ruby works the same way, but bundler dependency manager solves it anyway to give you per-project dependencies not just system-wide dependencies. (I believe other well-liked dependency managers like cargo are largely based on bundler's semantics). Perhaps ruby was more "hackable" by bundler. (Bundler has now become part of ruby stdlib, but didn't start out that way, it definitely started hacking around the way the more fundamental stdlib 'rubygems' worked).
    - orf 6y ago
      so they’ve just automated the virtual environment creation then. There’s nothing about pip that is global or not global. It unzips files from pypi into a directory. Python, not pip, doesn’t really support a “node_modules” style setup. We use virtual environments (venv/) which is somewhat similar.
pmiller2 6y ago
GP comment was objecting to "stupid" package managers. Maybe they think pip is stupid, because it's unquestionably a package manager.
vlovich123 6y ago
It's a package manager in the vein of "old-school" package managers that came from Linux distros and whatnot. It maintains a global dependency chain across your entire machine. This can be good for security fixes in that you only have 1 copy of a package & everyone references it. This is not good for development because it doesn't provide a sandbox'ed environment for you to do development in (ala cargo as others have mentioned). It also causes issue if you try to install 2 packages but they rely on incompatible versions of a popular package meaning you have to choose which package you want installed. Some of this has been mitigated with virtualenv but having a project express it's packages & have that automatically reflected in the environment. Finally, Cargo to my knowledge actually lets multiple dependencies exist (even within the same project!!!) so that you can have a dependency like: dep1 -- dep3 <= v1.6 / < my awesome project> -------- dep3 >= 3.0 \ dep2 -- dep3 >= 2.0 That's not possible if you don't have the right language hooks because module resolution needs to be aware of the version of the library (i.e. when you go `import numpy`, it actually needs to be aware of the package it's being imported from to resolve that correctly). Now whether or not it's a good idea to support this kind of dependency stuff can be controversial. In practice though clearly it does cause problems the larger your codebase gets as you're more likely to have some nested dependency chain that's time-consuming to upgrade so you'd rather move faster than make sure you're only running 1 version of the dependency.
- cactus2093 6y ago
  For some reason I've run into this type of dependency chain issue many times in JS, but have never run into it in Python despite using both languages pretty heavily. Maybe because the JS ethos is to change things so quickly, if you're not making a major breaking change to your package's API every year or two then you're getting left behind (only kind of joking). Also probably because the standard library is so small in JS (or at least it used to be, and may projects want to be compatible with at least some older browsers) so the average number of dependencies that a typical library has is probably much higher than in Python. I really don't get the fuss about the global dependency management though, maybe I would change my mind if Python shipedp a great implementation of it. But I feel like the problem is already solvable in multiple ways with containers, VMs, or virtualenvs and I don't think yet another abstraction to separate environments would add much value to my day to day workflow building python apps.
zo1 6y ago
Honestly - My non-polite, personal impression of all the complaints is that they're borne from a very specific development environment. One that includes lots of dependencies similar to "npm", wants "docker style" local development, and the devs seem to think dependency-management is hard and you need complicated "semantic" versioning and version operators... but really it's just because they're working in a complex ecosystem of microservices. But, for the other 99% projects though: Most of their dependencies won't break compatibility, you'll never uncover a hard version dependency that the package manager can't solve, you'll never need to "freeze" your dependency versions and you can pretty much just rely on a semi-persistent environment with all your necessary packages installed and semi-regularly updated. Essentially smooth-sailing.

I only know a bit about Python - in what sense is pip not a package manager?

by roel_v 6y ago

I only know a bit about Python - in what sense is pip not a package manager?

andi999 6y ago

in the functioning sense.

estomagordo 6y ago
care to elaborate?

nickjj 6y ago

> in what sense is pip not a package manager? It is a package manager but it lacks features that many other package managers have in Ruby, Node, Elixir, and other languages. For example there's no concept of a separate lock file with pip. Sure you can pip freeze your dependencies out to a file but this includes dependencies of dependencies, not just your app's top level dependencies. The frozen file is good to replicate versions across builds but it's really bad for human readability. Ideally we should have a file made for humans to define their top level dependencies (with version pinning support) and a lock file that has every dependency with exact pinned versions.

musingsole 6y ago
Isn't that effectively the result of a typical `setup.py -> pip compile -> requirements.txt` flow? The setup.py file contains a human readable designation of requirements and then `pip compile` generates a requirements.txt with all deps' (and deps of deps') versions specified.
xapata 6y ago
> Sure you can pip freeze your dependencies out to a file but this includes dependencies of dependencies, not just your app's top level dependencies. Isn't that a good thing? > no concept of a separate lock file with pip. setup.py/.cfg vs requirements.txt, no?
- baq 6y ago
  yes, but don't underestimate the power of convention. if you make pip run 'pip freeze > requirements.txt.lock' after every 'pip install whatever', you almost solve that particular problem if setup.py is configured to parse that (it isn't by default and there's no easy way to do that!)
  - xapata 6y ago
    That's the whole point of distinguishing between logical dependencies and reproducibility dependencies. I use setup.cfg to describe the logical dependencies, I supply a requirements.txt (or environment.yml, or a Dockerfile) to provide the tools necessary to create a deployable build.
- nickjj 6y ago
  > Isn't that a good thing? Yes, a very good thing. > setup.py/.cfg vs requirements.txt, no? A lot of web applications aren't proper packages in the sense that you pip install them. They end up being applications you run inside of a Python interpreter that happen to have dependencies and you kick things off by running a web app server like gunicorn or uwsgi. For a Python analogy vs what other languages do, you would end up having a requirements.txt file with your top level dependencies and when you run a pip install, it would auto-generate a separate requirements.lock file with all deps pinned to their exact versions. Then you'd commit both files to version control, but you would only ever modify your requirements.txt by hand. If a lock file is present that gets used during a pip install, otherwise it would use your requirements.txt file. The above work flow is how Ruby, Elixir and Node's package managers operate out of the box. It seems to work pretty well in practice for ensuring your top level deps are readable and your builds are deterministic. Currently there's no sane way to replicate that behavior using pip. That's partly why other Python package managers have come into existence over the years.
  - xapata 6y ago
    I don't understand the distinction you're making. Are you pip-installing or not? If not, why not? My method for deploying a web application is to have a Dockerfile which pip-installs the Python package, but I could see someone using a Makefile to pip-install from requirements.txt instead. In fact, I use `make` to run the commands in my Dockerfile.
vlovich123 6y ago
FWIW I had a lot of success using https://github.com/jazzband/pip-tools https://github.com/jazzband/pip-tools to have dependencies automatically managed in a virtualenv. * Basically I would have a single bash script that every `.py` entrypoint links to. * Beside that symlink is a `requirements.in` file that just lists the top-level dependencies I know about. * There's a `requirements.txt` file generated via pip-tools that lists all the dependencies with explicit version numbers. * The bash script then makes sure there's a virtual environment in that folder & the installed package list matches exactly the `requirements.txt` file (i.e. any extra packages are uninstalled, any missing/mismatched version packages are installed correctly). This was great because during development if you want to add a new dependency or change the installed version (i.e. pip-compile -U to update the dependency set), it didn't matter what the build server had & could test any diff independently & inexpensively. When developers pulled a new revision, they didn't have to muck about with the virtualenv - they could just launch the script without thinking about python dependencies. Finally, unrelated pieces of code would have their own dependency chains so there wasn't even a global project-wide set of dependencies (e.g. if 1 tool depends on component A, the other tools don't need to). I viewed the lack of `setup.py` as a good thing - deploying new versions of tools was a git push away rather than relying on chef or having users install new versions manually. This was the smoothest setup I've ever used for running python from source without adopting something like Bazel/BUCK (which add a lot of complexity for ingesting new dependencies as you can't leverage pip & they don't support running the python scripts in-place).

snazz 6y ago

It does a bad job of dealing with versioning conflicts and multiple projects, so Python developers resort to hacks like virtual environments to get work done. Compared to Cargo or even Go modules, it's not a great solution. It's also missing lots of features that are standard in other package managers.

vmchale 6y ago
> so Python developers resort to hacks like virtual environments to get work done It's really hard with many deps, it's why cabal (for instance) moved away from a global model.
deleted 6y ago
[deleted]
orf 6y ago
The need for a virtualenv has nothing to do with pip. Python only has “global” dependencies due to the way its import system works.
- jrochkind1 6y ago
  Ruby works the same way, but bundler dependency manager solves it anyway to give you per-project dependencies not just system-wide dependencies. (I believe other well-liked dependency managers like cargo are largely based on bundler's semantics). Perhaps ruby was more "hackable" by bundler. (Bundler has now become part of ruby stdlib, but didn't start out that way, it definitely started hacking around the way the more fundamental stdlib 'rubygems' worked).
  - orf 6y ago
    so they’ve just automated the virtual environment creation then. There’s nothing about pip that is global or not global. It unzips files from pypi into a directory. Python, not pip, doesn’t really support a “node_modules” style setup. We use virtual environments (venv/) which is somewhat similar.

pmiller2 6y ago

GP comment was objecting to "stupid" package managers. Maybe they think pip is stupid, because it's unquestionably a package manager.

vlovich123 6y ago

It's a package manager in the vein of "old-school" package managers that came from Linux distros and whatnot. It maintains a global dependency chain across your entire machine. This can be good for security fixes in that you only have 1 copy of a package & everyone references it. This is not good for development because it doesn't provide a sandbox'ed environment for you to do development in (ala cargo as others have mentioned). It also causes issue if you try to install 2 packages but they rely on incompatible versions of a popular package meaning you have to choose which package you want installed. Some of this has been mitigated with virtualenv but having a project express it's packages & have that automatically reflected in the environment. Finally, Cargo to my knowledge actually lets multiple dependencies exist (even within the same project!!!) so that you can have a dependency like: dep1 -- dep3 <= v1.6 / < my awesome project> -------- dep3 >= 3.0 \ dep2 -- dep3 >= 2.0 That's not possible if you don't have the right language hooks because module resolution needs to be aware of the version of the library (i.e. when you go `import numpy`, it actually needs to be aware of the package it's being imported from to resolve that correctly). Now whether or not it's a good idea to support this kind of dependency stuff can be controversial. In practice though clearly it does cause problems the larger your codebase gets as you're more likely to have some nested dependency chain that's time-consuming to upgrade so you'd rather move faster than make sure you're only running 1 version of the dependency.

cactus2093 6y ago
For some reason I've run into this type of dependency chain issue many times in JS, but have never run into it in Python despite using both languages pretty heavily. Maybe because the JS ethos is to change things so quickly, if you're not making a major breaking change to your package's API every year or two then you're getting left behind (only kind of joking). Also probably because the standard library is so small in JS (or at least it used to be, and may projects want to be compatible with at least some older browsers) so the average number of dependencies that a typical library has is probably much higher than in Python. I really don't get the fuss about the global dependency management though, maybe I would change my mind if Python shipedp a great implementation of it. But I feel like the problem is already solvable in multiple ways with containers, VMs, or virtualenvs and I don't think yet another abstraction to separate environments would add much value to my day to day workflow building python apps.

zo1 6y ago

Honestly - My non-polite, personal impression of all the complaints is that they're borne from a very specific development environment. One that includes lots of dependencies similar to "npm", wants "docker style" local development, and the devs seem to think dependency-management is hard and you need complicated "semantic" versioning and version operators... but really it's just because they're working in a complex ecosystem of microservices. But, for the other 99% projects though: Most of their dependencies won't break compatibility, you'll never uncover a hard version dependency that the package manager can't solve, you'll never need to "freeze" your dependency versions and you can pretty much just rely on a semi-persistent environment with all your necessary packages installed and semi-regularly updated. Essentially smooth-sailing.