5 ms·

> in what sense is pip not a package manager? It is a package manager but it lacks features that many other package managers have in Ruby, Node, Elixir, and ot

by nickjj 6y ago

> in what sense is pip not a package manager? It is a package manager but it lacks features that many other package managers have in Ruby, Node, Elixir, and other languages. For example there's no concept of a separate lock file with pip. Sure you can pip freeze your dependencies out to a file but this includes dependencies of dependencies, not just your app's top level dependencies. The frozen file is good to replicate versions across builds but it's really bad for human readability. Ideally we should have a file made for humans to define their top level dependencies (with version pinning support) and a lock file that has every dependency with exact pinned versions.

musingsole 6y ago
Isn't that effectively the result of a typical `setup.py -> pip compile -> requirements.txt` flow? The setup.py file contains a human readable designation of requirements and then `pip compile` generates a requirements.txt with all deps' (and deps of deps') versions specified.
xapata 6y ago
> Sure you can pip freeze your dependencies out to a file but this includes dependencies of dependencies, not just your app's top level dependencies. Isn't that a good thing? > no concept of a separate lock file with pip. setup.py/.cfg vs requirements.txt, no?
- baq 6y ago
  yes, but don't underestimate the power of convention. if you make pip run 'pip freeze > requirements.txt.lock' after every 'pip install whatever', you almost solve that particular problem if setup.py is configured to parse that (it isn't by default and there's no easy way to do that!)
  - xapata 6y ago
    That's the whole point of distinguishing between logical dependencies and reproducibility dependencies. I use setup.cfg to describe the logical dependencies, I supply a requirements.txt (or environment.yml, or a Dockerfile) to provide the tools necessary to create a deployable build.
- nickjj 6y ago
  > Isn't that a good thing? Yes, a very good thing. > setup.py/.cfg vs requirements.txt, no? A lot of web applications aren't proper packages in the sense that you pip install them. They end up being applications you run inside of a Python interpreter that happen to have dependencies and you kick things off by running a web app server like gunicorn or uwsgi. For a Python analogy vs what other languages do, you would end up having a requirements.txt file with your top level dependencies and when you run a pip install, it would auto-generate a separate requirements.lock file with all deps pinned to their exact versions. Then you'd commit both files to version control, but you would only ever modify your requirements.txt by hand. If a lock file is present that gets used during a pip install, otherwise it would use your requirements.txt file. The above work flow is how Ruby, Elixir and Node's package managers operate out of the box. It seems to work pretty well in practice for ensuring your top level deps are readable and your builds are deterministic. Currently there's no sane way to replicate that behavior using pip. That's partly why other Python package managers have come into existence over the years.
  - xapata 6y ago
    I don't understand the distinction you're making. Are you pip-installing or not? If not, why not? My method for deploying a web application is to have a Dockerfile which pip-installs the Python package, but I could see someone using a Makefile to pip-install from requirements.txt instead. In fact, I use `make` to run the commands in my Dockerfile.
    - nickjj 6y ago
      > Are you pip-installing or not? If not, why not? I am running a pip install -r requirements.txt when I do install new dependencies. I happen to be using Docker too, but I don't think that matters much in the end.
      - xapata 6y ago
        Docker does matter, because the Docker image should take the place of requirements.txt (your "locked" dependencies) in your deployment process. I suggest you pip-install the package, rather than the package's requirements.txt file.
vlovich123 6y ago
FWIW I had a lot of success using https://github.com/jazzband/pip-tools https://github.com/jazzband/pip-tools to have dependencies automatically managed in a virtualenv. * Basically I would have a single bash script that every `.py` entrypoint links to. * Beside that symlink is a `requirements.in` file that just lists the top-level dependencies I know about. * There's a `requirements.txt` file generated via pip-tools that lists all the dependencies with explicit version numbers. * The bash script then makes sure there's a virtual environment in that folder & the installed package list matches exactly the `requirements.txt` file (i.e. any extra packages are uninstalled, any missing/mismatched version packages are installed correctly). This was great because during development if you want to add a new dependency or change the installed version (i.e. pip-compile -U to update the dependency set), it didn't matter what the build server had & could test any diff independently & inexpensively. When developers pulled a new revision, they didn't have to muck about with the virtualenv - they could just launch the script without thinking about python dependencies. Finally, unrelated pieces of code would have their own dependency chains so there wasn't even a global project-wide set of dependencies (e.g. if 1 tool depends on component A, the other tools don't need to). I viewed the lack of `setup.py` as a good thing - deploying new versions of tools was a git push away rather than relying on chef or having users install new versions manually. This was the smoothest setup I've ever used for running python from source without adopting something like Bazel/BUCK (which add a lot of complexity for ingesting new dependencies as you can't leverage pip & they don't support running the python scripts in-place).

by nickjj 6y ago

musingsole 6y ago

Isn't that effectively the result of a typical `setup.py -> pip compile -> requirements.txt` flow? The setup.py file contains a human readable designation of requirements and then `pip compile` generates a requirements.txt with all deps' (and deps of deps') versions specified.

xapata 6y ago

> Sure you can pip freeze your dependencies out to a file but this includes dependencies of dependencies, not just your app's top level dependencies. Isn't that a good thing? > no concept of a separate lock file with pip. setup.py/.cfg vs requirements.txt, no?

baq 6y ago
yes, but don't underestimate the power of convention. if you make pip run 'pip freeze > requirements.txt.lock' after every 'pip install whatever', you almost solve that particular problem if setup.py is configured to parse that (it isn't by default and there's no easy way to do that!)
- xapata 6y ago
  That's the whole point of distinguishing between logical dependencies and reproducibility dependencies. I use setup.cfg to describe the logical dependencies, I supply a requirements.txt (or environment.yml, or a Dockerfile) to provide the tools necessary to create a deployable build.
nickjj 6y ago
> Isn't that a good thing? Yes, a very good thing. > setup.py/.cfg vs requirements.txt, no? A lot of web applications aren't proper packages in the sense that you pip install them. They end up being applications you run inside of a Python interpreter that happen to have dependencies and you kick things off by running a web app server like gunicorn or uwsgi. For a Python analogy vs what other languages do, you would end up having a requirements.txt file with your top level dependencies and when you run a pip install, it would auto-generate a separate requirements.lock file with all deps pinned to their exact versions. Then you'd commit both files to version control, but you would only ever modify your requirements.txt by hand. If a lock file is present that gets used during a pip install, otherwise it would use your requirements.txt file. The above work flow is how Ruby, Elixir and Node's package managers operate out of the box. It seems to work pretty well in practice for ensuring your top level deps are readable and your builds are deterministic. Currently there's no sane way to replicate that behavior using pip. That's partly why other Python package managers have come into existence over the years.
- xapata 6y ago
  I don't understand the distinction you're making. Are you pip-installing or not? If not, why not? My method for deploying a web application is to have a Dockerfile which pip-installs the Python package, but I could see someone using a Makefile to pip-install from requirements.txt instead. In fact, I use `make` to run the commands in my Dockerfile.
  - nickjj 6y ago
    > Are you pip-installing or not? If not, why not? I am running a pip install -r requirements.txt when I do install new dependencies. I happen to be using Docker too, but I don't think that matters much in the end.
    - xapata 6y ago
      Docker does matter, because the Docker image should take the place of requirements.txt (your "locked" dependencies) in your deployment process. I suggest you pip-install the package, rather than the package's requirements.txt file.

vlovich123 6y ago

FWIW I had a lot of success using https://github.com/jazzband/pip-tools https://github.com/jazzband/pip-tools to have dependencies automatically managed in a virtualenv. * Basically I would have a single bash script that every `.py` entrypoint links to. * Beside that symlink is a `requirements.in` file that just lists the top-level dependencies I know about. * There's a `requirements.txt` file generated via pip-tools that lists all the dependencies with explicit version numbers. * The bash script then makes sure there's a virtual environment in that folder & the installed package list matches exactly the `requirements.txt` file (i.e. any extra packages are uninstalled, any missing/mismatched version packages are installed correctly). This was great because during development if you want to add a new dependency or change the installed version (i.e. pip-compile -U to update the dependency set), it didn't matter what the build server had & could test any diff independently & inexpensively. When developers pulled a new revision, they didn't have to muck about with the virtualenv - they could just launch the script without thinking about python dependencies. Finally, unrelated pieces of code would have their own dependency chains so there wasn't even a global project-wide set of dependencies (e.g. if 1 tool depends on component A, the other tools don't need to). I viewed the lack of `setup.py` as a good thing - deploying new versions of tools was a git push away rather than relying on chef or having users install new versions manually. This was the smoothest setup I've ever used for running python from source without adopting something like Bazel/BUCK (which add a lot of complexity for ingesting new dependencies as you can't leverage pip & they don't support running the python scripts in-place).