4 ms·

I think the unfortunate thing is the lack of unified messaging around this. There are three/four problems that seem interlinked but are not really: - Managing

I think the unfortunate thing is the lack of unified messaging around this. There are three/four problems that seem interlinked but are not really: - Managing python versions (no packaging/enviroments) Pyenv is the solution if you don't want to use a global OS version. - Environments, you don't need anything that isn't bundled with python, `python -m venv env` is the "official solution" and works really well. - Packaging your app for running in production, `pip freeze > requirements.txt` and `pip install -r requirements.txt` is the "official solution", it works well enough. - Packaging a lib for public distribution, setuptools... I don't have enough experience there but not many people need to actually use it. But again it is the "official solution" Pyenv is the only none standard tool that I would recommend. As far as I'm concerned all of these are redundant or not needed: virtualenv, pipenv, poetry pyenv-virtualenv, and pip-tools.

goodoldneon 5y ago
I wouldn't say those tools are "redundant or not needed". Tools like pipenv add features on top of pip and venv. Stuff like separating development dependencies (e.g. mypy), since you don't need that stuff at runtime. Pipenv will also move transitive dependencies into a lockfile instead of cluttering your requirements.txt (or rather Pipfile in the pipenv world).
- samwillis 5y ago
  > separating development dependencies I just have: requirements-dev.txt (dev only requirements) requirements.txt (top level requirements) requirements-freeze.txt. (all requirements frozen)
  - goodoldneon 5y ago
    How do you make sure that transitive dev dependencies end up in requirements-dev.txt? Do you manually say "dev dep X depends on transitive deps A and B, so I'll put X, A, and B in requirements-dev.txt"?
    - samwillis 5y ago
      I only put top level dependancies in requirements-dev.txt, however you are quite right that a naive `pip freeze > requirements-freeze.txt` would result in it freezing dev dependancies. It's a problem that I hope PIP will solve one day, but I like to keep tooling simple. However because it's so easy to create venvs I regally just rebuild from scratch, do a `pip freeze > requirements-freeze.txt` then install dev dependencies (so not freezing dev dependencies). In many ways I think it's better to not mutate your venv and just rebuild if making significant changes. Fortunately the packaging culture in Python isn't quite the same as with Node and you don't tend to end up with hundreds or thousands of dependancies.
      - rbanffy 5y ago
        > It's a problem that I hope PIP will solve one day, but I like to keep tooling simple. The way I've been doing is to not refer to dev dependencies in requirements.txt and then install the dependencies from requirements.txt before doing a requirements-freeze.txt. I made pip-chill to make the process a bit easier.
      - goodoldneon 5y ago
        Yea, the JS community's "moar dependencies!" culture is a big downside. But I do think that npm is great and pip should take a lot of cues from it
  - rsyring 5y ago
    pip-compile facilitates this kind of setup and will generate the frozen/lock version for you.
gjulianm 5y ago
> Packaging your app for running in production, `pip freeze > requirements.txt` and `pip install -r requirements.txt` is the "official solution", it works well enough. It doesn't though. The default format for requirements doesn't specify which requirements are top-level and which are not. Updating dependencies on a project that doesn't have this distinction is complete pain. It also doesn't work when "production" might mean different environments with different Python versions. > Packaging a lib for public distribution, setuptools... I don't have enough experience there but not many people need to actually use it. Actually, using setuptools and setup.py (or a modern equivalent) is very helpful for packaging scripts and commands, defining top-level dependencies or other things. It's much easier to distribute a wheel file to production than a full repository.
- samwillis 5y ago
  Fair enough, I actually do `requirements.txt` and `requirements-freeze.txt` for top level and then everything. But honestly it works very well.
  - gjulianm 5y ago
    Most of the tools you mentioned appeared precisely because the basics with Python don't work that well. I had a project that started similarly, and in the end I had to use to pip-tools to actually separate top-level requirements and lockfiles for different "flavors" of the package (production and dev dependencies, for example). For new projects I would seriously consider poetry because it manages those two concepts directly. There's also the problem of changing requirement files and the environment not being updated to match 1-to-1 the files, either because someone installs something to test and doesn't put it in the requirements file or because you get an updated file from a coworker and forget to run the update. Even then, pip will not delete packages that it doesn't find on the files. It's a mess once you start digging a bit, Python dependency management is extremely basic. It's no wonder why alternative tools appear, they're not redundant, they actually fix pain points in the default tools.
    - samwillis 5y ago
ellisv 5y ago
Wow - someone shares my view! I use pyenv + direnv to automatically configure my Python interpreter and virtual environment for each repo. It is pretty painless. In addition to the redundant tools you already mentioned: conda.
- disgruntledphd2 5y ago
  Conda isn't redundant if you have multiple C-level dependencies like pretty much every DS project in existence. I wish it were, but it's not. I have to agree with the OP here, Python's dependency management story is just awful.
aequitas 5y ago
> - Environments, you don't need anything that isn't bundled with python, `python -m venv env` is the "official solution" and works really well. Except when you run on Debian based systems, where you need to install the module separately for the OS Python $ python -m venv env /usr/bin/python: No module named venv
- pid-1 5y ago
  Solution: don't use Debian. IMO the way most Linux Distro package managers install stuff is absolutely awful (system wide, no isolation, shared libs, shared PATH...)
- rbanffy 5y ago
  That's one package you need to install (and that you only install if you'll be developing or testing Python packages). You'll want to install a lot more stuff (python-dev, build-essential) in that case anyway. OTOH, if you will just use the OS, you won't need venv, as Debian packagers make sure everything works out of the box without it.
jvolkman 5y ago
One problem with pip freeze (and pip-tools) is that it won't capture platform-specific dependencies. For example, ipython depends on appnope only on macos. If you build your lock file on Linux or Windows, the appnope version won't be locked. Poetry solves this to an extent by including these dependencies along with their markers in the lock file.

goodoldneon 5y ago

I wouldn't say those tools are "redundant or not needed". Tools like pipenv add features on top of pip and venv. Stuff like separating development dependencies (e.g. mypy), since you don't need that stuff at runtime. Pipenv will also move transitive dependencies into a lockfile instead of cluttering your requirements.txt (or rather Pipfile in the pipenv world).

samwillis 5y ago
> separating development dependencies I just have: requirements-dev.txt (dev only requirements) requirements.txt (top level requirements) requirements-freeze.txt. (all requirements frozen)
- goodoldneon 5y ago
  How do you make sure that transitive dev dependencies end up in requirements-dev.txt? Do you manually say "dev dep X depends on transitive deps A and B, so I'll put X, A, and B in requirements-dev.txt"?
  - samwillis 5y ago
    I only put top level dependancies in requirements-dev.txt, however you are quite right that a naive `pip freeze > requirements-freeze.txt` would result in it freezing dev dependancies. It's a problem that I hope PIP will solve one day, but I like to keep tooling simple. However because it's so easy to create venvs I regally just rebuild from scratch, do a `pip freeze > requirements-freeze.txt` then install dev dependencies (so not freezing dev dependencies). In many ways I think it's better to not mutate your venv and just rebuild if making significant changes. Fortunately the packaging culture in Python isn't quite the same as with Node and you don't tend to end up with hundreds or thousands of dependancies.
    - rbanffy 5y ago
      > It's a problem that I hope PIP will solve one day, but I like to keep tooling simple. The way I've been doing is to not refer to dev dependencies in requirements.txt and then install the dependencies from requirements.txt before doing a requirements-freeze.txt. I made pip-chill to make the process a bit easier.
    - goodoldneon 5y ago
      Yea, the JS community's "moar dependencies!" culture is a big downside. But I do think that npm is great and pip should take a lot of cues from it
- rsyring 5y ago
  pip-compile facilitates this kind of setup and will generate the frozen/lock version for you.

gjulianm 5y ago

> Packaging your app for running in production, `pip freeze > requirements.txt` and `pip install -r requirements.txt` is the "official solution", it works well enough. It doesn't though. The default format for requirements doesn't specify which requirements are top-level and which are not. Updating dependencies on a project that doesn't have this distinction is complete pain. It also doesn't work when "production" might mean different environments with different Python versions. > Packaging a lib for public distribution, setuptools... I don't have enough experience there but not many people need to actually use it. Actually, using setuptools and setup.py (or a modern equivalent) is very helpful for packaging scripts and commands, defining top-level dependencies or other things. It's much easier to distribute a wheel file to production than a full repository.

samwillis 5y ago
Fair enough, I actually do `requirements.txt` and `requirements-freeze.txt` for top level and then everything. But honestly it works very well.
- gjulianm 5y ago
  Most of the tools you mentioned appeared precisely because the basics with Python don't work that well. I had a project that started similarly, and in the end I had to use to pip-tools to actually separate top-level requirements and lockfiles for different "flavors" of the package (production and dev dependencies, for example). For new projects I would seriously consider poetry because it manages those two concepts directly. There's also the problem of changing requirement files and the environment not being updated to match 1-to-1 the files, either because someone installs something to test and doesn't put it in the requirements file or because you get an updated file from a coworker and forget to run the update. Even then, pip will not delete packages that it doesn't find on the files. It's a mess once you start digging a bit, Python dependency management is extremely basic. It's no wonder why alternative tools appear, they're not redundant, they actually fix pain points in the default tools.
  - samwillis 5y ago

ellisv 5y ago

Wow - someone shares my view! I use pyenv + direnv to automatically configure my Python interpreter and virtual environment for each repo. It is pretty painless. In addition to the redundant tools you already mentioned: conda.

disgruntledphd2 5y ago
Conda isn't redundant if you have multiple C-level dependencies like pretty much every DS project in existence. I wish it were, but it's not. I have to agree with the OP here, Python's dependency management story is just awful.

aequitas 5y ago

> - Environments, you don't need anything that isn't bundled with python, `python -m venv env` is the "official solution" and works really well. Except when you run on Debian based systems, where you need to install the module separately for the OS Python $ python -m venv env /usr/bin/python: No module named venv

pid-1 5y ago
Solution: don't use Debian. IMO the way most Linux Distro package managers install stuff is absolutely awful (system wide, no isolation, shared libs, shared PATH...)
rbanffy 5y ago
That's one package you need to install (and that you only install if you'll be developing or testing Python packages). You'll want to install a lot more stuff (python-dev, build-essential) in that case anyway. OTOH, if you will just use the OS, you won't need venv, as Debian packagers make sure everything works out of the box without it.

jvolkman 5y ago

One problem with pip freeze (and pip-tools) is that it won't capture platform-specific dependencies. For example, ipython depends on appnope only on macos. If you build your lock file on Linux or Windows, the appnope version won't be locked. Poetry solves this to an extent by including these dependencies along with their markers in the lock file.