6 ms·

Probably going to catch hate for this, but the Python package/environment management debacle is one of the things that causes me to avoid the language in my own

Probably going to catch hate for this, but the Python package/environment management debacle is one of the things that causes me to avoid the language in my own personal projects. All of the following are real tools for Python package/environment management: virtualenv; pipenv; venv; pyenv; poetry; pyenv-virtualenv; and pip-tools. There are probably more. Which one is "best" varies by who you ask and with time. If you do a bunch of Python work, you'll probably need most of them installed since different projects use different tools. The messiness of the whole situation just takes the fun out of it for me. I don't hate Python, but when I code in my free time, I'd simply prefer to use a language that doesn't have those problems.

markburns 4y ago
With libraries that alter your path at runtime. I've encountered comments on issues that implied that this wasn't a complete hack. I have had the impression that this is all just overhead in getting an environment up and running, and not taken for granted that you should be able to achieve repeatable isolation relatively simply. So I think even if one of those packaging tools worked really well, a library somewhere could undo your best efforts at isolation. I also don't hate python itself, but I wish the one way of doing things ethos applied to managing dependencies and python versions.
- fock 4y ago
  I don't understand what the problem is: 1) you install ONE version of python, more can't coexist in your environment usually 2) then you install your packages in a venv (because python provides tools to configure this) Only part 2 is a problem of the Python ecosystem, part 1 is a problem of any environment to run binary programs. It applies to C-compilers, node, ..., perl... Naturally, as most people involved with Python seemed to agree with that view, there was a lot of room for "improvement". And people got their improvement aplenty. Other people just realized that this is a general problem and nowadays instead of keeping around their /opt they now use something like spack. Which allows you to install a range of python-versions just fine. As well as C-Compilers. As well as perl-versions.
samwillis 4y ago
I think the unfortunate thing is the lack of unified messaging around this. There are three/four problems that seem interlinked but are not really: - Managing python versions (no packaging/enviroments) Pyenv is the solution if you don't want to use a global OS version. - Environments, you don't need anything that isn't bundled with python, `python -m venv env` is the "official solution" and works really well. - Packaging your app for running in production, `pip freeze > requirements.txt` and `pip install -r requirements.txt` is the "official solution", it works well enough. - Packaging a lib for public distribution, setuptools... I don't have enough experience there but not many people need to actually use it. But again it is the "official solution" Pyenv is the only none standard tool that I would recommend. As far as I'm concerned all of these are redundant or not needed: virtualenv, pipenv, poetry pyenv-virtualenv, and pip-tools.
- goodoldneon 4y ago
  I wouldn't say those tools are "redundant or not needed". Tools like pipenv add features on top of pip and venv. Stuff like separating development dependencies (e.g. mypy), since you don't need that stuff at runtime. Pipenv will also move transitive dependencies into a lockfile instead of cluttering your requirements.txt (or rather Pipfile in the pipenv world).
  - samwillis 4y ago
    > separating development dependencies I just have: requirements-dev.txt (dev only requirements) requirements.txt (top level requirements) requirements-freeze.txt. (all requirements frozen)
    - goodoldneon 4y ago
      How do you make sure that transitive dev dependencies end up in requirements-dev.txt? Do you manually say "dev dep X depends on transitive deps A and B, so I'll put X, A, and B in requirements-dev.txt"?
      - 4y ago
rodelrod 4y ago
I don't disagree with you but pyenv is just nvm for Python. You use it to manage multiple Python versions without touching the system Python, which tends to get people into trouble.
pid-1 4y ago
You don't need any of those tools. One can use virtualenvs to avoid global Python pollution, but that's it. I've used poetry, pipenv and others... Nowadays my opinion is that I want my package managers to be simple. For each app, I mantain requirements.txt, requirements.dev.txt, requirements.test.txt... I edit those files manually and only use pip freeze for pinning requirements.freeze.txt For libs, flit + pyproject.toml is a breeze.
- ellisv 4y ago
  None of the the tools are needed. You could compile Python 3.10 for each project on separately and just set your PATH to use the one you want. But pyenv makes life a lot easier.
rsyring 4y ago
Come on..it's very, very far from a debacle. It has strengths and weaknesses, sure. But a debacle? pip, pip-tools (to generate a lock file), and virtualenv are all I've needed for years now. I just started using pyenv to install Python on Ubuntu. Before that I was using the deadsnakes ppa. The tools are there to use as needed. You don't have to install or learn them until...you have to. Which may very well be never. If you avoid Python and all its benefits because of its packaging situation, it's really your loss. I just hope no one reads this and gets scared off thinking that this top voted comment matches reality. It's not that hard to get your bearings and the Python foundation has invested heavily into the packaging ecosystem the last few years. For example: https://packaging.python.org/en/latest/ https://packaging.python.org/en/latest/
- disgruntledphd2 4y ago
  Unless you have multiple C-level dependencies and/or need to install a Google created framework :(
rbanffy 4y ago
I just use pip and, sometimes, virtualenvwrapper. You just pick one that works the way you want and be happy. Having said that, I dislike the way pyenv works by adding proxy interpreters that are globally visible.
nhumrich 4y ago
Is this any different/worse than JS/node ecosystem? Npm vs yarn vs bower. Webpack vs rollup, etc.
pyuser583 4y ago
It’s a total mess. But build and runtime environment are a weak point for many, many languages.
dundarious 4y ago
Very much agreed. I briefly tried to use poetry, tox, and pytest, and even using the support packages that are supposed to make poetry and tox play nicely together, it was a massive failure. tox would claim to be running tests with python3.8, say, but actually it was still using the default system python3.10, etc. I now refuse to use any packaging tools for anything other than generating a requirements.txt, and refuse to use any version matrix tools like tox. I don't have a preference for/against pyenv for sourcing python interpreters (I'll use an AUR or scoop sometimes), or poetry or pipenv or anything else for specifying dependencies, but whatever method is used, I will always only use them to export a requirements.txt, and I will manually create my own venv for each interpreter, and run anything I want inside each venv via oneliners like: bash -c '. "$0/bin/activate" && exec "$@"' venv38 pip install --upgrade pip setuptools wheel bash -c '. "$0/bin/activate" && exec "$@"' venv38 pip install -r requirements.txt bash -c '. "$0/bin/activate" && exec "$@"' venv38 pytest --doctest-modules -v Usually I'll wrap that `bash -c` preamble into a simple with_venv.sh script, and make a similar with_venv.ps1 for Windows. Create a Makefile with targets for those commands for each interpreter you want to test, and you can run everything in knowledge that you'll always know what's happening, instead of relying on stacked magic.
batisteo 4y ago
You're missing Pyflow which is writtenin Rust. But alpha at least. But wait, there's more…

markburns 4y ago

With libraries that alter your path at runtime. I've encountered comments on issues that implied that this wasn't a complete hack. I have had the impression that this is all just overhead in getting an environment up and running, and not taken for granted that you should be able to achieve repeatable isolation relatively simply. So I think even if one of those packaging tools worked really well, a library somewhere could undo your best efforts at isolation. I also don't hate python itself, but I wish the one way of doing things ethos applied to managing dependencies and python versions.

fock 4y ago
I don't understand what the problem is: 1) you install ONE version of python, more can't coexist in your environment usually 2) then you install your packages in a venv (because python provides tools to configure this) Only part 2 is a problem of the Python ecosystem, part 1 is a problem of any environment to run binary programs. It applies to C-compilers, node, ..., perl... Naturally, as most people involved with Python seemed to agree with that view, there was a lot of room for "improvement". And people got their improvement aplenty. Other people just realized that this is a general problem and nowadays instead of keeping around their /opt they now use something like spack. Which allows you to install a range of python-versions just fine. As well as C-Compilers. As well as perl-versions.

samwillis 4y ago

I think the unfortunate thing is the lack of unified messaging around this. There are three/four problems that seem interlinked but are not really: - Managing python versions (no packaging/enviroments) Pyenv is the solution if you don't want to use a global OS version. - Environments, you don't need anything that isn't bundled with python, `python -m venv env` is the "official solution" and works really well. - Packaging your app for running in production, `pip freeze > requirements.txt` and `pip install -r requirements.txt` is the "official solution", it works well enough. - Packaging a lib for public distribution, setuptools... I don't have enough experience there but not many people need to actually use it. But again it is the "official solution" Pyenv is the only none standard tool that I would recommend. As far as I'm concerned all of these are redundant or not needed: virtualenv, pipenv, poetry pyenv-virtualenv, and pip-tools.

goodoldneon 4y ago
I wouldn't say those tools are "redundant or not needed". Tools like pipenv add features on top of pip and venv. Stuff like separating development dependencies (e.g. mypy), since you don't need that stuff at runtime. Pipenv will also move transitive dependencies into a lockfile instead of cluttering your requirements.txt (or rather Pipfile in the pipenv world).
- samwillis 4y ago
  > separating development dependencies I just have: requirements-dev.txt (dev only requirements) requirements.txt (top level requirements) requirements-freeze.txt. (all requirements frozen)
  - goodoldneon 4y ago
    How do you make sure that transitive dev dependencies end up in requirements-dev.txt? Do you manually say "dev dep X depends on transitive deps A and B, so I'll put X, A, and B in requirements-dev.txt"?
    - 4y ago

rodelrod 4y ago

I don't disagree with you but pyenv is just nvm for Python. You use it to manage multiple Python versions without touching the system Python, which tends to get people into trouble.

pid-1 4y ago

You don't need any of those tools. One can use virtualenvs to avoid global Python pollution, but that's it. I've used poetry, pipenv and others... Nowadays my opinion is that I want my package managers to be simple. For each app, I mantain requirements.txt, requirements.dev.txt, requirements.test.txt... I edit those files manually and only use pip freeze for pinning requirements.freeze.txt For libs, flit + pyproject.toml is a breeze.

ellisv 4y ago
None of the the tools are needed. You could compile Python 3.10 for each project on separately and just set your PATH to use the one you want. But pyenv makes life a lot easier.

rsyring 4y ago

Come on..it's very, very far from a debacle. It has strengths and weaknesses, sure. But a debacle? pip, pip-tools (to generate a lock file), and virtualenv are all I've needed for years now. I just started using pyenv to install Python on Ubuntu. Before that I was using the deadsnakes ppa. The tools are there to use as needed. You don't have to install or learn them until...you have to. Which may very well be never. If you avoid Python and all its benefits because of its packaging situation, it's really your loss. I just hope no one reads this and gets scared off thinking that this top voted comment matches reality. It's not that hard to get your bearings and the Python foundation has invested heavily into the packaging ecosystem the last few years. For example: https://packaging.python.org/en/latest/ https://packaging.python.org/en/latest/

disgruntledphd2 4y ago
Unless you have multiple C-level dependencies and/or need to install a Google created framework :(

rbanffy 4y ago

I just use pip and, sometimes, virtualenvwrapper. You just pick one that works the way you want and be happy. Having said that, I dislike the way pyenv works by adding proxy interpreters that are globally visible.

nhumrich 4y ago

Is this any different/worse than JS/node ecosystem? Npm vs yarn vs bower. Webpack vs rollup, etc.

pyuser583 4y ago

It’s a total mess. But build and runtime environment are a weak point for many, many languages.

dundarious 4y ago

Very much agreed. I briefly tried to use poetry, tox, and pytest, and even using the support packages that are supposed to make poetry and tox play nicely together, it was a massive failure. tox would claim to be running tests with python3.8, say, but actually it was still using the default system python3.10, etc. I now refuse to use any packaging tools for anything other than generating a requirements.txt, and refuse to use any version matrix tools like tox. I don't have a preference for/against pyenv for sourcing python interpreters (I'll use an AUR or scoop sometimes), or poetry or pipenv or anything else for specifying dependencies, but whatever method is used, I will always only use them to export a requirements.txt, and I will manually create my own venv for each interpreter, and run anything I want inside each venv via oneliners like: bash -c '. "$0/bin/activate" && exec "$@"' venv38 pip install --upgrade pip setuptools wheel bash -c '. "$0/bin/activate" && exec "$@"' venv38 pip install -r requirements.txt bash -c '. "$0/bin/activate" && exec "$@"' venv38 pytest --doctest-modules -v Usually I'll wrap that `bash -c` preamble into a simple with_venv.sh script, and make a similar with_venv.ps1 for Windows. Create a Makefile with targets for those commands for each interpreter you want to test, and you can run everything in knowledge that you'll always know what's happening, instead of relying on stacked magic.

batisteo 4y ago

You're missing Pyflow which is writtenin Rust. But alpha at least. But wait, there's more…