6 ms·

I’ve been more and more impressed by the python runtime, which is getting pretty slick. The packaging story seems less, uh, put together. Is any consensus form

I’ve been more and more impressed by the python runtime, which is getting pretty slick. The packaging story seems less, uh, put together. Is any consensus forming on that?

rmorey 4y ago
I think poetry is emerging as a consensus somewhat, and it’s been the most reliable tool in my experience
- jerrygenser 4y ago
  dependency resolution even with caching takes way too long for larger projects, which requires a lot of setup and configuration when managing multiple inter-related projects that have many dependencies
- woodson 4y ago
  I really hope not. I used it for a while, but the “wontfix” attitude (e.g. PEP440 related issues: https://github.com/python-poetry/poetry/issues/6013 https://github.com/python-poetry/poetry/issues/6013, or the issues with pytorch cpu packages) made me go back to regular pip and use “pip freeze” for locking package versions in place.
  - DangitBobby 4y ago
    That is really unfortunate.
  - tomnipotent 4y ago
    The PEP440 issue you linked to is a positive for Poetry, not a negative. You shouldn't be installing packages into your distro-managed Python environment, that's just asking for problems.
    - woodson 4y ago
      This issue is not restricted to installing packages into your distro-managed Python environment. I had Poetry-managed environments that, after upgrading to poetry>=1.2.0 stopped working (e.g., I couldn't run "poetry update" anymore) because of these issues. I don't treat Python packaging tools as my religion. If one doesn't fulfill its purpose, I use another solution. Whether that's a positive or negative for the tool isn't my concern.
  - kortex 4y ago
    I understand your frustration, but the poetry team's response makes sense. The situation you got yourself is quite strange. System package python environments are notoriously nasty (frankly they were a mistake, but they emerged before venv, so it's hard to fault). Non-PEP440 compliant libraries are also a pain in the ass. Put both those facts together and you are playing with fire. In all likelihood you will eventually footgun yourself and end up with a python env superfund. Just use virtual envs for everything. I say this coming from over a decade of python experience and lots of time spent installing python in exotic environments.
    - woodson 4y ago
      Oh, I agree in principle, but as a user it means that something that worked up until poetry<1.2.0 broke. I understand that it's not their fault if Python distributions and packages don't comply with PEP440, and it's up to maintainers to fix it. Unfortunately, similar issues affect quite popular packages such as PyTorch, where the computation backend/version (cpu, cuda) is added as a local specifier ("1.12.1+cu116"), leading to all kinds of dependency resolver issues in poetry. Edit: Notably, not in other tools such as pip.
bombolo 4y ago
Well they removed distutils from the standard library so I guess the consensus is that they gave up?
- folkrav 4y ago
  They're just pushing for people to switch from distutils to setuptools, which is also under the PyPA.
  - bombolo 4y ago
    But must be installed separately. Is this incorrect?
    - uranusjr 4y ago
      Yes and no. It is technically a separate installation, but is installed automatically.
aniforprez 4y ago
I just use pip-tools and vanilla pip. Neither poetry, nor the rightfully maligned pipenv did anything to resolve this. Plain pip is extremely fast and has all the features I need and pip-tools' dependency resolver is fast enough that I have had no problems
- jvolkman 4y ago
  pip-tools unfortunately doesn't generate cross-platform lock files, so your locked requirements.txt may be different if you generate it on macos than if you generate it on Linux. Edited to add: both Poetry and PDM do generate cross-platform locks.
  - rbanffy 4y ago
    > may be different if you generate it on macos than if you generate it on Linux. That shouldn't happen. I wonder what could cause it. I never looked them up too closely, but same versions of Python should result in same version of packages regardless of platform.
    - DangitBobby 4y ago
      You won't run into it most of the time, but if you've got OS specific wheels you can have dependencies that require different requirements files for different OS.
    - jvolkman 4y ago
      Just generate a pip-tools lock file that includes ipython on both macos and Linux and you'll see differences. Only on macos does ipython depend on "appnope".
- kevmo314 4y ago
  Do you use virtualenv or something to manage different environments? That's the achilles heel of vanilla pip for me, having everything pollute global installs is not so cute.
rowanseymour 4y ago
We converted all our projects to Poetry over a year ago and don't have any major complaints. Also nice moving all tool configs into a single pyproject.toml file.
boxed 4y ago
There's stuff going on for sure. It's just a nasty problem due to backwards compatibility concerns.
dekhn 4y ago
for quick light development using pure python I pip install directly into my user dir of my system python. for real work I use conda because it manages multiple versions of python well. Then I mainly use pip to install stuff, occasionally falling back to conda or conda-forge. This has been the most reliable for me. I often have to trash my whole environment and start over after installing a few binary packaes like tensorflow.
- kortex 4y ago
  > for quick light development using pure python I pip install directly into my user dir of my system python. I'd advise against this habit, even for "light" development work. I've been bitten too many times by some library installing into the wrong env or picking up the wrong path of a library. Conda's generally a good move if you don't want to manually manage virtualenvs.
BerislavLopac 4y ago
Packaging has been increasingly improving, and I would dare to say that it is currently in a very good shape. The key problem is that of "blind scientists and an elephant": the term "packaging" means different things to different people, and details depend on specific circumstances. And Python has some quite specific details that are not present in many other languages, which adds to the complexity. Without going into all the available solutions for all possible cases, here's a list of use cases that Python packaging needs to cover: - dependency management and packaging: - abstract for libraries, concrete for applications [0] - pure Python dependencies - dependencies with binaries included - packaging format: sdist, wheel, conda - library repositories: public (PyPI), private, mirrors etc - installing the Python executable - multiple Python executables - management of (multiple) virtual environments Multiple standards have been adopted to deal with many of those cases, but many existing projects still haven't been updated and still need to be supported. Although we currently don't have a single tool that will cover every single possible use case, things are slowly but surely moving in the right direction. [0] https://caremad.io/posts/2013/07/setup-vs-requirement/ https://caremad.io/posts/2013/07/setup-vs-requirement/

I’ve been more and more impressed by the python runtime, which is getting pretty slick. The packaging story seems less, uh, put together. Is any consensus forming on that?

rmorey 4y ago

I think poetry is emerging as a consensus somewhat, and it’s been the most reliable tool in my experience

jerrygenser 4y ago
dependency resolution even with caching takes way too long for larger projects, which requires a lot of setup and configuration when managing multiple inter-related projects that have many dependencies
woodson 4y ago
I really hope not. I used it for a while, but the “wontfix” attitude (e.g. PEP440 related issues: https://github.com/python-poetry/poetry/issues/6013 https://github.com/python-poetry/poetry/issues/6013, or the issues with pytorch cpu packages) made me go back to regular pip and use “pip freeze” for locking package versions in place.
- DangitBobby 4y ago
  That is really unfortunate.
- tomnipotent 4y ago
  The PEP440 issue you linked to is a positive for Poetry, not a negative. You shouldn't be installing packages into your distro-managed Python environment, that's just asking for problems.
  - woodson 4y ago
    This issue is not restricted to installing packages into your distro-managed Python environment. I had Poetry-managed environments that, after upgrading to poetry>=1.2.0 stopped working (e.g., I couldn't run "poetry update" anymore) because of these issues. I don't treat Python packaging tools as my religion. If one doesn't fulfill its purpose, I use another solution. Whether that's a positive or negative for the tool isn't my concern.
- kortex 4y ago
  I understand your frustration, but the poetry team's response makes sense. The situation you got yourself is quite strange. System package python environments are notoriously nasty (frankly they were a mistake, but they emerged before venv, so it's hard to fault). Non-PEP440 compliant libraries are also a pain in the ass. Put both those facts together and you are playing with fire. In all likelihood you will eventually footgun yourself and end up with a python env superfund. Just use virtual envs for everything. I say this coming from over a decade of python experience and lots of time spent installing python in exotic environments.
  - woodson 4y ago
    Oh, I agree in principle, but as a user it means that something that worked up until poetry<1.2.0 broke. I understand that it's not their fault if Python distributions and packages don't comply with PEP440, and it's up to maintainers to fix it. Unfortunately, similar issues affect quite popular packages such as PyTorch, where the computation backend/version (cpu, cuda) is added as a local specifier ("1.12.1+cu116"), leading to all kinds of dependency resolver issues in poetry. Edit: Notably, not in other tools such as pip.

bombolo 4y ago

Well they removed distutils from the standard library so I guess the consensus is that they gave up?

folkrav 4y ago
They're just pushing for people to switch from distutils to setuptools, which is also under the PyPA.
- bombolo 4y ago
  But must be installed separately. Is this incorrect?
  - uranusjr 4y ago
    Yes and no. It is technically a separate installation, but is installed automatically.

aniforprez 4y ago

I just use pip-tools and vanilla pip. Neither poetry, nor the rightfully maligned pipenv did anything to resolve this. Plain pip is extremely fast and has all the features I need and pip-tools' dependency resolver is fast enough that I have had no problems

jvolkman 4y ago
pip-tools unfortunately doesn't generate cross-platform lock files, so your locked requirements.txt may be different if you generate it on macos than if you generate it on Linux. Edited to add: both Poetry and PDM do generate cross-platform locks.
- rbanffy 4y ago
  > may be different if you generate it on macos than if you generate it on Linux. That shouldn't happen. I wonder what could cause it. I never looked them up too closely, but same versions of Python should result in same version of packages regardless of platform.
  - DangitBobby 4y ago
    You won't run into it most of the time, but if you've got OS specific wheels you can have dependencies that require different requirements files for different OS.
  - jvolkman 4y ago
    Just generate a pip-tools lock file that includes ipython on both macos and Linux and you'll see differences. Only on macos does ipython depend on "appnope".
kevmo314 4y ago
Do you use virtualenv or something to manage different environments? That's the achilles heel of vanilla pip for me, having everything pollute global installs is not so cute.

rowanseymour 4y ago

We converted all our projects to Poetry over a year ago and don't have any major complaints. Also nice moving all tool configs into a single pyproject.toml file.

boxed 4y ago

There's stuff going on for sure. It's just a nasty problem due to backwards compatibility concerns.

dekhn 4y ago

for quick light development using pure python I pip install directly into my user dir of my system python. for real work I use conda because it manages multiple versions of python well. Then I mainly use pip to install stuff, occasionally falling back to conda or conda-forge. This has been the most reliable for me. I often have to trash my whole environment and start over after installing a few binary packaes like tensorflow.

kortex 4y ago
> for quick light development using pure python I pip install directly into my user dir of my system python. I'd advise against this habit, even for "light" development work. I've been bitten too many times by some library installing into the wrong env or picking up the wrong path of a library. Conda's generally a good move if you don't want to manually manage virtualenvs.

BerislavLopac 4y ago

Packaging has been increasingly improving, and I would dare to say that it is currently in a very good shape. The key problem is that of "blind scientists and an elephant": the term "packaging" means different things to different people, and details depend on specific circumstances. And Python has some quite specific details that are not present in many other languages, which adds to the complexity. Without going into all the available solutions for all possible cases, here's a list of use cases that Python packaging needs to cover: - dependency management and packaging: - abstract for libraries, concrete for applications [0] - pure Python dependencies - dependencies with binaries included - packaging format: sdist, wheel, conda - library repositories: public (PyPI), private, mirrors etc - installing the Python executable - multiple Python executables - management of (multiple) virtual environments Multiple standards have been adopted to deal with many of those cases, but many existing projects still haven't been updated and still need to be supported. Although we currently don't have a single tool that will cover every single possible use case, things are slowly but surely moving in the right direction. [0] https://caremad.io/posts/2013/07/setup-vs-requirement/ https://caremad.io/posts/2013/07/setup-vs-requirement/