4 ms·

I started a new Python project last month. I tried both Poetry and PDM but I decided not to use neither of them. PDM is currently basically one man show, and th

I started a new Python project last month. I tried both Poetry and PDM but I decided not to use neither of them. PDM is currently basically one man show, and the Poetry's doc isn't great - The doc page seems pretty but it only describes command line usages and does not tell how I can configure metadata. Most importantly Poetry does not support the standard PEP621 yet. So I stick with this setup: - Use pyenv to manage different Python versions and virtual environments. - Use the standard PEP621 specification as a high-level dependency description: https://www.python.org/dev/peps/pep-0621/#example https://www.python.org/dev/peps/pep-0621/#example - Use pip freeze > requirements.txt as a "lockfile".

postpawl 5y ago
That probably works for smaller projects without many dependencies, but it’s just going to install the sub-dependency versions that satisfy whatever comes last in the requirements file. The pip docs describe that situation here: https://pip.pypa.io/en/latest/topics/dependency-resolution/ https://pip.pypa.io/en/latest/topics/dependency-resolution/ The pip docs also suggest using pip-tools to create lock files. Pip-tools is only for creating lock files (it’s not trying to fix virtualenvs like poetry is), and it works great.
- kbumsik 5y ago
  Interesting. So use `pip-compile` instead of `pip freeze > requirements.txt`?
  - postpawl 5y ago
    Sort of - you need to make a requirements.in file that specifies only your top level dependencies, then you run pip-compile on that file to generate the requirements.txt.
    - kbumsik 5y ago
      So another requirements.in file ignoring PEP621? I'm confused... Maybe I will just try first. pip-tools seems to acknowledging PEP621 https://pip-tools.readthedocs.io/en/latest/changelog/?highlight=pyproject#id5 https://pip-tools.readthedocs.io/en/latest/changelog/?highli... https://github.com/jazzband/pip-tools/issues/1047 https://github.com/jazzband/pip-tools/issues/1047
      - Groxx 5y ago
        pip-tools predates PEP621 by ~5-8 years, depending on how you count. So yes, it "ignores" it.
- edgyquant 5y ago
  Yeah our codebase has a requirements file that takes over an hour to install with the new dependency resolver (and over 10 minutes using the deprecated resolver.) This is on a 6c/12t ryzen with 32g of ram and a gigabit connection.
  - klyrs 5y ago
    This has been my experience, too, and with a moderately simple set of dependencies. I don't understand why this is considered acceptable enough to deprecate the old resolver.
  - minusf 5y ago
    it seems like you are posting on the wrong platform, github is that --> way :} the pip team has introduced strict performance checks to ensure the new resolver was not significantly slower than the old. they solicited requirements files to weed out situations exactly like the one you describe. for example https://github.com/pypa/pip/issues/8664 https://github.com/pypa/pip/issues/8664 if you want it improved, please report it.
- Groxx 5y ago
  I believe pipenv (not pyenv!) is also a viable option for correct versioning, though I'm not actually sure whether it or pip-tools is more actively developed these days. Last I used pipenv though (~2 years ago), it was a nicer virtualenv + pip-tools combination, but had worse version resolution / less useful verbose output for no apparent reason (since iirc it shares tons of code with pip-tools). Putting that aside though: yes, 100% pip-tools or an equivalent (which pyenv is not). It's the only sane way to both freeze dependencies for reproducibility, and maintain upgradability. I've used pip-tools for years on many large, complex projects, and it has been a huge benefit every single time. And it routinely revealed significant library-incompatibility problems that teams had only luckily dodged due to not using feature X yet, because pip's resolver has been so braindead for forever.
  - aschampion 5y ago
    Pipenv quickly becomes unusably slow. As in half hour to change one dependency. Poetry is strictly better, and I say that believing poetry is not a great solution either. There aren't many good, general recommendations to give people regarding Python package management, but "not pipenv" is one of the few I'm confident in.
    - Groxx 5y ago
      Good to know, thanks :)
    - d0mine 5y ago
      How long ago did you try pipenv? I don't use it for large projects (just a few dozens dependencies), it is not fast but it works (I expect it to be just a convenience wrapper around corresponding functionality provided by pip/pip-tools (there were substantial changes in pip's dependency resolution).
RandomWorker 5y ago
I 100% agree, I’ve made over 100 projects in the past 8 years of being deep into Python, ranging from enterprise software to little hobby projects. I’ve settled on exactly the setup. It’s great when you have multiple projects on the same machine, different dependencies, versions, maybe even specially forked changed and altered versions of libraries. It’s super versatile and easy to share. No problems running my software at all. I’ve even written a script that each time I commit for git it will quickly generate a new requirements file such that it’s always up to date. Thank you for sharing.
bb88 5y ago
$ poetry init $ poetry add django $ poetry shell It's pretty simple. Check in the lock file, and then run $ poetry install to replicate it. > - Use pip freeze > requirements.txt as a "lockfile". There's lots of reasons to not do this anymore, and Dependency Hell is real, and has been for 25 years with RedHat RPM's, etc. Even if you don't want to rely upon poetry for building in prod, poetry can still export a requirements.txt file for you, so you're not locked into using poetry, but you still get to specify the high level packages you want, and let it solve the dep graph for you.
- rbanffy 5y ago
  > There's lots of reasons to not do this anymore, and Dependency Hell is real If you want to be relaxed about dependencies, you can use "pip-chill". pip install pip-chill pip-chill > requirements.txt And, if you are even more relaxed, pip-chill --no-version > requirements.txt
Caligatio 5y ago
PEP621 literally includes how Poetry lists dependencies as a synonym for the PEP's "dependencies" section. Poetry does, in fact, adhere to PEP621.
Starlevel001 5y ago
> - Use pip freeze > requirements.txt as a "lockfile". This is not and has never ever been correct. It makes it infinitely harder to install an application vs the standard `pip install -e .` which works on every package manager and avoids PYTHONPATH issues, as well as being able to publish your application to PyPI for easy installation (as simple as pip install --user app or pipx install app).

postpawl 5y ago

That probably works for smaller projects without many dependencies, but it’s just going to install the sub-dependency versions that satisfy whatever comes last in the requirements file. The pip docs describe that situation here: https://pip.pypa.io/en/latest/topics/dependency-resolution/ https://pip.pypa.io/en/latest/topics/dependency-resolution/ The pip docs also suggest using pip-tools to create lock files. Pip-tools is only for creating lock files (it’s not trying to fix virtualenvs like poetry is), and it works great.

kbumsik 5y ago
Interesting. So use `pip-compile` instead of `pip freeze > requirements.txt`?
- postpawl 5y ago
  Sort of - you need to make a requirements.in file that specifies only your top level dependencies, then you run pip-compile on that file to generate the requirements.txt.
  - kbumsik 5y ago
    So another requirements.in file ignoring PEP621? I'm confused... Maybe I will just try first. pip-tools seems to acknowledging PEP621 https://pip-tools.readthedocs.io/en/latest/changelog/?highlight=pyproject#id5 https://pip-tools.readthedocs.io/en/latest/changelog/?highli... https://github.com/jazzband/pip-tools/issues/1047 https://github.com/jazzband/pip-tools/issues/1047
    - Groxx 5y ago
      pip-tools predates PEP621 by ~5-8 years, depending on how you count. So yes, it "ignores" it.
edgyquant 5y ago
Yeah our codebase has a requirements file that takes over an hour to install with the new dependency resolver (and over 10 minutes using the deprecated resolver.) This is on a 6c/12t ryzen with 32g of ram and a gigabit connection.
- klyrs 5y ago
  This has been my experience, too, and with a moderately simple set of dependencies. I don't understand why this is considered acceptable enough to deprecate the old resolver.
- minusf 5y ago
  it seems like you are posting on the wrong platform, github is that --> way :} the pip team has introduced strict performance checks to ensure the new resolver was not significantly slower than the old. they solicited requirements files to weed out situations exactly like the one you describe. for example https://github.com/pypa/pip/issues/8664 https://github.com/pypa/pip/issues/8664 if you want it improved, please report it.
Groxx 5y ago
I believe pipenv (not pyenv!) is also a viable option for correct versioning, though I'm not actually sure whether it or pip-tools is more actively developed these days. Last I used pipenv though (~2 years ago), it was a nicer virtualenv + pip-tools combination, but had worse version resolution / less useful verbose output for no apparent reason (since iirc it shares tons of code with pip-tools). Putting that aside though: yes, 100% pip-tools or an equivalent (which pyenv is not). It's the only sane way to both freeze dependencies for reproducibility, and maintain upgradability. I've used pip-tools for years on many large, complex projects, and it has been a huge benefit every single time. And it routinely revealed significant library-incompatibility problems that teams had only luckily dodged due to not using feature X yet, because pip's resolver has been so braindead for forever.
- aschampion 5y ago
  Pipenv quickly becomes unusably slow. As in half hour to change one dependency. Poetry is strictly better, and I say that believing poetry is not a great solution either. There aren't many good, general recommendations to give people regarding Python package management, but "not pipenv" is one of the few I'm confident in.
  - Groxx 5y ago
    Good to know, thanks :)
  - d0mine 5y ago
    How long ago did you try pipenv? I don't use it for large projects (just a few dozens dependencies), it is not fast but it works (I expect it to be just a convenience wrapper around corresponding functionality provided by pip/pip-tools (there were substantial changes in pip's dependency resolution).

RandomWorker 5y ago

I 100% agree, I’ve made over 100 projects in the past 8 years of being deep into Python, ranging from enterprise software to little hobby projects. I’ve settled on exactly the setup. It’s great when you have multiple projects on the same machine, different dependencies, versions, maybe even specially forked changed and altered versions of libraries. It’s super versatile and easy to share. No problems running my software at all. I’ve even written a script that each time I commit for git it will quickly generate a new requirements file such that it’s always up to date. Thank you for sharing.

bb88 5y ago

$ poetry init $ poetry add django $ poetry shell It's pretty simple. Check in the lock file, and then run $ poetry install to replicate it. > - Use pip freeze > requirements.txt as a "lockfile". There's lots of reasons to not do this anymore, and Dependency Hell is real, and has been for 25 years with RedHat RPM's, etc. Even if you don't want to rely upon poetry for building in prod, poetry can still export a requirements.txt file for you, so you're not locked into using poetry, but you still get to specify the high level packages you want, and let it solve the dep graph for you.

rbanffy 5y ago
> There's lots of reasons to not do this anymore, and Dependency Hell is real If you want to be relaxed about dependencies, you can use "pip-chill". pip install pip-chill pip-chill > requirements.txt And, if you are even more relaxed, pip-chill --no-version > requirements.txt

Caligatio 5y ago

PEP621 literally includes how Poetry lists dependencies as a synonym for the PEP's "dependencies" section. Poetry does, in fact, adhere to PEP621.

Starlevel001 5y ago

> - Use pip freeze > requirements.txt as a "lockfile". This is not and has never ever been correct. It makes it infinitely harder to install an application vs the standard `pip install -e .` which works on every package manager and avoids PYTHONPATH issues, as well as being able to publish your application to PyPI for easy installation (as simple as pip install --user app or pipx install app).