8 ms·

Genuine question: If I am starting a Python project NOW, which one do I use? I have been using pipenv for quite some time and it works great but locking speed h

by fb03 5y ago

Genuine question: If I am starting a Python project NOW, which one do I use? I have been using pipenv for quite some time and it works great but locking speed has been problematic, specially after your project grows large enough (minutes waiting for it to lock without any progress warning at all). Should I just upgrade to Poetry or should I just dive headfirst into PDM? Keep myself at Pipenv? I'm at a loss. Thanks in advance!

stavros 5y ago
Use Poetry. It does everything you need and PDM is a bit too new still.
- fb03 5y ago
  Great. I have heard "anecdata evidence" that sometimes poetry fails to install a combination of packages or something along those lines, did you find any of those shenanigans in your own experience?
  - akdor1154 5y ago
    (Not GP) Occasionally, but still use poetry. The issues happen because poetry is strict about version constraints. Pip used to be lax, so some packages could get away with poor/overly restrictive constraints.. Now pip is strict as well, and poetry is also getting a lot more common, so maintainers have had pressure to fix most of these issues.
  - nbadg 5y ago
    I've run into this a couple of times. Typically it's when starting a new project and gradually adding dependencies. By default, when adding (poetry add <dep>), poetry adds the most recent version of the library, and in pyproject.toml that becomes the minimum possible version. If you then later try to add something that depends on that library, but isn't up-to-date on the most recent version, poetry will (correctly!) error out because it fails to find a compatible solution. For example, something like this: poetry add ingredients # most recent version: 1.0 poetry add cake # requires ingredients, but <1.0 So here, poetry would add ingredients to your pyproject.toml with version = "^1.0.0" in the first step. In the second step, when you tried to add cake as a new dependency, it would say it can't find a version of cake compatible with your pyproject.toml's requirement for ingredients. The solution is to change pyproject.toml to be more permissive about the version of ingredients, resulting in it downgrading to a compatible version. Or submit an issue and/or PR with the maintainers of cake to bump up the supported versions.
    - frostming 5y ago
      In PDM, add command supports a `--unconstrained` option to relax the version constraints to find a solution. You don't need to edit the file manually.
    - andreareina 5y ago
      Another way it happens: I was stuck on python 3.6 for a while and it meant that I couldn't just poetry add pandas because the latest version isn't compatible with 3.6 and it won't look for other versions.
  - acidbaseextract 5y ago
    No shenanigans for any library-ish code with few dependencies that targets relatively modern Python. I've had one or two fundamental version conflicts with a 5+ year old application with 100+ dependencies and a decent amount of legacy stuff. They were a pain in the ass, and the sdispater's stance on not allowing overrides is a pain in the ass. We ended up forking the upstream libraries to resolve the version conflict. With all of that, poetry is amazing and a huge step forward. I'd advocate it wholeheartedly.
  - gooseyard 5y ago
    I've spent the last year or so comparing these packages. Initially, I liked poetry, but I began to run into issues related to client cert auth with private repositories. I and several other users who encountered these problems submitted pull requests to address them, but found that it was nearly impossible to get any of them reviewed, much less merged. Out of frustration, I decided to try pdm instead, and although pdm doesn't handle pushing to pypi repositories, Twine works fine for it, and I have had a pain-free experience using pdm for dependency management. I like the idea of poetry, but it's unusable if I can't get bugs fixed, and I gave up after nearly a year of trying.
    - bb88 5y ago
      > but I began to run into issues related to client cert auth with private repositories. Yeah, I think that's the worst thing about private infrastructure in 2021 is getting those certs correctly installed and seen by the software and both sides. I'm thinking that tailscale or something like it may be a better alternative in the future where authorization is at the link layer before the tcp connect can happen.
  - stavros 5y ago
    As others have said, this is related to how Poetry does dependency resolution. If you say your program uses Python "^3.8", you would expect any Python version in that range to work, but instead, ALL Python versions in that range must work. If one of your dependencies won't work with 3.9, the dependency graph cannot resolve. The author of Poetry said that this is because Poetry is meant for libraries, where this makes more sense.
    - amethyst 5y ago
      Except Poetry's recommendations/decisions around version capping doesn't make sense for libraries. https://iscinumpy.dev/post/bound-version-constraints/ https://iscinumpy.dev/post/bound-version-constraints/
- jreese 5y ago
  As a Python infra person, I would refine that to: If you're building an application, use Poetry. If you're building a library, use Flit. Use PEP621 metadata in pyproject.toml regardless. Poetry is much more focused on managing dependencies for applications than dealing with libraries that have to be used by other libraries or applications. See this deep discussion for some timely/relevant examples: https://iscinumpy.dev/post/bound-version-constraints/ https://iscinumpy.dev/post/bound-version-constraints/
  - kbumsik 5y ago
    Can we mix Poetry and PEP621 though? I tried to use project.dependencies and project.optional-dependencies fields in pyproject.toml defined by PEP621. However Poetry seems to ignore it and it tries to stick with its own tools.poetry section.
    - jreese 5y ago
      Ah, you're right. I thought Poetry had adopted PEP 621 already, but it looks like there's still a debate about if/how they want to do that. A bit ironic considering the author of Poetry was listed as an author on PEP 621, and that was accepted just over a year ago.
      - kbumsik 5y ago
        It went final this March though. So Poetry might needs more time to think about the transitioning plan. Python packaging is again in a chaos since 2021 because of the rise of standardized PEPs.
        spurgu 5y ago
        I know you didn't mean it that way, but "since 2021"..? :D I was working actively with Python (and loving it) between 2012-2015 (during which the 2 vs 3 transition/conflicts were raging) and the couple of times I've tried to get back into the groove since then (in side projects) I keep hitting walls and getting confused about how to even get the basic things running. I've heard good things about Poetry but never tried it. PDM being closer to npm/yarn sounds like an attractive option (since they're very familar), just to tinker around (not to build a complex application with specific requirements). Will give it a shot!
- zmmmmm 5y ago
  Sold everyone on using poetry a few months ago and now red faced as we have a litany of problems and time waste due to it. We are now sitting on some bleeding edge branch because specific dependencies cannot work at all without some new fangled feature and everyone wishes we were just using plain virtualenv as we had much less problems with that.
  - smohare 5y ago
    In other words, you work with people who have no clue what’s going on.
    - spurgu 5y ago
      Thanks, that's good advice!
  - zelphirkalt 5y ago
    You would also not have any real control over what gets installed with just venv. So everyone might think the grass was greener on the other side, but it might not actually be, if you value reproducible environments.
    - funstuff007 5y ago
      What you've never spent hours trying out n! orderings of requirements.txt until you get the results you want/can live with? You have not lived until you've done this. /s
      - bb88 5y ago
        Dep Hell is real.
- burgerrito 5y ago
  Is Poetry used a lot by Python devs? What package management a lot of Python devs use? I mostly code on JavaScript and (obviously) use NPM a lot, and it makes me wonder.
  - stavros 5y ago
    I use it for all of my projects, yes. I haven't really had many problems, just one or two that were easily resolved.
  - saila 5y ago
    I've been using Poetry exclusively for the past few years. I had a couple small problems with it early on, but lately I haven't run into any issues.
odiroot 5y ago
Just go for virtualenv and pip. Virtualenvwrapper to have handy shell tools.
- Nextgrid 5y ago
  Alternatively, pyenv and pyenv-virtualenv for shell integration and seamless virtualenv activation. To be fair, I'm not saying there's anything wrong with virtualenvwrapper, just that I've never used it and for my purposes the above solution works well.
  - timothylaurent 5y ago
    This is our setup - fits perfectly, don't even miss `workon`
- smitty1e 5y ago
  That's been my thing, but I do nothing of interest. Maybe 3.11 can make python packaging less of a beautiful disaster.
- tandav 5y ago
  Just go docker
- xchaotic 5y ago
  This doesn’t solve dependency management? All it does is it separates your env and you can install what you need there. But installing with pip is still subject to version incompatibility etc.
cwp 5y ago
I just started a Python project, using PDM. So far, I like it a lot. If I hit a show-stopper, no big deal, it's pretty easy to switch.
mindwok 5y ago
I tried them all a few months ago. Poetry was the best in my opinion.
gorgoiler 5y ago
Do you suffer from requiring different versions of the same library, or have some other kind of non trivial dependencies? This will mark me out as a Luddite but I am still quite happy with a 5 line setup.py and “pip install .” https://news.ycombinator.com/item?id=29446715 https://news.ycombinator.com/item?id=29446715 $ cat setup.py from setuptools import setup setup( install_requires=['arrow'], packages=['dogclock'], scripts=['scripts/dogclock'])
siquick 5y ago
I have never experienced any issues with a virtualenv and pip. python3 -m venv venv && source venv/bin/activate && pip install -r requirements.txt Python standard library is great and its a nice languate if you like the syntax but aside from a few constants like Django, Flask, and Pandas, the ecosystem feels like it is slowly turning into a fragmented mess.
- kaidon 5y ago
  > python3 -m venv venv && source venv/bin/activate && pip install -r requirements.txt ... bless my `zsh` shell history for these incantations. I don't think I have any hope of remembering it -- probably because of all the old virtualenv incantations! Kind of agree with pipenv though. It's painfully slow, but it abstracts away having to worry about various requirements files (eg: dev vs prod) and the .lock keeps things consistent.
  - nsotelo 5y ago
    I use this same setup but with a requirements folder. Inside I have a base.txt, dev.txt and tests.txt where dev.txt starts with '-r tests.txt', etc. It's very handy for having different environments but doesn't lend itself to a lockfile.
- gjvc 5y ago
  Same, I use two small bash scripts in my projects bin/ directory, "venv-create" for the creation of the .venv/ and "venv-python" for running the effective version of Python from the .venv/ -- this sets environment variables such as PYTHONPATH and PYTHONDONTWRITEBYTECODE and provides a single approach for running project files and packages. I get versioned requirements files for the project base requirements, and also for each (version, implementation) of python, in case they are changes, and this has proven to be reliable for me. It's all about finding the minimal-but-complete convenience / ergonomic solution over the, err, inconvenience of packaging. I also marvel at when I attempt to explain these things to experienced programmers, I only manage to convince them 50% of the time at most.
- globular-toast 5y ago
smohare 5y ago
Pipenv is garbage. Don’t use it. I’d just use pip and requirements files if you can. It’s doubtful that your requirements are sufficiently complex as to require a more complex resolver, although that depends on your ML needs.
frostming 5y ago
PDM author here, if anyone want to know the advantage of __pypackages__ over virtualenv-based tools(venv, virtualenv, poetry, pipenv), here it is: The fact that virtualenvs come with a cloned(or symlinked) interpreter makes it vulnerable when users want to upgrade the host interpreter in-place unless you keep all the old installations in your system, which is what pyenv is doing. You can imagine how many interpreters, including virtualenv-embedded ones are on your machine. You can regard __pypackages__ as a virtualenv WITHOUT the interpreter, it can easily work with any python interpreter you choose as long as it has the same major.minor version as the packages folder.
bb88 5y ago
Poetry is good. We've used it 3 years in production without issue of large amounts of time to solve the dependencies. $ poetry init # to init the pyproject.toml file. $ poetry add <packages> $ poetry shell # to activate inside the virutalenv. Lock in the poetry.lock file after any change to your project dependencies, and other people can duplicate the project using after doing a git pull. $ poetry install

by fb03 5y ago

stavros 5y ago

Use Poetry. It does everything you need and PDM is a bit too new still.

fb03 5y ago
Great. I have heard "anecdata evidence" that sometimes poetry fails to install a combination of packages or something along those lines, did you find any of those shenanigans in your own experience?
- akdor1154 5y ago
  (Not GP) Occasionally, but still use poetry. The issues happen because poetry is strict about version constraints. Pip used to be lax, so some packages could get away with poor/overly restrictive constraints.. Now pip is strict as well, and poetry is also getting a lot more common, so maintainers have had pressure to fix most of these issues.
- nbadg 5y ago
  I've run into this a couple of times. Typically it's when starting a new project and gradually adding dependencies. By default, when adding (poetry add <dep>), poetry adds the most recent version of the library, and in pyproject.toml that becomes the minimum possible version. If you then later try to add something that depends on that library, but isn't up-to-date on the most recent version, poetry will (correctly!) error out because it fails to find a compatible solution. For example, something like this: poetry add ingredients # most recent version: 1.0 poetry add cake # requires ingredients, but <1.0 So here, poetry would add ingredients to your pyproject.toml with version = "^1.0.0" in the first step. In the second step, when you tried to add cake as a new dependency, it would say it can't find a version of cake compatible with your pyproject.toml's requirement for ingredients. The solution is to change pyproject.toml to be more permissive about the version of ingredients, resulting in it downgrading to a compatible version. Or submit an issue and/or PR with the maintainers of cake to bump up the supported versions.
  - frostming 5y ago
    In PDM, add command supports a `--unconstrained` option to relax the version constraints to find a solution. You don't need to edit the file manually.
  - andreareina 5y ago
    Another way it happens: I was stuck on python 3.6 for a while and it meant that I couldn't just poetry add pandas because the latest version isn't compatible with 3.6 and it won't look for other versions.
- acidbaseextract 5y ago
  No shenanigans for any library-ish code with few dependencies that targets relatively modern Python. I've had one or two fundamental version conflicts with a 5+ year old application with 100+ dependencies and a decent amount of legacy stuff. They were a pain in the ass, and the sdispater's stance on not allowing overrides is a pain in the ass. We ended up forking the upstream libraries to resolve the version conflict. With all of that, poetry is amazing and a huge step forward. I'd advocate it wholeheartedly.
- gooseyard 5y ago
  I've spent the last year or so comparing these packages. Initially, I liked poetry, but I began to run into issues related to client cert auth with private repositories. I and several other users who encountered these problems submitted pull requests to address them, but found that it was nearly impossible to get any of them reviewed, much less merged. Out of frustration, I decided to try pdm instead, and although pdm doesn't handle pushing to pypi repositories, Twine works fine for it, and I have had a pain-free experience using pdm for dependency management. I like the idea of poetry, but it's unusable if I can't get bugs fixed, and I gave up after nearly a year of trying.
  - bb88 5y ago
    > but I began to run into issues related to client cert auth with private repositories. Yeah, I think that's the worst thing about private infrastructure in 2021 is getting those certs correctly installed and seen by the software and both sides. I'm thinking that tailscale or something like it may be a better alternative in the future where authorization is at the link layer before the tcp connect can happen.
- stavros 5y ago
  As others have said, this is related to how Poetry does dependency resolution. If you say your program uses Python "^3.8", you would expect any Python version in that range to work, but instead, ALL Python versions in that range must work. If one of your dependencies won't work with 3.9, the dependency graph cannot resolve. The author of Poetry said that this is because Poetry is meant for libraries, where this makes more sense.
  - amethyst 5y ago
    Except Poetry's recommendations/decisions around version capping doesn't make sense for libraries. https://iscinumpy.dev/post/bound-version-constraints/ https://iscinumpy.dev/post/bound-version-constraints/
jreese 5y ago
As a Python infra person, I would refine that to: If you're building an application, use Poetry. If you're building a library, use Flit. Use PEP621 metadata in pyproject.toml regardless. Poetry is much more focused on managing dependencies for applications than dealing with libraries that have to be used by other libraries or applications. See this deep discussion for some timely/relevant examples: https://iscinumpy.dev/post/bound-version-constraints/ https://iscinumpy.dev/post/bound-version-constraints/
- kbumsik 5y ago
  Can we mix Poetry and PEP621 though? I tried to use project.dependencies and project.optional-dependencies fields in pyproject.toml defined by PEP621. However Poetry seems to ignore it and it tries to stick with its own tools.poetry section.
  - jreese 5y ago
    Ah, you're right. I thought Poetry had adopted PEP 621 already, but it looks like there's still a debate about if/how they want to do that. A bit ironic considering the author of Poetry was listed as an author on PEP 621, and that was accepted just over a year ago.
    - kbumsik 5y ago
      It went final this March though. So Poetry might needs more time to think about the transitioning plan. Python packaging is again in a chaos since 2021 because of the rise of standardized PEPs.
      - spurgu 5y ago
        I know you didn't mean it that way, but "since 2021"..? :D I was working actively with Python (and loving it) between 2012-2015 (during which the 2 vs 3 transition/conflicts were raging) and the couple of times I've tried to get back into the groove since then (in side projects) I keep hitting walls and getting confused about how to even get the basic things running. I've heard good things about Poetry but never tried it. PDM being closer to npm/yarn sounds like an attractive option (since they're very familar), just to tinker around (not to build a complex application with specific requirements). Will give it a shot!
zmmmmm 5y ago
Sold everyone on using poetry a few months ago and now red faced as we have a litany of problems and time waste due to it. We are now sitting on some bleeding edge branch because specific dependencies cannot work at all without some new fangled feature and everyone wishes we were just using plain virtualenv as we had much less problems with that.
- smohare 5y ago
  In other words, you work with people who have no clue what’s going on.
  - spurgu 5y ago
    Thanks, that's good advice!
- zelphirkalt 5y ago
  You would also not have any real control over what gets installed with just venv. So everyone might think the grass was greener on the other side, but it might not actually be, if you value reproducible environments.
  - funstuff007 5y ago
    What you've never spent hours trying out n! orderings of requirements.txt until you get the results you want/can live with? You have not lived until you've done this. /s
    - bb88 5y ago
      Dep Hell is real.
burgerrito 5y ago
Is Poetry used a lot by Python devs? What package management a lot of Python devs use? I mostly code on JavaScript and (obviously) use NPM a lot, and it makes me wonder.
- stavros 5y ago
  I use it for all of my projects, yes. I haven't really had many problems, just one or two that were easily resolved.
- saila 5y ago
  I've been using Poetry exclusively for the past few years. I had a couple small problems with it early on, but lately I haven't run into any issues.

odiroot 5y ago

Just go for virtualenv and pip. Virtualenvwrapper to have handy shell tools.

Nextgrid 5y ago
Alternatively, pyenv and pyenv-virtualenv for shell integration and seamless virtualenv activation. To be fair, I'm not saying there's anything wrong with virtualenvwrapper, just that I've never used it and for my purposes the above solution works well.
- timothylaurent 5y ago
  This is our setup - fits perfectly, don't even miss `workon`
smitty1e 5y ago
That's been my thing, but I do nothing of interest. Maybe 3.11 can make python packaging less of a beautiful disaster.
tandav 5y ago
Just go docker
xchaotic 5y ago
This doesn’t solve dependency management? All it does is it separates your env and you can install what you need there. But installing with pip is still subject to version incompatibility etc.

cwp 5y ago

I just started a Python project, using PDM. So far, I like it a lot. If I hit a show-stopper, no big deal, it's pretty easy to switch.

mindwok 5y ago

I tried them all a few months ago. Poetry was the best in my opinion.

gorgoiler 5y ago

Do you suffer from requiring different versions of the same library, or have some other kind of non trivial dependencies? This will mark me out as a Luddite but I am still quite happy with a 5 line setup.py and “pip install .” https://news.ycombinator.com/item?id=29446715 https://news.ycombinator.com/item?id=29446715 $ cat setup.py from setuptools import setup setup( install_requires=['arrow'], packages=['dogclock'], scripts=['scripts/dogclock'])

siquick 5y ago

I have never experienced any issues with a virtualenv and pip. python3 -m venv venv && source venv/bin/activate && pip install -r requirements.txt Python standard library is great and its a nice languate if you like the syntax but aside from a few constants like Django, Flask, and Pandas, the ecosystem feels like it is slowly turning into a fragmented mess.

kaidon 5y ago
> python3 -m venv venv && source venv/bin/activate && pip install -r requirements.txt ... bless my `zsh` shell history for these incantations. I don't think I have any hope of remembering it -- probably because of all the old virtualenv incantations! Kind of agree with pipenv though. It's painfully slow, but it abstracts away having to worry about various requirements files (eg: dev vs prod) and the .lock keeps things consistent.
- nsotelo 5y ago
  I use this same setup but with a requirements folder. Inside I have a base.txt, dev.txt and tests.txt where dev.txt starts with '-r tests.txt', etc. It's very handy for having different environments but doesn't lend itself to a lockfile.
gjvc 5y ago
Same, I use two small bash scripts in my projects bin/ directory, "venv-create" for the creation of the .venv/ and "venv-python" for running the effective version of Python from the .venv/ -- this sets environment variables such as PYTHONPATH and PYTHONDONTWRITEBYTECODE and provides a single approach for running project files and packages. I get versioned requirements files for the project base requirements, and also for each (version, implementation) of python, in case they are changes, and this has proven to be reliable for me. It's all about finding the minimal-but-complete convenience / ergonomic solution over the, err, inconvenience of packaging. I also marvel at when I attempt to explain these things to experienced programmers, I only manage to convince them 50% of the time at most.
globular-toast 5y ago

smohare 5y ago

Pipenv is garbage. Don’t use it. I’d just use pip and requirements files if you can. It’s doubtful that your requirements are sufficiently complex as to require a more complex resolver, although that depends on your ML needs.

frostming 5y ago

PDM author here, if anyone want to know the advantage of __pypackages__ over virtualenv-based tools(venv, virtualenv, poetry, pipenv), here it is: The fact that virtualenvs come with a cloned(or symlinked) interpreter makes it vulnerable when users want to upgrade the host interpreter in-place unless you keep all the old installations in your system, which is what pyenv is doing. You can imagine how many interpreters, including virtualenv-embedded ones are on your machine. You can regard __pypackages__ as a virtualenv WITHOUT the interpreter, it can easily work with any python interpreter you choose as long as it has the same major.minor version as the packages folder.

bb88 5y ago

Poetry is good. We've used it 3 years in production without issue of large amounts of time to solve the dependencies. $ poetry init # to init the pyproject.toml file. $ poetry add <packages> $ poetry shell # to activate inside the virutalenv. Lock in the poetry.lock file after any change to your project dependencies, and other people can duplicate the project using after doing a git pull. $ poetry install