9 ms·

I occasionally ask our principle / sr. Python engineers about this, and their response is always, "These things come and go, virtualenv/wrappers + pip + require

I occasionally ask our principle / sr. Python engineers about this, and their response is always, "These things come and go, virtualenv/wrappers + pip + requirements.txt works fine - no need to look at anything else." We've got about 15 repos, with the largest repo containing about 1575 files and 34MBytes of .py source, 14 current developers (with about 40 over the last 10 years) - and they really are quite proficient, but haven't demonstrated any interest at looking at anything outside pip/virtualenv. Is there a reason to look at poetry if you've got the pip/virtualenv combination working fine? People who use poetry seem to love it - so I'm interested in whether it provides any new abilities / flexibility that pip doesn't.

mplewis 6y ago
You're losing the ability to easily update using fuzzy specs and a lockfile if you're using pip vanilla without pip-tools or poetry.
- skrtskrt 6y ago
  To clarify further - lockfile == reproducible builds without having to pip freeze every single dependency in the tree. Fuzzy specs == effortless upgrades according to your risk tolerance for a given library (major version for boto3, minor version for pandas, something like that). Poetry gets you the combination of the two: Let your dep versions float, and easily revert back to a previous deterministic build using the version-controlled lockfile if something breaks.
  - xapata 6y ago
    setup.cfg vs requirements.txt offers this.
    - skrtskrt 6y ago
      Poetry is by no means the only option for this. Lots of people like pip-tools, it would feel a lot more lightweight and closer to pip than Poetry does. Pipenv exists but... steer clear for a multitude of reasons. Personally I like that Poetry centers itself around the pyproject.toml standard. I also think that its usability and the enthusiasm of both the maintainers of the users is going to really carry it more into the Python mainstream in the coming years.
      - deleted 6y ago
        [deleted]
      - crad 6y ago
        Personally I'm very disappointed that we keep inventing new standards, like pyproject.toml when other things have like setup.cfg, existed for an extended period of time, works well, and is supported for reading and writing by the stdlib.
        OJFord 6y ago
        I see pyproject.toml as more like 'tox.ini is nice, it's good that so many tools use it, but it's really nothing to do with tox', and bringing them (and hopefully those declining tox too) and setup.cfg into one.
  - ghshephard 6y ago
    I'm wondering why we wouldn't want to pip freeze every single dependency in the tree. I'm looking at our requirements.txt, and everything is of the form: vine==1.1.4 urllib3==1.25.10 wcwidth==0.1.7 I know that changing the versions of any of the underlying libraries is always a big conversation. (We're still locked in on pandas==0.23.4) So, if I understand correctly, with Poetry, we might be able to say, "Keep Pandas at 0.23.4 and sqlalchemy at 1.2.7 but figure out all the other dependencies for us and load them." Or, even better, "Keep Pandas at 0.23.x and sqlalchemy at 1.x.x but figure out all the other dependencies for us and load them." The advantage here is security patches in underlying libraries come for free, while we focus on porting code for the really high-level + important Libraries which aren't always backwards compatible (Pandas) Also - if we want to stick with specific versions, that's also possible with the lockfile - so every library will be exactly the same as the one in a build that worked. The thing I don't understand - is when I do: pip install pandas==0.23.4 It does load the dependencies. Indeed, if I create a requirements.txt that just has: pandas==1.0.3 pytz==2020.1 six==1.14.0 Then pip install -r requirements.txt goes and does: $ pip install -r requirements.txt Collecting pandas==1.0.3 Using cached pandas-1.0.3-cp38-cp38-manylinux1_x86_64.whl (10.0 MB) Collecting pytz==2020.1 Using cached pytz-2020.1-py2.py3-none-any.whl (510 kB) Collecting six==1.14.0 Using cached six-1.14.0-py2.py3-none-any.whl (10 kB) Collecting python-dateutil>=2.6.1 Using cached python_dateutil-2.8.1-py2.py3-none-any.whl (227 kB) Collecting numpy>=1.13.3 Using cached numpy-1.19.4-cp38-cp38-manylinux2010_x86_64.whl (14.5 MB) Installing collected packages: six, python-dateutil, pytz, numpy, pandas Successfully installed numpy-1.19.4 pandas-1.0.3 python- dateutil-2.8.1 pytz-2020.1 six-1.14.0 So - I'm still at a loss of the advantage of poetry vs pip install, given that pip loads dependencies as well - the advantage of "fuzzy specs" seems minimal given it's such a big deal to upgrade the big packages.
    - lmm 6y ago
      Nothing locks the version of numpy that you got there. If you run the same thing again in a few weeks you might get a completely different version, and have no way to revert to the version you had before.
- ptx 6y ago
  Pip supports this now. It calls them constraints files: https://pip.pypa.io/en/latest/user_guide/#constraints-files https://pip.pypa.io/en/latest/user_guide/#constraints-files
acdha 6y ago
If you have a working setup and especially a strategy for updating routinely (e.g. using pip-tools) and pinning dependencies with hashes, there’s less reason to change. The biggest reason I’d give is consistency and ease of adoption, which might be coming down to “do you spend much time on-boarding new developers?” or “are you supporting an open source community where this causes friction?”. If you aren’t spending much time on it, perhaps try it on new projects to see what people think in a low-friction situation - in my experience that’s basically been “I like spending less time on tool support”.
takeda 6y ago
I was one of those setup.py + requirements.txt (generated by pip-compile). Though, poetry is actually quite good. There are still some things that I wish it had, like plugin support (for example I really miss setuptools_scm) or being able to use it for C packages. But if your code is pure python it is great from my experience. The dependency resolver is especially good.
gjvc 6y ago
beware of zealot geeks bearing gifts. if your environment is currently working fine and you are only interested in running one version of python and perhaps experimenting with a later one then venv + pip is all you need, with some wrapper scripts as you say to make it ergonomic (to set the PYTHON* environment variables for your project, for example)
- dalemyers 6y ago
  The main reason I use Poetry is for its lockfile support. I've written about why lockfiles are good here: https://myers.io/2019/01/13/what-is-the-purpose-of-a-lock-file-for-package-managers/ https://myers.io/2019/01/13/what-is-the-purpose-of-a-lock-fi...
  - gjvc 6y ago
    What does this do/have over files generated by "pip freeze" ?
  - westurner 6y ago
    Pip supports constraints files. https://pip.pypa.io/en/stable/user_guide/#constraints-files https://pip.pypa.io/en/stable/user_guide/#constraints-files : > Constraints files are requirements files that only control which version of a requirement is installed, not whether it is installed or not. Their syntax and contents is nearly identical to Requirements Files. There is one key difference: Including a package in a constraints file does not trigger installation of the package. > Use a constraints file like so: python -m pip install -c constraints.txt
    - jshen 6y ago
      What does that command do? Does it install dependencies, or simply verify the versions of the installed dependencies?
      - ptx 6y ago
        It just gives you an error: ERROR: You must give at least one requirement to install (see "pip help install") So it seems like a strange choice of usage example. You have to provide both requirements and constraints for it to do anything useful (applying the version constraints to the requirements and their dependencies).
nickjj 6y ago
How do you deal with keeping your top level dependencies and exact versions of all of your dependencies of dependencies separate in a way that's sane and 100% reproduceable for a typical web app / repo that might not in itself be a Python package? I'm in the same boat as you in that I'd like to keep using pip but the lack of a lock file is very dangerous because it doesn't guarantee reproduceable builds (even if you use Docker). In Ruby, Elixir and Node the official package managers have the idea of a lock file. That is the only reason I ever look into maybe switching away from pip. Running a pip freeze to generate a requirements.txt file doesn't work nicely when you use a requirements.txt file to define your top level dependencies. I've been bitten by issues like this so many times in the past with Python where I forgot to define and pin some inner dependency of a tool. Like werkzeug when using Flask. Or a recent issue with Celery 4.3.0 where they forgot to version lock a dependency of their own and suddenly builds that worked one day started to break the next day. These sets of problems go away with a lock file.
- xapata 6y ago
  > Running a pip freeze to generate a requirements.txt file doesn't work nicely when you use a requirements.txt file to define your top level dependencies. Use setup.cfg to define your top level dependencies. Use requirements.txt as your "lock" file. But even then you won't get reproducible builds across different OSes, or with different non-Python things installed on your machines. Use Docker images to guarantee staging and production will be identical.
- ptx 6y ago
  There is one! Pip supports, since a few years back, a constraints file (i.e. a lock file) that does just this. This is a nice guide on how to use it: https://www.lutro.me/posts/better-pip-dependency-management https://www.lutro.me/posts/better-pip-dependency-management
  - dalemyers 6y ago
    This isn't quite the same. Yes, I can update my root dependencies in the `requirements.txt` file, run `pip install -r requirements.txt` and the run `pip freeze > requirements.txt`, but that's convoluted and requires me to know exactly what my root dependencies are. Is `astroid` something our tools use directly, or is it just a dependency of `pylint`? It's not clear. A lockfile clears this up.
nitely 6y ago
They either haven't experience the pain or are oblivious to it. Pip old resolver is borked[0]: > [The new resolver] will reduce inconsistency: it will no longer install a combination of packages that is mutually inconsistent. At the moment, it is possible for pip to install a package which does not satisfy the declared requirements of another installed package. For example, right now, pip install "six<1.12" "virtualenv==20.0.2" does the wrong thing, “successfully” installing six==1.11, even though virtualenv==20.0.2 requires six>=1.12.0,<2 (defined here). The new resolver would, instead, outright reject installing anything if it got that input. [0]: https://pyfound.blogspot.com/2020/03/new-pip-resolver-to-roll-out-this-year.html https://pyfound.blogspot.com/2020/03/new-pip-resolver-to-rol...
- ptx 6y ago
  That would probably be caught by the CI, though, wouldn't it? One of the packages would fail to install in a clean venv and the tests would fail.
  - ArchOversight 6y ago
    Not if you used the old version of pip. That is hopefully fixed now with the new resolver.
    - ptx 6y ago
      Hm, you're right. I misread it as saying that it would 1) install six, 2) fail to install virtualenv and 3) fail to report an error. But the old resolver actually installs both packages, even though it should just abort.
  - kstrauser 6y ago
    No, it won't be caught in CI. The old version of pip didn't recognize that as an error condition.
- divbzero 6y ago
  I haven’t found a compelling need to switch to Poetry, but independent experimentation and competition can be good thing. I wouldn’t be surprised if the new pip resolver was partly inspired by Poetry, similar to how some NPM improvements were motivated by Yarn. [1] [1]: https://javascript.christmas/2019/10 https://javascript.christmas/2019/10
cik 6y ago
poetry has several advantages which I can no longer live without. 1. Packages are downloaded in parallel. This means dramatically quicker dependency resolution and download times. 2. Packages can be separated for development versions production environments. 3. Poetry only pins the packages you actually care about, unlike a pip freeze. One application I work on has 15 dependencies, which yields a little over 115 packages download. pip freeze makes it impossible to track actual dependencies, whereas poetry tracks my dependencies - and the non-pinned packages are in the poetry.lock file. The rest is nice, but the above is essential.
- jsmeaton 6y ago
  Hang about. Poetry is SLOW because it does not operate in parallel. It’s the one thing holding me back from a large project. Unless this has recently changed?
  - dalemyers 6y ago
    I think this changed a month or two ago. It installs each "level" of the dependency tree in parallel.
  - aden1ne 6y ago
    Poetry 1.1 introduced parallel downloads. Even before that, it was an order of magnitude faster than its competitor Pipenv.
deleted 6y ago
[deleted]
user5994461 6y ago
Your senior engineers are right, there is absolutely no reason to look beyond pip + venv because they work just fine. The commenter pushing for poetry must have a stake in it (one of the developers?) because he's always pushing for it in HN discussions.
- dalemyers 6y ago
  I assume you are referring to me? I can assure you, I have absolutely zero stake in Poetry. I'm not one of the developers, I'm not even on of the contributors (excluding opening a bug report or two).
- maximilianroos 6y ago
  > The commenter pushing for poetry must have a stake in it One benefit of open source projects is how easy is it to discount your comment
  - SQueeeeeL 6y ago
    Notoriety is still a stake, people routinely get job offers for being big contributors to PyTorch
- dang 6y ago
  This comment breaks the site guidelines—would you mind reviewing them and sticking to the rules when posting here? Note this one: "Please don't post insinuations about astroturfing, shilling, brigading, foreign agents and the like. It degrades discussion and is usually mistaken. If you're worried about abuse, email hn@ycombinator.com and we'll look at the data." https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html Here's plenty of past explanation of why we have this rule: https://hn.algolia.com/?sort=byDate&dateRange=all&type=comment&storyText=false&prefix=true&page=0&query=by:dang%20astroturf https://hn.algolia.com/?sort=byDate&dateRange=all&type=comme.... The short version is that internet users are too quick to reach for "astroturfing" etc. as an explanation for what they don't like, and it poisons the ecosystem when people post these accusations without evidence. Someone else having different views—different tastes in package managers, for example—does not clear the bar for evidence.
Humphrey 6y ago
I was happy with Pip until I spent time in the NPM/Yarn world. Frustrated with Pip I switched half of our projects to Pipenv. However, I found that it struggled to resolve dependencies. Poetry works like a dream now, and life is so much easy now that we have switched all our projects to it. The methodology of specifying your core dependencies, but also having locked version of your dependency's dependencies works really well. AND you can easily export to requirements.txt if you prefer to use that in production.
- jtdev 6y ago
  What does NPM/yarn do better than virtualenv/pip? “Added 17000332 packages (including is-even) 875 vulnerabilities found, have fun with that info. Yours truly, NPM”
  - linkdd 6y ago
    Ẁhile I agree the situation is ridiculous, what prevents anyone to do the same in Python? I can publish is-even on PyPI if I want, is that Pip's fault?
    - cookiecaper 6y ago
      It's a difference in the community's engineering values. JavaScript devs pull in a dependency for virtually everything, whereas Python distributes an extensive standard library with the language. It's less important that the same thing is hypothetically possible in both communities and more important that specific communities have chosen to use similar toolkits differently.
jshen 6y ago
I’m curious how they ensure every developer is using the same versions of things, as well as how they manage dev dependencies and transitive dependencies. pip + venv + requirements.txt doesn’t solve this out of the box while most languages have common tools that do. Either they’ve rolled their own way to manage these things, or they’re rolling the dice every time they deploy.
- Denvercoder9 6y ago
  You can pin versions in requirements.txt, and have a separate requirements-dev.txt.
- ghshephard 6y ago
  We have the following: $ wc -l requirements.txt 118 requirements.txt And every module in it is locked to a particular version: alembic==0.8.8 amqp==2.2.2 anyjson==0.3.3 azure-storage==0.36.0 backports.shutil-get-terminal-size==1.0.0 billiard==3.5.0.2 etc... I don't really understand the "Dependencies" thing (Or the difference between dev dependencies/transitive dependencies)- we literally list every single module in our environment, and its version - It's not clear to me what other dependencies there could be in a python development environment. I do note we have three requirements.txt files, a requirements.txt, requirements-test.txt, and a requirements-dev.txt. So, presumably there is a need for different requirements that you've identified that I don't understand. So there's that.
  - jshen 6y ago
    Dev dependencies: a library you need during development, but that isn’t needed in production. I think your -test and -dev are this, but it’s not clear how you are maintaining all of these, and building for prod. This is the main complaint, most modern languages have a standard set of tools and flows for achieving this. Python doesn’t, and everyone does it a bit differently, and when starting a new project, you have to hand roll your own flow. Or, use something like poetry but the python community as a whole doesn’t have a commonly used solution.
WhyNotHugo 6y ago
> Is there a reason to look at poetry if you've got the pip/virtualenv combination working fine? You probably have a bunch of scripts that do what poetry does (either that, or you repeat the same commands over and over A LOT). Switching to poetry might have some initial overhead, but a big upside is that you stop using custom, internal tooling, and use something industry-standard. Importantly, it makes it easier for you to understand external projects (since you're familiar with the standard tooling), and faster to onboard newcomers.
throwaway5752 6y ago
I will give it 5-10 years. If they are serious about it, it will last that long. They're about halfway there. Package management is terrible work. Nobody appreciates it. It's extremely complex, it has to work in enormous numbers of configurations, and very minor errors can have catastrophic impact to security or availability of systems.
saghm 6y ago
> I occasionally ask our principle / sr. Python engineers about this, and their response is always, "These things come and go, virtualenv/wrappers + pip + requirements.txt works fine - no need to look at anything else." At the macro level, this seems like a bit of a self-fulfilling prophecy: if all the senior and principal engineers using Python don't care to take a look at something else, then it's not too surprising that new solutions don't end up sticking around. That isn't too say that there does necessarily need to be a change in the Python community's choice of package manager, but the rationale for not even considering looking at other options doesn't seem super compelling.
- edoceo 6y ago
  Or the new thing needs a really compelling case
m4rtink 6y ago
I personally rather avoid any per language package managers and opt for distro level package management (rpm/deb). Thats something that has bern there pretty much for ever & has all the hard & dirty depsolving issues long solved. Also I can use it for all software regardless of what laguage it's written in, not to mention having to learn multiple half-baked package managers per laguage! And lastly any non trivial software project will need dependencies outside of the world of its own package manager anyway, so why not go all in properly with rmp/deb & make everything easier for your users & your future self.
- cozzyd 6y ago
  python setup.py bdist_rpm is also useful (although it's quite naive, doesn't always do the right thing and you'd usually want to add dependencies).

mplewis 6y ago

You're losing the ability to easily update using fuzzy specs and a lockfile if you're using pip vanilla without pip-tools or poetry.

skrtskrt 6y ago
To clarify further - lockfile == reproducible builds without having to pip freeze every single dependency in the tree. Fuzzy specs == effortless upgrades according to your risk tolerance for a given library (major version for boto3, minor version for pandas, something like that). Poetry gets you the combination of the two: Let your dep versions float, and easily revert back to a previous deterministic build using the version-controlled lockfile if something breaks.
- xapata 6y ago
  setup.cfg vs requirements.txt offers this.
  - skrtskrt 6y ago
    Poetry is by no means the only option for this. Lots of people like pip-tools, it would feel a lot more lightweight and closer to pip than Poetry does. Pipenv exists but... steer clear for a multitude of reasons. Personally I like that Poetry centers itself around the pyproject.toml standard. I also think that its usability and the enthusiasm of both the maintainers of the users is going to really carry it more into the Python mainstream in the coming years.
    - deleted 6y ago
      [deleted]
    - crad 6y ago
      Personally I'm very disappointed that we keep inventing new standards, like pyproject.toml when other things have like setup.cfg, existed for an extended period of time, works well, and is supported for reading and writing by the stdlib.
      - OJFord 6y ago
        I see pyproject.toml as more like 'tox.ini is nice, it's good that so many tools use it, but it's really nothing to do with tox', and bringing them (and hopefully those declining tox too) and setup.cfg into one.
- ghshephard 6y ago
  I'm wondering why we wouldn't want to pip freeze every single dependency in the tree. I'm looking at our requirements.txt, and everything is of the form: vine==1.1.4 urllib3==1.25.10 wcwidth==0.1.7 I know that changing the versions of any of the underlying libraries is always a big conversation. (We're still locked in on pandas==0.23.4) So, if I understand correctly, with Poetry, we might be able to say, "Keep Pandas at 0.23.4 and sqlalchemy at 1.2.7 but figure out all the other dependencies for us and load them." Or, even better, "Keep Pandas at 0.23.x and sqlalchemy at 1.x.x but figure out all the other dependencies for us and load them." The advantage here is security patches in underlying libraries come for free, while we focus on porting code for the really high-level + important Libraries which aren't always backwards compatible (Pandas) Also - if we want to stick with specific versions, that's also possible with the lockfile - so every library will be exactly the same as the one in a build that worked. The thing I don't understand - is when I do: pip install pandas==0.23.4 It does load the dependencies. Indeed, if I create a requirements.txt that just has: pandas==1.0.3 pytz==2020.1 six==1.14.0 Then pip install -r requirements.txt goes and does: $ pip install -r requirements.txt Collecting pandas==1.0.3 Using cached pandas-1.0.3-cp38-cp38-manylinux1_x86_64.whl (10.0 MB) Collecting pytz==2020.1 Using cached pytz-2020.1-py2.py3-none-any.whl (510 kB) Collecting six==1.14.0 Using cached six-1.14.0-py2.py3-none-any.whl (10 kB) Collecting python-dateutil>=2.6.1 Using cached python_dateutil-2.8.1-py2.py3-none-any.whl (227 kB) Collecting numpy>=1.13.3 Using cached numpy-1.19.4-cp38-cp38-manylinux2010_x86_64.whl (14.5 MB) Installing collected packages: six, python-dateutil, pytz, numpy, pandas Successfully installed numpy-1.19.4 pandas-1.0.3 python- dateutil-2.8.1 pytz-2020.1 six-1.14.0 So - I'm still at a loss of the advantage of poetry vs pip install, given that pip loads dependencies as well - the advantage of "fuzzy specs" seems minimal given it's such a big deal to upgrade the big packages.
  - lmm 6y ago
    Nothing locks the version of numpy that you got there. If you run the same thing again in a few weeks you might get a completely different version, and have no way to revert to the version you had before.
ptx 6y ago
Pip supports this now. It calls them constraints files: https://pip.pypa.io/en/latest/user_guide/#constraints-files https://pip.pypa.io/en/latest/user_guide/#constraints-files

acdha 6y ago

If you have a working setup and especially a strategy for updating routinely (e.g. using pip-tools) and pinning dependencies with hashes, there’s less reason to change. The biggest reason I’d give is consistency and ease of adoption, which might be coming down to “do you spend much time on-boarding new developers?” or “are you supporting an open source community where this causes friction?”. If you aren’t spending much time on it, perhaps try it on new projects to see what people think in a low-friction situation - in my experience that’s basically been “I like spending less time on tool support”.

takeda 6y ago

I was one of those setup.py + requirements.txt (generated by pip-compile). Though, poetry is actually quite good. There are still some things that I wish it had, like plugin support (for example I really miss setuptools_scm) or being able to use it for C packages. But if your code is pure python it is great from my experience. The dependency resolver is especially good.

gjvc 6y ago

beware of zealot geeks bearing gifts. if your environment is currently working fine and you are only interested in running one version of python and perhaps experimenting with a later one then venv + pip is all you need, with some wrapper scripts as you say to make it ergonomic (to set the PYTHON* environment variables for your project, for example)

dalemyers 6y ago
The main reason I use Poetry is for its lockfile support. I've written about why lockfiles are good here: https://myers.io/2019/01/13/what-is-the-purpose-of-a-lock-file-for-package-managers/ https://myers.io/2019/01/13/what-is-the-purpose-of-a-lock-fi...
- gjvc 6y ago
  What does this do/have over files generated by "pip freeze" ?
- westurner 6y ago
  Pip supports constraints files. https://pip.pypa.io/en/stable/user_guide/#constraints-files https://pip.pypa.io/en/stable/user_guide/#constraints-files : > Constraints files are requirements files that only control which version of a requirement is installed, not whether it is installed or not. Their syntax and contents is nearly identical to Requirements Files. There is one key difference: Including a package in a constraints file does not trigger installation of the package. > Use a constraints file like so: python -m pip install -c constraints.txt
  - jshen 6y ago
    What does that command do? Does it install dependencies, or simply verify the versions of the installed dependencies?
    - ptx 6y ago
      It just gives you an error: ERROR: You must give at least one requirement to install (see "pip help install") So it seems like a strange choice of usage example. You have to provide both requirements and constraints for it to do anything useful (applying the version constraints to the requirements and their dependencies).

nickjj 6y ago

How do you deal with keeping your top level dependencies and exact versions of all of your dependencies of dependencies separate in a way that's sane and 100% reproduceable for a typical web app / repo that might not in itself be a Python package? I'm in the same boat as you in that I'd like to keep using pip but the lack of a lock file is very dangerous because it doesn't guarantee reproduceable builds (even if you use Docker). In Ruby, Elixir and Node the official package managers have the idea of a lock file. That is the only reason I ever look into maybe switching away from pip. Running a pip freeze to generate a requirements.txt file doesn't work nicely when you use a requirements.txt file to define your top level dependencies. I've been bitten by issues like this so many times in the past with Python where I forgot to define and pin some inner dependency of a tool. Like werkzeug when using Flask. Or a recent issue with Celery 4.3.0 where they forgot to version lock a dependency of their own and suddenly builds that worked one day started to break the next day. These sets of problems go away with a lock file.

xapata 6y ago
> Running a pip freeze to generate a requirements.txt file doesn't work nicely when you use a requirements.txt file to define your top level dependencies. Use setup.cfg to define your top level dependencies. Use requirements.txt as your "lock" file. But even then you won't get reproducible builds across different OSes, or with different non-Python things installed on your machines. Use Docker images to guarantee staging and production will be identical.
ptx 6y ago
There is one! Pip supports, since a few years back, a constraints file (i.e. a lock file) that does just this. This is a nice guide on how to use it: https://www.lutro.me/posts/better-pip-dependency-management https://www.lutro.me/posts/better-pip-dependency-management
- dalemyers 6y ago
  This isn't quite the same. Yes, I can update my root dependencies in the `requirements.txt` file, run `pip install -r requirements.txt` and the run `pip freeze > requirements.txt`, but that's convoluted and requires me to know exactly what my root dependencies are. Is `astroid` something our tools use directly, or is it just a dependency of `pylint`? It's not clear. A lockfile clears this up.

nitely 6y ago

They either haven't experience the pain or are oblivious to it. Pip old resolver is borked[0]: > [The new resolver] will reduce inconsistency: it will no longer install a combination of packages that is mutually inconsistent. At the moment, it is possible for pip to install a package which does not satisfy the declared requirements of another installed package. For example, right now, pip install "six<1.12" "virtualenv==20.0.2" does the wrong thing, “successfully” installing six==1.11, even though virtualenv==20.0.2 requires six>=1.12.0,<2 (defined here). The new resolver would, instead, outright reject installing anything if it got that input. [0]: https://pyfound.blogspot.com/2020/03/new-pip-resolver-to-roll-out-this-year.html https://pyfound.blogspot.com/2020/03/new-pip-resolver-to-rol...

ptx 6y ago
That would probably be caught by the CI, though, wouldn't it? One of the packages would fail to install in a clean venv and the tests would fail.
- ArchOversight 6y ago
  Not if you used the old version of pip. That is hopefully fixed now with the new resolver.
  - ptx 6y ago
    Hm, you're right. I misread it as saying that it would 1) install six, 2) fail to install virtualenv and 3) fail to report an error. But the old resolver actually installs both packages, even though it should just abort.
- kstrauser 6y ago
  No, it won't be caught in CI. The old version of pip didn't recognize that as an error condition.
divbzero 6y ago
I haven’t found a compelling need to switch to Poetry, but independent experimentation and competition can be good thing. I wouldn’t be surprised if the new pip resolver was partly inspired by Poetry, similar to how some NPM improvements were motivated by Yarn. [1] [1]: https://javascript.christmas/2019/10 https://javascript.christmas/2019/10

cik 6y ago

poetry has several advantages which I can no longer live without. 1. Packages are downloaded in parallel. This means dramatically quicker dependency resolution and download times. 2. Packages can be separated for development versions production environments. 3. Poetry only pins the packages you actually care about, unlike a pip freeze. One application I work on has 15 dependencies, which yields a little over 115 packages download. pip freeze makes it impossible to track actual dependencies, whereas poetry tracks my dependencies - and the non-pinned packages are in the poetry.lock file. The rest is nice, but the above is essential.

jsmeaton 6y ago
Hang about. Poetry is SLOW because it does not operate in parallel. It’s the one thing holding me back from a large project. Unless this has recently changed?
- dalemyers 6y ago
  I think this changed a month or two ago. It installs each "level" of the dependency tree in parallel.
- aden1ne 6y ago
  Poetry 1.1 introduced parallel downloads. Even before that, it was an order of magnitude faster than its competitor Pipenv.

deleted 6y ago

[deleted]

user5994461 6y ago

Your senior engineers are right, there is absolutely no reason to look beyond pip + venv because they work just fine. The commenter pushing for poetry must have a stake in it (one of the developers?) because he's always pushing for it in HN discussions.

dalemyers 6y ago
I assume you are referring to me? I can assure you, I have absolutely zero stake in Poetry. I'm not one of the developers, I'm not even on of the contributors (excluding opening a bug report or two).
maximilianroos 6y ago
> The commenter pushing for poetry must have a stake in it One benefit of open source projects is how easy is it to discount your comment
- SQueeeeeL 6y ago
  Notoriety is still a stake, people routinely get job offers for being big contributors to PyTorch
dang 6y ago
This comment breaks the site guidelines—would you mind reviewing them and sticking to the rules when posting here? Note this one: "Please don't post insinuations about astroturfing, shilling, brigading, foreign agents and the like. It degrades discussion and is usually mistaken. If you're worried about abuse, email hn@ycombinator.com and we'll look at the data." https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html Here's plenty of past explanation of why we have this rule: https://hn.algolia.com/?sort=byDate&dateRange=all&type=comment&storyText=false&prefix=true&page=0&query=by:dang%20astroturf https://hn.algolia.com/?sort=byDate&dateRange=all&type=comme.... The short version is that internet users are too quick to reach for "astroturfing" etc. as an explanation for what they don't like, and it poisons the ecosystem when people post these accusations without evidence. Someone else having different views—different tastes in package managers, for example—does not clear the bar for evidence.

Humphrey 6y ago

I was happy with Pip until I spent time in the NPM/Yarn world. Frustrated with Pip I switched half of our projects to Pipenv. However, I found that it struggled to resolve dependencies. Poetry works like a dream now, and life is so much easy now that we have switched all our projects to it. The methodology of specifying your core dependencies, but also having locked version of your dependency's dependencies works really well. AND you can easily export to requirements.txt if you prefer to use that in production.

jtdev 6y ago
What does NPM/yarn do better than virtualenv/pip? “Added 17000332 packages (including is-even) 875 vulnerabilities found, have fun with that info. Yours truly, NPM”
- linkdd 6y ago
  Ẁhile I agree the situation is ridiculous, what prevents anyone to do the same in Python? I can publish is-even on PyPI if I want, is that Pip's fault?
  - cookiecaper 6y ago
    It's a difference in the community's engineering values. JavaScript devs pull in a dependency for virtually everything, whereas Python distributes an extensive standard library with the language. It's less important that the same thing is hypothetically possible in both communities and more important that specific communities have chosen to use similar toolkits differently.

jshen 6y ago

I’m curious how they ensure every developer is using the same versions of things, as well as how they manage dev dependencies and transitive dependencies. pip + venv + requirements.txt doesn’t solve this out of the box while most languages have common tools that do. Either they’ve rolled their own way to manage these things, or they’re rolling the dice every time they deploy.

Denvercoder9 6y ago
You can pin versions in requirements.txt, and have a separate requirements-dev.txt.
ghshephard 6y ago
We have the following: $ wc -l requirements.txt 118 requirements.txt And every module in it is locked to a particular version: alembic==0.8.8 amqp==2.2.2 anyjson==0.3.3 azure-storage==0.36.0 backports.shutil-get-terminal-size==1.0.0 billiard==3.5.0.2 etc... I don't really understand the "Dependencies" thing (Or the difference between dev dependencies/transitive dependencies)- we literally list every single module in our environment, and its version - It's not clear to me what other dependencies there could be in a python development environment. I do note we have three requirements.txt files, a requirements.txt, requirements-test.txt, and a requirements-dev.txt. So, presumably there is a need for different requirements that you've identified that I don't understand. So there's that.
- jshen 6y ago
  Dev dependencies: a library you need during development, but that isn’t needed in production. I think your -test and -dev are this, but it’s not clear how you are maintaining all of these, and building for prod. This is the main complaint, most modern languages have a standard set of tools and flows for achieving this. Python doesn’t, and everyone does it a bit differently, and when starting a new project, you have to hand roll your own flow. Or, use something like poetry but the python community as a whole doesn’t have a commonly used solution.

WhyNotHugo 6y ago

> Is there a reason to look at poetry if you've got the pip/virtualenv combination working fine? You probably have a bunch of scripts that do what poetry does (either that, or you repeat the same commands over and over A LOT). Switching to poetry might have some initial overhead, but a big upside is that you stop using custom, internal tooling, and use something industry-standard. Importantly, it makes it easier for you to understand external projects (since you're familiar with the standard tooling), and faster to onboard newcomers.

throwaway5752 6y ago

I will give it 5-10 years. If they are serious about it, it will last that long. They're about halfway there. Package management is terrible work. Nobody appreciates it. It's extremely complex, it has to work in enormous numbers of configurations, and very minor errors can have catastrophic impact to security or availability of systems.

saghm 6y ago

> I occasionally ask our principle / sr. Python engineers about this, and their response is always, "These things come and go, virtualenv/wrappers + pip + requirements.txt works fine - no need to look at anything else." At the macro level, this seems like a bit of a self-fulfilling prophecy: if all the senior and principal engineers using Python don't care to take a look at something else, then it's not too surprising that new solutions don't end up sticking around. That isn't too say that there does necessarily need to be a change in the Python community's choice of package manager, but the rationale for not even considering looking at other options doesn't seem super compelling.

edoceo 6y ago
Or the new thing needs a really compelling case

m4rtink 6y ago

I personally rather avoid any per language package managers and opt for distro level package management (rpm/deb). Thats something that has bern there pretty much for ever & has all the hard & dirty depsolving issues long solved. Also I can use it for all software regardless of what laguage it's written in, not to mention having to learn multiple half-baked package managers per laguage! And lastly any non trivial software project will need dependencies outside of the world of its own package manager anyway, so why not go all in properly with rmp/deb & make everything easier for your users & your future self.

cozzyd 6y ago
python setup.py bdist_rpm is also useful (although it's quite naive, doesn't always do the right thing and you'd usually want to add dependencies).