4 ms·

With the move to pyproject.toml in 2022[0], Poetry has become our goto method. With the lock file being default, we don't worry about different installs in dif

With the move to pyproject.toml in 2022[0], Poetry has become our goto method. With the lock file being default, we don't worry about different installs in different envs. Having come from the Rails world, Bundler system was solved for... a decade? So I was surprised it was such a mess in Python until so recently. At the core, the thing that makes Poetry and Bundler system so predictable is 1, lock file. 2, ability to install different versions in diff locations and referencing the version you need to load. Each alone isn't enough. npm had the same problem pip suffered from, you may have a version installed different what the req.txt, proj.js or even lockfile says but because it exists inside the location, it gets loaded. It wasn't until yarn2 did node_modules finally get moved out such that side-by-side versions wasn't awkward. [EDIT] If you're not using Poetry + Docker for deployment yet, I 100% recommend it as the "boring" method. RUN curl -sSL https://install.python-poetry.org | python - RUN /root/.local/bin/poetry config virtualenvs.create false COPY poetry.lock pyproject.toml ./ RUN --mount=type=secret,id=gh_priv_key,target=/root/.ssh/id_rsa \ /root/.local/bin/poetry install --without dev --no-root [0] https://packaging.python.org/en/latest/tutorials/packaging-projects/ https://packaging.python.org/en/latest/tutorials/packaging-p...

EdwardDiego 3y ago
Notably Poetry lockfiles are farrrrrr better cross-env then Pipenv's, which if you lock on an ARM Mac, will not install on a x64 Linux box. Poetry just works.
andenacitelli 3y ago
Hard agree. This article makes good points but I feel like Poetry just simplifies so much of it and makes it so much less manual. We’ve used it at $DAYJOB for months and it’s been awesome. In the same sense as the article — it’s been sufficiently boring to rarely be the source of problems, and when it is, it’s almost always our fault rather than a design flaw in Poetry.
doctorpangloss 3y ago
Despite these comments, Poetry is pretty limited. You cannot install PyTorch with it for example, and the developers do not care: https://github.com/python-poetry/poetry/issues/6409 https://github.com/python-poetry/poetry/issues/6409 So basically the single most exciting thing to ever happen to the ecosystem isn’t supported by Poetry.
- irjustin 3y ago
  Based on this comment 5 days ago[0], it's working? I'm not sure didn't dig in too far but based on that comment it seems fair to say that it's not fully Poetry's fault because torch removed hashes (which poetry needs to be effective) for a while only recently adding it back in. Not sure where I would stand if I fully investigated it tho. [0] https://github.com/python-poetry/poetry/issues/6409#issuecomment-1917702311 https://github.com/python-poetry/poetry/issues/6409#issuecom...
  - doctorpangloss 3y ago
    Every comment in that thread, including the comment you referenced, is wrong. Due to Poetry's architecture, it can't satisfy all three at the same time: - the platonic ideal of build isolation and lockfiles - installing the appropriate accelerator specific version of pytorch for the current platform non-interactively or one that the user selects - dependencies for pytorch that are transitively compatible with the way dependencies for pytorch are expressed elsewhere This is something you can achieve with setuptools and setup.py by forfeiting the platonic ideals of build isolation and lockfiles. Poetry, on the other hand, does not let you choose which lamb to sacrifice. Everyone in the thread, for the last two years, who has reported that they have had some success are misunderstanding the state of their install, and have interacted with flaws in all three situations I'm describing. They have something that will not correctly install anything that is dependent itself on PyTorch, which is useless, since everything in the PyTorch ecosystem is is dependent on it, and the main workaround the community uses - installing torch first, followed by installing dependencies from a requirements.txt, followed by copying a dump of scripts - is not compatible with poetry. Here's the facts of the matter: - 1.2M requirements.txts https://github.com/search?q=path%3A%2F%5Erequirements.txt%24%2F&type=code https://github.com/search?q=path%3A%2F%5Erequirements.txt%24... - 664k setup.pys https://github.com/search?q=path%3A%2F%5Esetup.py%24%2F&type=code https://github.com/search?q=path%3A%2F%5Esetup.py%24%2F&type... - setup.pys that reference requirements.txts https://github.com/search?q=path%3A%2F%5Esetup.py%24%2F+requirements.txt&type=code https://github.com/search?q=path%3A%2F%5Esetup.py%24%2F+requ... 67.1k - only 30k pyproject.toml specified with dependencies https://github.com/search?q=path%3A%2F%5Epyproject.toml%24%2F+"dependencies+%3D"&type=code https://github.com/search?q=path%3A%2F%5Epyproject.toml%24%2... 2/3rds of Python end users do not engage with packaging at all. pyproject.toml with dependencies is about 1.5% of the ecosystem. It provides only downsides compared to setup.py and pinning your package versions by commit in your setup.py dependencies aka doing what golang does, and this does not require any external tools in Python. In my opinion, the Poetry developers need to fix pytorch or they will not get adoption during Peak Python.
BiteCode_dev 3y ago
I'm going to sound like a zealot at this point but: '"Why not tell people to "simply" use pyenv, poetry or anaconda' https://www.bitecode.dev/p/why-not-tell-people-to-simply-use https://www.bitecode.dev/p/why-not-tell-people-to-simply-use
- tempaccount1234 3y ago
  Why not simply use pip? At least that works…
  - BiteCode_dev 3y ago
    Well, since I'm spamming, let's commit to it :) That's pretty much what "Relieving your Python packaging pain" advocates. https://www.bitecode.dev/p/relieving-your-python-packaging-pain https://www.bitecode.dev/p/relieving-your-python-packaging-p...
- aragilar 3y ago
  I agree with your posts(s), but conda is different to pyenv, poetry and similar tools (rye, hatch etc.) in that unlike the others which only manage Python, it (tries) to manage your whole environment. Can the others manage R packages (because your using r2py or reticulate)? What about installing pandoc? Or latex? I suspect we're going to continue in having churn in what people seem to be calling project/environment managers simply because there's a bunch of possible workflows (given the various possible deployment scenarios), and there's enough library support that starting your own is a weekend project (and then all the corner cases are discovered and you make a choice that works for some but not others and the cycle begins anew). conda seems to have dominated its niche (I haven't seen activepython nor enthought in years), but I suspect that part of the ecosystem is more naturally stable.
tlonny 3y ago
Very tangential, but wanted to share a small QoL trick that makes using poetry (and venv in general) much nicer... Instead of having to type "poetry shell" to activate the virtualenv, I use a tool called direnv that automatically modifies envars when you enter a directory - undoing the changes when you leave. In my direnv config (.envrc) for the directory I write: PATH_add .venv/bin To add all the virtualenv executables to PATH. Now I can just call "python/pip" directly and never have to worry if I'm inside my virtualenv or not. N.B. for this you need to update poetry config to ensure virtualenvs are placed in the project directory vs. the default centralised location
- PufPufPuf 3y ago
  It's better to activate the virtualenv properly: . .venv/bin/activate Setting up just the PATH may confuse some tools.
  - regularfry 3y ago
    I personally intensely dislike tools that change the environment based on the directory contents. I use an `exec` script in the root (or parent, sometimes) that just does this: ``` #!/bin/bash source .venv/bin/activate exec $SHELL $@ ``` That gives me a subshell with the virtual env active, and the path to the virtual env goes into $PS1 so I've got a very visible "you are working in this project" signifier. I don't know why subshells aren't more popular for this sort of thing, they remove a lot of ambiguity (and subsequent opportunities to screw things up) for me.
mrweasel 3y ago
Poetry solves a very real problem, but I kinda hate it. It's not a pleasant tool to use. Allow me to out-bore you, because there's way to much excitement in your solution. We just build Debian packages and deploy our applications that way. The Debian packaging infrastructure could be better, but mostly it just wraps around setuptools/pyproject. You need to rebuild your containers every time there's a security update to package, we just set apt to auto-update.
alfons_foobar 3y ago
We use poetry inside docker in a very similar way, but use a multistage docker file - the "builder" image has poetry installed and installs the application into a venv, we then copy that venv to the "prod" image and run the application there (so we do not have poetry or any other packages that are only required at build time in the final image). Any thoughts on that approach?
- linkdd 3y ago
  For Django projects, I made a gist as a template of what I use: https://gist.github.com/linkdd/4aac2c2efc4a51af6ca4b05f395de728 https://gist.github.com/linkdd/4aac2c2efc4a51af6ca4b05f395de... In there, you'll find the settings.py split by environment (like you would have in the Elixir world), and a multistage Docker image designed to be cache-friendly (for faster Docker builds). Some stages even run the test suite, so it could be compared to a CI pipeline. Not having build dependencies in the production image is definitely a good approach.
- irjustin 3y ago
  It's definitely fine with no problems. We used to do this too. But, it optimizes for something that only matters if you're in a space constrained setup. Turns out for "almost-all" web setups, a few extra hundreds of MBs don't matter even a few GB possibly. Even in AWS Lambda where cold-start is king, size doesn't matter[0] - I am the top answer. As long as you have fast starts and that's determined by code paths, libraries etc. So as long as your program readies itself quickly, having gcc and a few other std libs don't impact any key metric. [0] https://stackoverflow.com/questions/69512271/will-the-cold-starts-of-my-aws-lambda-function-take-longer-if-i-use-an-ecr-image/74383087#74383087 https://stackoverflow.com/questions/69512271/will-the-cold-s...
baggiponte 3y ago
Not a huge fan of poetry since it does not comply with standards (PEP621 PEP518...). But it has a great UX compared to alternatives. My go-to has been PDM: which is basically PEP-compliant poetry. But it is being maintained by one dev only and support for containers is not great...
awongh 3y ago
Also coming from Rails, besides the way pip, etc. works, I was also surprised at just how fragmented the landscape is. And with a lot of strong opinions on how that person’s viewpoint is correct. When I was figuring this stuff out it was very confusing because of course these strongly held opinions neglect to list the huge downsides. I think it’s an interesting note about the culture of Python.
- irjustin 3y ago
  YES!! 100%, I emphatically CANNOT stand how terrible model validations are in Django (obvious vs Rails) and they are a direct result of "we do it better". full_clean vs Form.full_clean vs DRF. It's just an absolute shit show. Even if it is better, it's just annoying and not worth the benefits. For us the overall benefits outweigh the drawbacks but I get angry every time when I need to do a model validation.
brnt 3y ago
What does Poetry provide that the 'official' PyPA tooling doesn't? I'm referring to the somewhat unfortunately named 'wheel' and 'build'? Also, `pyproject.toml` itself is an innovation from outside of Poetry afaik. And for what use-cases does Poetry not work, if any?
- irjustin 3y ago
  ... i mentioned it in my post. but, it is fair to say, if you've never experienced a versioning issue then you wouldn't find poetry amazing.
  - brnt 3y ago
    How is it different from pip freeze -l > requirements.txt?
rirze 3y ago
Poetry is a step up from plain pip but it's not complete either. I recommend switching to PDM. PDM follows more Python standards and is faster too.

EdwardDiego 3y ago

Notably Poetry lockfiles are farrrrrr better cross-env then Pipenv's, which if you lock on an ARM Mac, will not install on a x64 Linux box. Poetry just works.

andenacitelli 3y ago

Hard agree. This article makes good points but I feel like Poetry just simplifies so much of it and makes it so much less manual. We’ve used it at $DAYJOB for months and it’s been awesome. In the same sense as the article — it’s been sufficiently boring to rarely be the source of problems, and when it is, it’s almost always our fault rather than a design flaw in Poetry.

doctorpangloss 3y ago

Despite these comments, Poetry is pretty limited. You cannot install PyTorch with it for example, and the developers do not care: https://github.com/python-poetry/poetry/issues/6409 https://github.com/python-poetry/poetry/issues/6409 So basically the single most exciting thing to ever happen to the ecosystem isn’t supported by Poetry.

irjustin 3y ago
Based on this comment 5 days ago[0], it's working? I'm not sure didn't dig in too far but based on that comment it seems fair to say that it's not fully Poetry's fault because torch removed hashes (which poetry needs to be effective) for a while only recently adding it back in. Not sure where I would stand if I fully investigated it tho. [0] https://github.com/python-poetry/poetry/issues/6409#issuecomment-1917702311 https://github.com/python-poetry/poetry/issues/6409#issuecom...
- doctorpangloss 3y ago
  Every comment in that thread, including the comment you referenced, is wrong. Due to Poetry's architecture, it can't satisfy all three at the same time: - the platonic ideal of build isolation and lockfiles - installing the appropriate accelerator specific version of pytorch for the current platform non-interactively or one that the user selects - dependencies for pytorch that are transitively compatible with the way dependencies for pytorch are expressed elsewhere This is something you can achieve with setuptools and setup.py by forfeiting the platonic ideals of build isolation and lockfiles. Poetry, on the other hand, does not let you choose which lamb to sacrifice. Everyone in the thread, for the last two years, who has reported that they have had some success are misunderstanding the state of their install, and have interacted with flaws in all three situations I'm describing. They have something that will not correctly install anything that is dependent itself on PyTorch, which is useless, since everything in the PyTorch ecosystem is is dependent on it, and the main workaround the community uses - installing torch first, followed by installing dependencies from a requirements.txt, followed by copying a dump of scripts - is not compatible with poetry. Here's the facts of the matter: - 1.2M requirements.txts https://github.com/search?q=path%3A%2F%5Erequirements.txt%24%2F&type=code https://github.com/search?q=path%3A%2F%5Erequirements.txt%24... - 664k setup.pys https://github.com/search?q=path%3A%2F%5Esetup.py%24%2F&type=code https://github.com/search?q=path%3A%2F%5Esetup.py%24%2F&type... - setup.pys that reference requirements.txts https://github.com/search?q=path%3A%2F%5Esetup.py%24%2F+requirements.txt&type=code https://github.com/search?q=path%3A%2F%5Esetup.py%24%2F+requ... 67.1k - only 30k pyproject.toml specified with dependencies https://github.com/search?q=path%3A%2F%5Epyproject.toml%24%2F+"dependencies+%3D"&type=code https://github.com/search?q=path%3A%2F%5Epyproject.toml%24%2... 2/3rds of Python end users do not engage with packaging at all. pyproject.toml with dependencies is about 1.5% of the ecosystem. It provides only downsides compared to setup.py and pinning your package versions by commit in your setup.py dependencies aka doing what golang does, and this does not require any external tools in Python. In my opinion, the Poetry developers need to fix pytorch or they will not get adoption during Peak Python.

BiteCode_dev 3y ago

I'm going to sound like a zealot at this point but: '"Why not tell people to "simply" use pyenv, poetry or anaconda' https://www.bitecode.dev/p/why-not-tell-people-to-simply-use https://www.bitecode.dev/p/why-not-tell-people-to-simply-use

tempaccount1234 3y ago
Why not simply use pip? At least that works…
- BiteCode_dev 3y ago
  Well, since I'm spamming, let's commit to it :) That's pretty much what "Relieving your Python packaging pain" advocates. https://www.bitecode.dev/p/relieving-your-python-packaging-pain https://www.bitecode.dev/p/relieving-your-python-packaging-p...
aragilar 3y ago
I agree with your posts(s), but conda is different to pyenv, poetry and similar tools (rye, hatch etc.) in that unlike the others which only manage Python, it (tries) to manage your whole environment. Can the others manage R packages (because your using r2py or reticulate)? What about installing pandoc? Or latex? I suspect we're going to continue in having churn in what people seem to be calling project/environment managers simply because there's a bunch of possible workflows (given the various possible deployment scenarios), and there's enough library support that starting your own is a weekend project (and then all the corner cases are discovered and you make a choice that works for some but not others and the cycle begins anew). conda seems to have dominated its niche (I haven't seen activepython nor enthought in years), but I suspect that part of the ecosystem is more naturally stable.

tlonny 3y ago

Very tangential, but wanted to share a small QoL trick that makes using poetry (and venv in general) much nicer... Instead of having to type "poetry shell" to activate the virtualenv, I use a tool called direnv that automatically modifies envars when you enter a directory - undoing the changes when you leave. In my direnv config (.envrc) for the directory I write: PATH_add .venv/bin To add all the virtualenv executables to PATH. Now I can just call "python/pip" directly and never have to worry if I'm inside my virtualenv or not. N.B. for this you need to update poetry config to ensure virtualenvs are placed in the project directory vs. the default centralised location

PufPufPuf 3y ago
It's better to activate the virtualenv properly: . .venv/bin/activate Setting up just the PATH may confuse some tools.
- regularfry 3y ago
  I personally intensely dislike tools that change the environment based on the directory contents. I use an `exec` script in the root (or parent, sometimes) that just does this: ``` #!/bin/bash source .venv/bin/activate exec $SHELL $@ ``` That gives me a subshell with the virtual env active, and the path to the virtual env goes into $PS1 so I've got a very visible "you are working in this project" signifier. I don't know why subshells aren't more popular for this sort of thing, they remove a lot of ambiguity (and subsequent opportunities to screw things up) for me.

mrweasel 3y ago

Poetry solves a very real problem, but I kinda hate it. It's not a pleasant tool to use. Allow me to out-bore you, because there's way to much excitement in your solution. We just build Debian packages and deploy our applications that way. The Debian packaging infrastructure could be better, but mostly it just wraps around setuptools/pyproject. You need to rebuild your containers every time there's a security update to package, we just set apt to auto-update.

alfons_foobar 3y ago

We use poetry inside docker in a very similar way, but use a multistage docker file - the "builder" image has poetry installed and installs the application into a venv, we then copy that venv to the "prod" image and run the application there (so we do not have poetry or any other packages that are only required at build time in the final image). Any thoughts on that approach?

linkdd 3y ago
For Django projects, I made a gist as a template of what I use: https://gist.github.com/linkdd/4aac2c2efc4a51af6ca4b05f395de728 https://gist.github.com/linkdd/4aac2c2efc4a51af6ca4b05f395de... In there, you'll find the settings.py split by environment (like you would have in the Elixir world), and a multistage Docker image designed to be cache-friendly (for faster Docker builds). Some stages even run the test suite, so it could be compared to a CI pipeline. Not having build dependencies in the production image is definitely a good approach.
irjustin 3y ago
It's definitely fine with no problems. We used to do this too. But, it optimizes for something that only matters if you're in a space constrained setup. Turns out for "almost-all" web setups, a few extra hundreds of MBs don't matter even a few GB possibly. Even in AWS Lambda where cold-start is king, size doesn't matter[0] - I am the top answer. As long as you have fast starts and that's determined by code paths, libraries etc. So as long as your program readies itself quickly, having gcc and a few other std libs don't impact any key metric. [0] https://stackoverflow.com/questions/69512271/will-the-cold-starts-of-my-aws-lambda-function-take-longer-if-i-use-an-ecr-image/74383087#74383087 https://stackoverflow.com/questions/69512271/will-the-cold-s...

baggiponte 3y ago

Not a huge fan of poetry since it does not comply with standards (PEP621 PEP518...). But it has a great UX compared to alternatives. My go-to has been PDM: which is basically PEP-compliant poetry. But it is being maintained by one dev only and support for containers is not great...

awongh 3y ago

Also coming from Rails, besides the way pip, etc. works, I was also surprised at just how fragmented the landscape is. And with a lot of strong opinions on how that person’s viewpoint is correct. When I was figuring this stuff out it was very confusing because of course these strongly held opinions neglect to list the huge downsides. I think it’s an interesting note about the culture of Python.

irjustin 3y ago
YES!! 100%, I emphatically CANNOT stand how terrible model validations are in Django (obvious vs Rails) and they are a direct result of "we do it better". full_clean vs Form.full_clean vs DRF. It's just an absolute shit show. Even if it is better, it's just annoying and not worth the benefits. For us the overall benefits outweigh the drawbacks but I get angry every time when I need to do a model validation.

brnt 3y ago

What does Poetry provide that the 'official' PyPA tooling doesn't? I'm referring to the somewhat unfortunately named 'wheel' and 'build'? Also, `pyproject.toml` itself is an innovation from outside of Poetry afaik. And for what use-cases does Poetry not work, if any?

irjustin 3y ago
... i mentioned it in my post. but, it is fair to say, if you've never experienced a versioning issue then you wouldn't find poetry amazing.
- brnt 3y ago
  How is it different from pip freeze -l > requirements.txt?

rirze 3y ago

Poetry is a step up from plain pip but it's not complete either. I recommend switching to PDM. PDM follows more Python standards and is faster too.