9 ms·
Poetry: Dependency Management for Python
- andyljones 7y agoI can't find a good comparison to Conda - is the main distinction simply that Poetry uses the official repos? For people working outside of scientific Python: conda is a package and env manager maintained by a private company that's become the go-to because it's really good at handling binary dependencies.
- tduberne 7y agoCan you really build packages using solely Conda? As far as my understanding goes, conda focuses on providing virtual environments for interactive use, but to build and distribute a package on conda forge, you still need to rely on setuptools/distutils. My impression here is that Poetry aims more at _replacing_ the pip + setuptools toolchain. Users of your package could still install it using conda, if relevant. It seems a bit limited in what it can do at the build step, unfortunately, so it is not a replacement yet. Coming from the Java world, I pass my days dreaming of a "maven for python", and this project definitely goes into that direction. I will definitely keep it on my radar.
- flyingdeertown 7y agoWell conda packages don't need to be python based, so technically yes you can build and package up whatever your heart desires. conda build recipes are essentially just shell scripts that conda runs in a sandbox, taking care of library paths and so on. You could, if you really wanted to, develop a poetry based Python package and then have your conda build script use/run poetry to build and package the python package, and then add whatever lines are necessary to the build recipe to make that the package that gets installed by conda.
- tduberne 7y agoThat basically confirms the feeling I was trying to express, in that conda and poetry seem to solve different problems: conda focuses on easing the distribution to the end user, whereas poetry tries to simplify managing the build on the developer side.
- flyingdeertown 7y agoI use conda as a python manager and poetry as an environment manager. For every project I just create a new conda environment with the bare minimum python 3.x version I need. Then I use poetry to handle the dependencies and manage the environment for me. For stuff that are strictly scientific or numerical in nature and I need things from conda (ie MKL accelerated numpy) I skip the part with poetry and just use conda to install my packages to avoid any potential issues.
- Areading314 7y agoThis looks awesome, well done! Is there a TLDR about how it improves on pip or other python dependency management?
- LaundroMat 7y agoThe GitHub readme explains how it improves on Pipenv. I like Poetry a lot; I hope platforms like Heroku will start supporting it soon.
- jsmeaton 7y agoI raised an official support request with Heroku for this. “It’s coming” along with other improvements to python build packs.
- tduberne 7y ago> The GitHub readme explains how it improves on Pipenv. That's a shame it is not in the project webpage, though. This is the first information I searched, without success.
- nickserv 7y agoHow is the execution time when resolving a non-trivial project? Like your typical Django app, for example.
- LaundroMat 7y agoAdding dependencies takes a (little) bit of time, because of the lockfile. But deployment (ie installing dependencies) is as fast as any dependency manager (pip, pipenv, ...).
- heyoni 7y agoCongrats on hitting version 1.0! I’ve been putting off completely switching off of pipenv until this milestone.
- LaundroMat 7y agoI'm not being snarky, it's probably just too early in the morning, but where did you see the project hit v1.0?
- j88439h84 7y agoPoetry is very good. I think projects should use it. I hope the rest of the ecosystem can catch up quickly. Tox and pip and and pex need full support for PEP 517/518.
- sametmax 7y agoThey can't support reading the poetry data because it's not generic. Nor are most practical implementations of tools using pyproject.toml because they all use custom tool-namespaced fields in pyproject.toml: it's far from a standardized format yet, despite the push for using it from the people that made it. I still wish poetry would support setup.cfg in parallel. It has been working for 2 years, is compatible with pip, setuptools and the whole legacy stack, and most fields are standards and documented.
- j88439h84 7y agoI should be more specific about what I meant. I would like to see tox support `install_commands` (https://github.com/tox-dev/tox/issues/715 https://github.com/tox-dev/tox/issues/715) I would like to see pex support the `build-system` feature. (https://github.com/pantsbuild/pex/issues/660 https://github.com/pantsbuild/pex/issues/660) For pip, I'm not sure what's involved in getting everything to work generically. This is probably not simple. It has the consequence that when we don't have control over the install command, we end up needing hacks like using `extras` for ReadTheDocs (e.g. https://github.com/readthedocs/readthedocs.org/issues/4912#issuecomment-483756581 https://github.com/readthedocs/readthedocs.org/issues/4912#i...)
- sametmax 7y agoLove poetry, I hope it gets to feature parity with pew regarding project naming and jumping.
- oweiler 7y agoLooks great! So, dependency management in Python is now a solved problem?
- flixic 7y agoSeems to be solved only five or six times; not nearly enough.
- korijn 7y agoIt took me a while to learn to love pipenv. It also bothers me how many developers "blame pipenv" for any problem they can't easily explain. Anyway, has Poetry caught up to the dependency resolution provided by the pipenv lock command yet? Last time I tried it (~6 months ago), it couldn't produce working environments for my requirements. BTW, for anyone who wants to educate themselves, read the issue description here and follow the links: https://github.com/pypa/pip/issues/988 https://github.com/pypa/pip/issues/988
- tduberne 7y agoThat's funny, Poetry's README [1] claims exactly the opposite (that it resolves dependencies better than pipenv). And looking at the git history, that section of the README is 2 years old, so that claim was made when you experimented. Was the problem with dependency resolution you had similar to what Poetry describes in it README? Or are there other problems the README fails to mention? The description they make got me interested, but up to now all alternatives I tried to build moderately complex python projects failed to deliver on their promises, so definitely interested in what you have to say. [1] https://github.com/sdispater/poetry/blob/master/README.md#dependency-resolution https://github.com/sdispater/poetry/blob/master/README.md#de...
- guitarbill 7y agoI do really like the workflow pipenv provides. But last time I checked, the mayor downside of pipenv is it doesn't and won't support multiple Python versions [0]. For libraries, this can be a dealbreaker. For applications, especially ones you're deploying to a controlled environment, this isn't an issue. Been meaning to look into poetry. Not a huge fan of TOML, but hopefully all the tooling supports pyproject.toml now (I know black does, but not sure about flake8, isort, pylint, pytest, coverage). I know there's still some hacks required for tox + poetry though. [0] https://github.com/pypa/pipenv/issues/1050 https://github.com/pypa/pipenv/issues/1050
- korijn 7y agoI use pyenv to switch between python versions (or just the `py` launcher when on windows), and pipenv will follow suit. It does require regeneration of the lockfile for each python version/environment in the scenario you describe, but you can also keep them organized in a folder and swap them out for each environment. You're right that it doesn't come included in the tool though.
- zys5945 7y agoNever used poetry before. How does it compare against pipenv?
- j88439h84 7y agoMuch better
- HunOL 7y agoWorked a little with and it looks really nice and promising, but my biggest concern is number of issues on github and that vast majority of commits is done by one person.
- epage 7y agoFrom what I've gathered, the project grew faster than the original dev had time to build up other people to also monitor PRs and has since gotten very busy, slowing down the review process. I know I have several straightforward outstanding PRs. I hope this is able to be resolved so we can have a sustainable community.
- StavrosK 7y agoThat is also my worry. I've opened issues that were show-stoppers for many people and we couldn't even get the author to give us an update on when the fix would be released. I wish he'd run the project a bit better. Other than that, it's stellar software.
- micimize 7y agoI love poetry, and am using it in all my python projects, but despite that I would hesitate to recommend it for this reason. That said, "ejecting" a plain requirements.txt is possible on the alpha branch via `poetry export`
- angrygoat 7y agoThe big win seems to be the lock file - like Cargo in Rust, or yarn in the JS world. It's really, really hard to lock down dependencies reliably in Python, especially when you are talking dependencies of the primary packages you are installing (and their dependencies, ...). One solution at the moment is to run 'pip freeze' and put that in as your requirements file, but that very much feels like an 'and now I have a different problem' solution.
- deleted 7y ago[deleted]
- pbreit 7y agoCould an effort like this ever be implemented as an (backwards compatible) extension to or modification of something existing with traction?
- dissent 7y agoCan I use this to build a package that can be uploaded to a pypi repository for other people to depend on?
- mjs2600 7y agoYep, it works really well for that. https://poetry.eustace.io/docs/cli/#publish https://poetry.eustace.io/docs/cli/#publish
- deleted 7y ago[deleted]
- SmirkingRevenge 7y agoAfter years of python dev professionally, I just quit worrying about it, and embraced: - vanilla virtualenv: I don't even bother with the wrapper most of the time. - vanilla setup.py/setup.cfg: Its just really not that bad. Forget about project.toml or whatever the next big thing is. - pip-tools: ditching pipenv and using this for pinning app requirements, has made my life so much simpler.
- deleted 7y ago[deleted]
- aequitas 7y agoMost of the time I just stick with this in a .envrc file: export VIRTUAL_ENV=${XDG_CACHE_HOME:-~/.cache}/virtualenvs/$(basename $PWD) export PATH="${VIRTUAL_ENV}/bin:$PATH" python3 -m venv $VIRTUAL_ENV I just enter the directory and the environment is created/'activated'. In all the years I never had any need to use the real activate method. Just running Python and scripts directory from the bin/ in the virtualenv is enough to run 'inside the env'.
- aiven 7y agoSame. Plus script to automatically freeze dependencies into requirements.txt after each `pip install/update/uninstall`.
- theptip 7y ago`pip-tools` has served me well, but an annoying gripe that pipenv/Poetry resolves is that you can't install git links in non-editable mode, i.e. -e git+https://github.com/... Works, but git+https://github.com/... Doesn't. Also can you pin your dependencies to a specific hash? Last I checked you cannot. A major advantage of pipenv/Poetry is that the lockfile protects you against compromise of the package manager; if someone swaps in a new binary for yourdep==1.2.3, the lockfile will fail to validate the hash and you'll get an error.
- taion 7y agopip-compile added support for pinning hashes in 1.8.0 (in 2016) and support for non-editable URL dependencies in 3.7.0 (in May).
- 1337shadow 7y agoIf you like yarn I think you will like poetry. Using --user here, no per-project virtualenv, used to have a virtualenv in ~/.env, where i keep everything up to date and have many -e installs from ~/src. For production deployments: --user in containers. This means all my projects have to work with the same versions of dependencies. This means all my dependencies have to work with the same versions of dependencies. All my dependencies have to be up to date: all versions must be latest, or next to-be-released. When don't support new versions of other dependencies: I deploy a fork and open a PR. At such, I'm not spending time trying to make obsolete versions work, rather, I'm spending time making new versions work together. My perception is that this strategy offers better ROI than all others which I have tried, not only for me, but for the whole ecosystem. That's also how I made my Python 2 to 3 transition, and have been using 100% Python 3 for a while (once you've ported one big software, porting others and their smaller dependencies is easy). I'm extremely happy about this strategy and my life has just been better since I started working this way, and highly recommend it. For my own (countless) packages: I use setupmeta which simplifies auto-updates (I think it succeeds what pbr tried to achieve).
- dec0dedab0de 7y agoThis means all my projects have to work with the same versions of dependencies. This means all my dependencies have to work with the same versions of dependencies. Then why not just install everything globally?
- SmirkingRevenge 7y agoOne reason might be, that you can blow away your user dir and start clean if something goes horribly wrong.
- 1337shadow 7y agoNot to mess with system packages, but I sometimes do so in containers. I just started to switch to --user in containers so that my users would not have to rebuild python dependencies from scratch when they use it for development: with source bind mounted into /app, which is declared as the home of the app user in my container, as such, .local remains between builds on the developer checkout (like node_modules)
- 0xbadcafebee 7y agoFor dependency management needs, 99% of what you need can be done easily with vanilla virtualenv, vanilla pip, vanilla setup.py, and a Makefile. A small shell wrapper that sources bin/activate and runs your Python app is all you need to allow anyone to run your code without additional steps. The biggest problem with Python packages isn't a dependency manager, it's that Python developers don't reuse and extend existing projects. They just keep churning out new similar projects with different features. All of the most commonly used Python packages are basically one-offs that use a bunch of other one-offs. This creates a cycle of reinventing the wheel and friction in ramping up on existing technology, due to all the extra integration work and split development, and later lift-and-shift into completely different dependencies that implement the same thing. PyPI is awash with duplicate abandoned one-offs to the point that just trying to just find a useful module to do what you want can take a long time.
- bbmario 7y ago> done easily with vanilla virtualenv, vanilla pip, vanilla setup.py, and a Makefile I wouldn't call that "easy", but OK.
- 0xbadcafebee 7y agoWould you consider this easy? # Makefile venv: python3 -m virtualenv --version 1>/dev/null 2>/dev/null || \ ( echo "Please install virtualenv (python3 -m pip install virtualenv wheel setuptools)" && false ) [ -d venv.d ] || python3 -m virtualenv -p python3 venv.d ./venv.d/bin/pip install -r requirements.txt $ make venv # Running your project without setup.py $ ./venv.d/bin/python3 my-program.py # Running your project with setup.py $ ./venv.d/bin/pip install . $ ./venv.d/bin/my-program.py The setup.py you can copy from another project and customize to your liking.
- ilovecaching 7y agoPython is the worst thing to happen to SWE (specifically SWE, not ML, Education, etc). Python makes huge sacrifices in readability and efficiency for a hypothetical win in writeability. It's also fundamentally at odds with the multicore world we're living in an will continue to live in, an issue that many people have tried to fix and failed at. I can't count the number of times I've had to spend my entire night reading through an older Python service without type annotations, desperately trying to understand the types of inputs and outputs. It's just a mess, and the bias of programmers to believe that they are writing readable code (when in reality it's only readable to them) exacerbates the use of names that provide little context to solve the problem. Python is awful, and it's an uphill battle to fix it. Asyncio can solve some performance issues, but negates the benefit of Python's simplicity by destroying the sequential mental model that Programmers can easily grok. It also requires a complete rewrite of libraries and a split between non-asyncio land and asyncio land. Type annotations can make Python more readable and thus more maintainable, but gradual typing still leaves lots of unanswered questions when you're interfacing with untyped code. Python is really not good unless you are using it to prototype or build something on your own. It's led to a world of slow, buggy software that is impossible to rewrite. It's downsides are easy to measure, but its benefits are difficult to quantify.
- j88439h84 7y agoHttp://trio.rtfd.org replaces asyncio with a model that feels much closer to sequential. And it's very good.
- macawfish 7y agoI like how transparent poetry is about what's happening when you run it, and how well presented that information is. I've come to loathe pipenv's progress bar. Running it in verbose mode isn't much better. I can't be too mad at pipenv, but all in all poetry is a better experience.
- bbmario 7y agoHow do you start a new Python project in 2019? I mean, there's so much stuff. Poetry, pip, virtualenv... oh, my. With PHP, I just composer init and that's it.
- micimize 7y agoThere are a few highly dismissive comments here. While the python community has (clearly) been getting by with requirements.txt/setup.cfg/setup.py, the project/package management story is far less usable than more recently developed systems like npm and pub. With poetry+pyproject.toml, I have one file that is flat, simple, and readable. @sdispater has done incredible work on this project, and I hope they get more resources for development.
- 1337shadow 7y agoYour comment does not explain why you think that the package management story is better in npm, it's merely like saying "node-gyp is a pile of debt" and not going into details, but I can surely tell I have no idea why node_modules ends up so fat and why my npm install eats a lot of bandwidth, time and disk space ... is npm trying to workaround incompatibilities by installing different versions of a same package in the same node_modules ? I hope not, because that would be just perfect to accumulate debt in the whole ecosystem. Not to mention that I was just handed a frontend source code where building the source requires node-gyp which requires g++ and python2 :) Quick question since you seem to know npm very well, is there a better solution than this to automate npm package publishing nowadays ? sed -i "s/GIT_TAG/${CI_COMMIT_REF_NAME/v/}/" package.json (I don't have this issue with Python's setupmeta)
- deleted 7y ago[deleted]
- markandrewj 7y agoI was excited by this project, and pipenv, but unfortunately I haven't had consistent results cross platform with either. I have ended up sticking with venv because of this.
- ausjke 7y agowhat poetry can do that venv can not do? considering venv is the default tool and is solid and fairly easy to use already?
- dragonwriter 7y ago> what poetry can do that venv can not do? Venv doesn't do dependency management, it just provides an isolated python environment. By default, poetry uses venv for environment isolation, but also does dependency management.
- ausjke 7y agowhat about pip freeze to requirements.txt for manually but solid dependency management under venv? pip takes care of dependencies be it venv or poetry, requirements.txt will record what's need so you make your project reproducible and portable. what else does poetry bring to the table? If there is a strong selling point I would like to try poetry again. yarn came out great but these days I switched back to npm for the sake of simplicity of my workflow, especially npm absorbed yarn's good features. same thing happens to parcel/webpack, the neat new tool beats the old players, but then the old dominant one catches up by taking in good stuff from its smaller competitors, quickly.
- julienkervizic 7y agoNeat, definitely something I would want to give a try. I see though that it only supports pure python for building packages, does that mean that it doesn't build if you are dependent on compiled libraries? Is there also a plan to add some of the functionality of bundling tools such as web-pack into this build phase? like automated css optimization, image compression... Could be handy for some django/flask projects.
- fouc 7y agoDoes anyone know if there's a name for the "terminal" theme used on the poetry site?