7 ms·
Poetry – Dependency Management for Python – v1.0
- EsssM7QVMehFPAs 7y agoWhy would I use this instead of Anaconda?
- dguaraglia 7y agoThey are pretty different tools. Poetry is a dependency manager and packaging tool, Anaconda is a pre-configured environment for scientific Python.
- aprdm 7y agoI guess the comparison would be to Conda which also allows for declaring dependencies and building environments.
- billfruit 7y agoThey might be referring to conda, the package manager used with Anaconda.
- aldanor 7y agoThey are referring to conda, which is a generic package manager, dependency resolution and build tool.
- fphilipe 7y agoCan you define dependencies loosely and still have a lock file in Anaconda so when you come back to the project a year later you can install the exact same dependencies at the same versions? (Honest question, not familiar with Anaconda.)
- yegle 7y agoIf you have these requirements, why not vendorize all your dependencies?
- delaaxe 7y agoYou can pip freeze
- aldanor 7y agoSo you want to define your dependencies loosely or lock them?
- fjp 7y agoYou can do both! Your general file can loosely define dependencies like greater than this version, less than this version, skip this version, etc Then you “lock” and it builds a dependency tree with the newest version of everything that satisfies your loose requirements. Then you can choose to install dependencies from your lockfile, which is exact frozen versions OR you can re-run the lock later, which will again attempt to update everything to the latest versions that satisfy your loose constraints
- lidHanteyk 7y agoWhy would I use Anaconda instead of Nix?
- adisbladis 7y agoA bit of a shameless self-plug, but you can use both Poetry _and_ Nix! https://github.com/nix-community/poetry2nix https://github.com/nix-community/poetry2nix
- sansnomme 7y agoJust to note, dev dependencies in Poetry is still rather flaky vis-à-vis projects forever in pre-release mode e.g. Black. Consider sticking with Pipenv if you have heavy dev dependencies requirements.
- djrobstep 7y agoIt definitely was in the past, but I haven't had any troubles recently. Are you still experiencing issues?
- aprdm 7y agoIsn't that Black fault more than Poetry fault? They do not have a non alpha release.
- j88439h84 7y agoNowadays its just --allow-prerelease
- hobofan 7y agoIf I read the 1.0 changelog correctly, I think you don't need that flag anymore
- fermigier 7y agoI have been bitten by this. I love Black to death, but I don't understand why they're still tagging their project as bêta when it's been so stable over the last year, at least. Anyway, the solution is simple: put black = { version = "*", allows-prereleases = true } in your pyproject.toml file. Or use --allow-prerelease as suggested by others.
- starpilot 7y agoIs there a good way to build a portable package containing dependencies for installing on an airgapped system?
- ctrlc-root 7y agoTake a look at Twitter PEX. I gave a short talk on it earlier this year (https://www.youtube.com/watch?v=abnwINA50DE https://www.youtube.com/watch?v=abnwINA50DE).
- ipsum2 7y agoAnother one: https://github.com/facebookincubator/xar https://github.com/facebookincubator/xar "XAR lets you package many files into a single self-contained executable file. This makes it easy to distribute and install... A popular example is Python application archives that include all Python source code files, as well as native shared libraries, configuration files, other data."
- viraptor 7y agoDo you depend on any native library? If not, you should be able to install everything in a virtualenv and move that whole directory.
- j88439h84 7y agoMake a wheel or use shiv. Either is very easy.
- starpilot 7y ago
- dhruvkar 7y agoI've been watching poetry for the last couple years, hoping that it is THE replacement to the virtualenv/virtualenvwrapper/pyenv workflow. pipenv never felt quite right. poetry has felt really close. I'd love to know if anyone has made this jump and their experience.
- hobofan 7y agoSame here. Been using Poetry for a few months, and it's been a blast! Before that I tried to use pipenv, but it's been too buggy and unpredictably slow with installs. I've always had bad experiences with virtualenv etc., and mostly stayed away from Python development due to that. With Poetry that has changed immensely, as I now have the reproducibility in environment I'm used to from Rust/Ruby/Node. All that said, I think there is still quite a way to go for Poetry. E.g. I had (hard to reproduce) bugs where if the install was canceled, I had to purge the whole virtualenv, as it was unable to recover itself. I'm pretty opimistic about the project as a whole though!
- siddboots 7y agoSame here. I got really excited by pipenv, but every single time I've tried it, it takes 10+ minutes to determine the dependencies and build a lock file (see issue 2284 [1]). It just makes it un-usable for a lot of my use-cases, e.g. setting up a clean environment for exploratory data analysis. Does anyone know if poetry improves in this area? [1] https://github.com/pypa/pipenv/issues/2284 https://github.com/pypa/pipenv/issues/2284
- hobofan 7y ago(See my other comment) Yes, I haven't encountered any of the slowness in dependency resolution that Pipenv has with Poetry. It does still feel a bit slow compared to package managers from other languages, but not in a buggy way, so I would attribute it to not being that optimized yet.
- Caligatio 7y agoWhile I'm excited that Poetry hit 1.0, I'm actually more excited about the fact more devs were added to the maintenance team and that it transitioned to a project namespace. Sébastien was previously the sole dev with commit access so I'm hopeful this gives the project a much healthier future.
- oefrha 7y agoThe project org has two (public) members and the second highest contributor has 10 commits with <600 lines. It would inspire more confidence if the project is transferred (invited) to pypa some day.
- calibas 7y agoMessing around with major/minor Python version compatibility and dependency conflicts has always been my least favorite part of Python. A tool like this is much appreciated!
- skept 7y agoIt's not clear from the current title but Poetry v1.0.0 was just released: https://github.com/python-poetry/poetry/releases/tag/1.0.0 https://github.com/python-poetry/poetry/releases/tag/1.0.0
- Townley 7y agoEven putting aside it's approach to dependency management, Poetry has a fond place in my heart for making it so much easier to publish to PyPI. If poetry did nothing but simply the process of publishing and updating python packages, it would still be an amazing tool.
- hiccuphippo 7y agoI've read of another project for publishing to PyPI called flit. Anyone knows how they compare to each other?
- brettcannon 7y agoPoetry tries to help you manage your whole project while flit is just for building your library (and optionally uploading; you can always use twine to do the upload). It's all-in-one versus one-job-one-tool.
- divbzero 7y agoSimpler PyPI publishing sounds very appealing. I still don’t feel hungry for new dependency management solutions (requirements.txt + virtualenv works reliably for me) but I do like the idea of refactoring setup.cfg and setup.py for simpler PyPI publishing.
- juanuys 7y agoLooks like Randall's comic needs an update! https://github.com/dephell/dephell https://github.com/dephell/dephell https://xkcd.com/1987/ https://xkcd.com/1987/
- gonational 7y agoNeeds updating to include pyenv also.
- fermigier 7y agoBig up to dephell (https://github.com/dephell/dephell https://github.com/dephell/dephell) which makes it pretty straightforward to convert from one dependency management system to another, while the dust settles.
- merricksb 7y agoBig discussion about project 4 months ago: https://news.ycombinator.com/item?id=20672436 https://news.ycombinator.com/item?id=20672436
- yegle 7y agoWhat's the advantage of using poetry compare to vendorize your dependencies and check in the whl files (which can be manylinux binary) into your version management system?
- yegle 7y agoThis is actually pretty straightforward: pip wheel -w ./wheelhouse -r requirements.txt Then you can modify your script to insert every whl in sys.path. Add wheelhouse directory to version control and you get a repo that can easily reproduce anywhere that manylinux is supported.
- dragonsh 7y agoWe have moved our dependency management from pipenv to poetry and hasn’t been more happier than now working with and publish Python modules. Now we use pyenv with poetry to run all our projects in Python with venv in project instead of decided by poetry default settings. We changed the settings with: poetry config settings.virtualenvs.in-project true After this in hgignore or gitignore added .venv folder. Now everything is the way we wanted, with Pipenv many times faced issues that the project worked with requirements.txt but not in Pipenv. Didn’t have this issue moving to poetry yet. Also with emacs support and automated deployment tools the predictability of poetry made our team easy to work with and publish Python application modules.
- ageofwant 7y agoNice. I do however much prefer keeping the venv out of the source tree where they do not interfere with `tree` and `grep` and whatnot. I'm curios what benefit you see with having the venv cluttering the work-space ?
- dragonsh 7y agoWe setup systemd with gunicorn or uwsgi start in the project venv. So when they are created inside the project folders is easy to have the automated deployment for Python softwares easy with systemd services or local scripts. Our developers use mac, windows and Linux for development and each one have different location of venv with poetry, if we start gunicorn or uwsgi they may make mistakes. With predictable venv location all can work the same way from within project directory.
- delaaxe 7y agoYour app is self contained in a single folder (not 2), run scripts are simpler (can use relative paths like ./venv/ Scripts/foo, no need to “activate”...
- wp381640 7y agoIt took me a while to be convinced to switch to in-folder venv but the benefits are * VS Code configuration so much easier as you can point to the correct env using a relative path in settings.json and check that in * Predictable paths to execute tasks / build * No outside of development tree vens that you forget to clean up or forget to deactivate * No different to node_modules, .git or similar when it comes to "polluting" the local project tree as you can .gitignore it
- acemarke 7y agoFor whatever reason, I've never understood the point or actual benefit of virtualenvs. Having lived more in the JS ecosystem for the last several years, my ideal workflow would be a copy of how the Yarn package manager works: - Top-level dependencies defined in an editable metadata file - Transitive dependencies with hashes generated based on the calculated dependency tree - All dependencies installed locally to the project, in the equivalent of a `node_modules` folder - All package tarballs / wheels / etc cached locally and committed in an "offline mirror" folder for easy and consistent installation - Attempting to reinstall packages when they already are installed in that folder should be an almost instantaneous no-op PEP-582 (adding standard handling for a "__pypackages__" folder) appears to be the equivalent of `node_modules` that I've wanted, but tools don't seem to support it yet. I'd looked through several Python packaging tools over the last year, but none of them seemed to support it yet (including Poetry [0]). The only tool that I can find that really supports PEP-582 atm is `pythonloc` [1], which is really just a wrapper around `python` and pip` that adds that folder to the path. Using that and `pip-tools`, I was able to mostly cobble together a workflow that mimics the one I want. I wrote a `requirements.in` file with my main deps, generated a `requirements.txt` with the pinned versions and hashes with `pip-compile`, was able to download and cache them using `pip`, and installed them locally with `piploc`. Admittedly, I've only tried this out once a few weeks ago on an experimental task, but it seemed to work out sufficiently, and I intend to implement that workflow on several of our Python services in the near future. If anyone's got suggestions on better / alternate approaches, I'd be interested. [0] https://github.com/python-poetry/poetry/issues/872 https://github.com/python-poetry/poetry/issues/872 [1] https://github.com/cs01/pythonloc https://github.com/cs01/pythonloc [2] https://github.com/jazzband/pip-tools https://github.com/jazzband/pip-tools
- Izkata 7y ago> - All dependencies installed locally to the project, in the equivalent of a `node_modules` folder That's basically what a virtualenv is, the python version of a node_modules directory. You just need to explicitly activate/deactivate to move in and out of that environment, instead of it being based on the directory tree. And the requirements file is equivalent to the "dependencies" section(s) of package.json
- 7y ago
- MegaLeon 7y agoI thought the billions ways to sort out dependencies and environments Python were its messiest part. I've been using a custom little batch script on windows to streamline that a bit as I mainly use python for small utilities as opposed to big projects - https://gist.github.com/leoncvlt/10e67d9415e61eff0f5010ef6fec51d8#file-evenvo-bat-L33 https://gist.github.com/leoncvlt/10e67d9415e61eff0f5010ef6fe... - but interested in giving this a spin!
- guitarbill 7y agohopefully v1.0 now means it's pretty stable. using poetry so far has been painful, the gains you got with the improved dependency resolution were unfortunately somewhat impacted by crazy annoying bugs, like `poetry shell` not actually activating the virtual environment [0] [0] https://github.com/python-poetry/poetry/issues/571 https://github.com/python-poetry/poetry/issues/571 edit: gah, still hitting baffling errors: $ poetry env use /usr/local/bin/python3 [NoCompatiblePythonVersionFound] The specified Python version (3.7) is not supported by the project (^3.7). Please choose a compatible version or loosen the python constraint specified in the pyproject.toml file. $ poetry env use /usr/local/bin/python3 [NoCompatiblePythonVersionFound] The specified Python version (3.7) is not supported by the project (>=3.7). Please choose a compatible version or loosen the python constraint specified in the pyproject.toml file. v1.0.0 generally seems to work smoother though
- RocketSyntax 7y agoit has a nice comparison to pipenv
- dec0dedab0de 7y agoI don't think that was a nice comparison. It just says that he doesn't like the pipenv cli, and some decisions made, but doesn't say why or how poetry is different. The only thing it goes into detail with is a bug in pipenvs dependency resolution, and how their resolver avoids that bug.
- lkjaero 7y agoThis looks pretty nice! Definitely reminds me of the node ecosystem.
- SaberTail 7y agoDoes anyone know if the situation with using `poetry` with `pyenv` has gotten any better? I like poetry's approach, but I had a very difficult time getting it to play nice with my system last time I tried.
- beisner 7y agoI use this workflow with poetry 1.0.0 just fine
- himynameisdom 7y agoSince pipenv is pretty much dormant, this seems to be a nice alternative. Thanks for sharing!
- smnrchrds 7y agoThe recommended installation method still doesn't work without manual intervention. For the reference, I use WSL with the latest Ubuntu LTS version. - I tried using the recommended installation method of curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py | python Didn't work. I don't have python installed, just python3. Fair enough. - Using the above method, but replacing `| python` with `| python3`: The installation works. But trying to invoke poetry gives this error: /usr/bin/env: ‘python’: No such file or directory Turns out poetry executable is calling python in the shebang. - Manually changed shebang in `$HOME/.poetry/bin/poetry` to `#!/usr/bin/env python3`: Now poetry runs successfully. I mean, it works, but it is fragile (shebang probably needs to be manually reset at every update), not beginner-friendly, and not ergonomic. Installing using pip3 has the same issue. In comparison, pipenv is much easier to install. I was hoping this issue would not be present for poetry v1.0, but unfortunately it is.
- blaisio 7y agoI had the same experience. It tells me that poetry is on the right track but not ready for serious use yet.
- yellowapple 7y agoCurious what's stopping you from symlinking python3 to python? In fact, I'm surprised the package manager didn't already do that (I thought it did for Debian and its derivatives, including Ubuntu). Does WSL throw a wrench in that or something?
- rat87 7y agoBecause python link/binary is python 2, which you might also have installed Being able to rely on python being python 2 and python3 being python 3 is pretty useful
- BerislavLopac 7y agoThe problem is exactly with you having `python3` but not `python`. The convention is to use them both only if you have both Python versions 2 and 3; otherwise there should always be a `python` command on your system. If you insist on having only `python3` you will keep running into problems. My recommendation is to either create an alias for `python3`, or even better use `pyenv`: https://github.com/pyenv/pyenv https://github.com/pyenv/pyenv