15 ms·
PDM: A Modern Python Package Manager
- laike9m 5y agoI switched from Poetry to PDM and is feeling great. If there's one reason to use PDM: the maintainer fixes bugs fairly quickly, not like Poetry, where many bugs are left open and nobody is taking care of it.
- frafra 5y agoI can totally confirm that. Great support, even when the fault was mine. Well done @frostming!
- roomey 5y agoIs virtualenv not very useful? I use pipenv and find it extremely useful having my various projects separate. Also when deploying to kubernetes or whatever it works the same way
- qbasic_forever 5y agoYou kind of answered your own question. Virtualenv alone is too much work so you're using a tool like pipenv to streamline it. This is a similar idea to have less ritual and a faster developer workflow.
- johnhowardstein 5y agoIt's a single command more, and then you don't need to prefix everything with pipenv run.
- bruiseralmighty 5y agoI use only virtualenv and have never really had any issues with it for separating out my project environments. Perhaps this is more of an issue with larger projects, but I also noticed that 'no need to use virtualenv' as a positive.
- majormunky 5y agoSo this is sort of like a virtualenv, except you don't need to activate anything, it just looks within the main project folder for a __pypackages__ folder, and uses that to look for packages
- ggregoire 5y ago> I use pipenv > When deploying to kubernetes it works the same way Can you elaborate on this?
- dagw 5y agoI use pipenv Pipenv is only focused on solving one problem, deploying complete packages you wrote to a server you have complete control over. It doesn't really care about the problem of developing applications and libraries for distribution and installation by third parties and on platforms running different python version/OS etc. than what it was developed on.
- kaesar14 5y agoAnother package manager for Python is exactly what the world needed
- smegsicle 5y agoover and over until one doesn't suck
- nilslindemann 5y agoPIP doesn't suck, repeat after me. Please. Just PIP, which improves, that's the way. Let all other managers DIE.
- deleted 5y ago[deleted]
- Kinrany 5y agoHow does it compare to poetry?
- jefurii 5y agoWe keep having to repost this: https://xkcd.com/1987/ https://xkcd.com/1987/ Why don't we ever learn?
- denysvitali 5y agoThanks! I was really missing yet another package manager
- mindwok 5y agoI tried PDM earlier this year and there's a few things worth noting: - PEP582 which it is based on is still in draft, and some tools (VS Code) won't fully support it until it's accepted. - If you want to develop or test using different Python versions, you still need to use a virtual environment. PDM does handle this for you though. IMO, the Python packaging ecosystem has been a dumpster fire for a long time, but the flurry of recent PEPs that provide a standard way of defining a package with pyproject.toml have made it so much better. Now it's just a matter of the tools catching up.
- bayesian_horse 5y agoDumpster fire compared to what? Nuget? NPM? It may be that some more recent languages managed to start a saner solution and keep it sane. But really, it's a hard problem, between cross-platform support, backwards compatibility, security concerns, hosting, most authors being volunteers and so on. And still, even with "just" pip or even conda I am enjoying the Python experience more than some other packaging solutions I've seen.
- zerocount 5y agoCompared to nothing. Dumpster fire just means it sucks. To be fair, most of them do though.
- short_sells_poo 5y agoIt's objectively a dumpster fire. I don't care about other languages also being a dumpster fire on the packaging front, because I don't use other languages :) It also doesn't help me to know that NPM or .NET have rubbish packaging systems, that's their problem and not mine. I'd primarily want python's packaging system to be good. I mean, just look at this thread. Someone asks "So in light of this, what should I use for python packaging?", and they get two dozen different answers loaded with weirdness like pyenv, python-venv, virtualenvwrapper, etc... If I wasn't using python, I would've thought this is some cruel python in-joke that outsiders don't get. Just looking at those names, I'm already confused as to what the hell each are doing and why do I need them? But let's go back to pip and conda. Conda is unbearably slow. Pip is not entirely reliable in resolving versions properly. There's also not entirely cooperative interaction between conda and pip. If you use conda, you should not use pip (or just minimally) because it'll result in a mess. Yes, packaging is hard, but it feels like python has managed to solve (or not solve) it in a uniquely obtuse and bad way so far. Hopefully the slew of new PEPs will finally bring some clarity to this mess.
- tapia 5y agoSo, what's the difference with poetry? It seems pretty similar to me. Do we really need another python package manager?
- lelandbatey 5y agoThe big distinguisher of PDM is that it support PEP 582[0]. That means it works less like Pip and works more like NPM of the JS world. To quote PEP 582: > This PEP proposes to add to Python a mechanism to automatically recognize a __pypackages__ directory and prefer importing packages installed in this location over user or global site-packages. This will avoid the steps to create, activate or deactivate "virtual environments". Python will use the __pypackages__ from the base directory of the script when present. Thus, the idea of PDM is that it will create a directory, called `__pypackages__` in the root of your project and in that folder it'll populate all the dependencies for that project. Then, when you run scripts in the root folder of your project, your Python install will see that there's a `__pypackages__` folder and use that folder to look up dependencies. This style of "dependencies inside the project directory" is similar to how npm of the Javascript ecosystem works, where it creates a `node_modules/` folder in the root of your project and fills that folder with the dependencies for your project. This style of dependency management is different from other package managers such as Poetry (Python), Pip (Python), go (Golang), and cargo (Rust), all of which instead have a sort of "secret folder acting as cache of dependencies at particular versions", a folder that's usually pretty hidden out of the way, in which the package manager automatically manages the acquisition, storage, and versioning/version resolution (Poetry, Go, Cargo, all do this but Pip does not). That's a very fast and probably wrong rundown on what makes this package manager different from others. [0] - https://www.python.org/dev/peps/pep-0582/ https://www.python.org/dev/peps/pep-0582/
- jrimbault 5y agoComposer (php) also uses the "project local" approach
- divbzero 5y agoI’ve long been of the opinion that pip and venv (and sometimes pyenv) is good enough. PEP 582 is a rare instance where a new packaging proposal makes sense right away when I read it and could beat pip and venv in simplicity. It seems functionally similar to venv but has the benefit of standardizing the location of dependencies to __pypackages__/3.x/*. With venv the developer selects some arbitrarily named directory that is sometimes but not always .venv/*.
- fb03 5y agoGenuine question: If I am starting a Python project NOW, which one do I use? I have been using pipenv for quite some time and it works great but locking speed has been problematic, specially after your project grows large enough (minutes waiting for it to lock without any progress warning at all). Should I just upgrade to Poetry or should I just dive headfirst into PDM? Keep myself at Pipenv? I'm at a loss. Thanks in advance!
- stavros 5y agoUse Poetry. It does everything you need and PDM is a bit too new still.
- fb03 5y agoGreat. I have heard "anecdata evidence" that sometimes poetry fails to install a combination of packages or something along those lines, did you find any of those shenanigans in your own experience?
- akdor1154 5y ago(Not GP) Occasionally, but still use poetry. The issues happen because poetry is strict about version constraints. Pip used to be lax, so some packages could get away with poor/overly restrictive constraints.. Now pip is strict as well, and poetry is also getting a lot more common, so maintainers have had pressure to fix most of these issues.
- nbadg 5y agoI've run into this a couple of times. Typically it's when starting a new project and gradually adding dependencies. By default, when adding (poetry add <dep>), poetry adds the most recent version of the library, and in pyproject.toml that becomes the minimum possible version. If you then later try to add something that depends on that library, but isn't up-to-date on the most recent version, poetry will (correctly!) error out because it fails to find a compatible solution. For example, something like this: poetry add ingredients # most recent version: 1.0 poetry add cake # requires ingredients, but <1.0 So here, poetry would add ingredients to your pyproject.toml with version = "^1.0.0" in the first step. In the second step, when you tried to add cake as a new dependency, it would say it can't find a version of cake compatible with your pyproject.toml's requirement for ingredients. The solution is to change pyproject.toml to be more permissive about the version of ingredients, resulting in it downgrading to a compatible version. Or submit an issue and/or PR with the maintainers of cake to bump up the supported versions.
- zuj 5y agoOh, no. Please, not one more.
- sterwill 5y agoThere should be a rule that before you create a new Python package manager, you have to kill off an existing one first.
- deleted 5y ago[deleted]
- tjalfi 5y agoThere's always Jim Waldo's proposed rule for the C++ standards committee. Every extension proposal should be required to be accompanied by a kidney. People would submit only serious proposals, and nobody would submit more than two. — Jim Waldo
- kirubakaran 5y agoIt doesn't say it has to be your own kidney.
- akdor1154 5y agoCool to see an implementation of __pypackages__ support. Ill still use Poetry, but this could be paving the way for Poetry to work without virtualenvs as well one day.
- confiq 5y agoI think I'm starting to like poetry too!
- cosmotic 5y agoPlease use descriptive words instead of 'modern'. Modern, to me, in this context, means untested and unreliable.
- tamentis 5y ago"The Python Package Manager That Is Newer Than All The Others"
- framecowbird 5y ago"Have you tried TPPMTINTATO? It's great!"
- cozzyd 5y agoI can't wait for the postmodern package managers...
- cdot2 5y agopackages are really a social construct
- dangerbird2 5y agoInspired by Michel Foucault's Pypi and Punish
- tamentis 5y agoYou `pdm install requests` and it installs Django.
- diogenesjunior 5y ago>Modern, to me, in this context, means untested and unreliable. that's what PDM is
- deleted 5y ago[deleted]
- gigatexal 5y agoLooks a lot like poetry
- rossipedia 5y ago[flagged]
- carabiner 5y agoSo many of these... I now need a tool to manage the managers.
- xmcqdpt2 5y agoYou jest and yet... https://github.com/dephell/dephell https://github.com/dephell/dephell Dephell is a converter for python packaging systems. It can turn poetry files into requirements.txt, or setuptools' setup.py into pipenv's Pipfile etc. Python Packaging: There is More Than One Way to Do It
- divbzero 5y agoPDM’s support of PEP 582 seems promising. What would be even cooler is if the maintainers of pip agree with PEP 582 and incorporate it into pip itself.
- pyuser583 5y agoAnother one?
- eatox 5y agoIt's sad to see so many negative comments here. Quoting the GitHub page[0]: > PDM is meant to be a next generation Python package management tool. It was originally built for personal use. If you feel you are going well with Pipenv or Poetry and don't want to introduce another package manager, just stick to it. But if you are missing something that is not present in those tools, you can probably find some goodness in pdm. Having used PDM a bit, its ambition in my opinion may not be to replace existing tools, but rather to experiment and implement the most recent PEPs related to packaging. While you can argue about PEP 582[1] implementation (which is still in draft), PDM doesn't prevent anyone from using virtual environments, and even provides a plugin[2] to support that. PDM also implements PEP 631[3], which most other package managers have been relunctant to support or slow to adopt. [0]: https://github.com/pdm-project/pdm https://github.com/pdm-project/pdm [1]: https://www.python.org/dev/peps/pep-0582/ https://www.python.org/dev/peps/pep-0582/ [2]: https://github.com/pdm-project/pdm-venv https://github.com/pdm-project/pdm-venv [3]: https://www.python.org/dev/peps/pep-0631/ https://www.python.org/dev/peps/pep-0631/
- Quekid5 5y agoThis is understandable. But if it is experimental, please label is as such. No mention of that on the landing page right now. EDIT: Oh, I should say: If it's meant to take over the world, say so, as well!
- deleted 5y ago[deleted]
- NeutralForest 5y agoGod those PEP's are exactly what I need
- frostming 5y agoThanks for the kind words on PDM. At time of creating PDM I don't want it to be similar with any other package mangers, so I chose PEP 582, and I thought I can play more new stuff on it. But as PDM becomes mature, it is acknowleged by the Python packaging people, I also work hard to make PDM fit more people's workflow, fortunately, it has a strong plugin system. You can add virtualenv support(pdm-venv), publish command(pdm-publish) and more. In the future, I would like to see it can eventually push the iteration of PEP 582 and make it finalized.
- knowsuchagency 5y agoHaving used PDM now for several projects, it's my preferred package manager over poetry and others. Its dependency resolver is both faster and more forgiving than poetry's. I also like the built-in task management system similar to npm's.
- nilslindemann 5y agoThat's cool. A local folder named '__pypackage__', I can get that into my head. This is the one obvious way to do it.
- nilslindemann 5y agoBut please, let this be PIP, not PDM
- msie 5y agoThis sounds great. Why hasn’t it been done before? It seems pythonic.
- deleted 5y ago[deleted]
- xiaodai 5y agoNot another one... please
- throwaway2568 5y agoSuprised there is so much love for Poetry. Having only used it sparingly (and having some dependency issues with it at the time) I do not see the benefits over conda (miniforge version to easily avoid anaconda) or mamba if you care about speed.
- est 5y agoDoes it handle compiling C stuff easier? If not, then I'd stick with minimal pip3 or fullblown anaconda3
- kbumsik 5y agoI started a new Python project last month. I tried both Poetry and PDM but I decided not to use neither of them. PDM is currently basically one man show, and the Poetry's doc isn't great - The doc page seems pretty but it only describes command line usages and does not tell how I can configure metadata. Most importantly Poetry does not support the standard PEP621 yet. So I stick with this setup: - Use pyenv to manage different Python versions and virtual environments. - Use the standard PEP621 specification as a high-level dependency description: https://www.python.org/dev/peps/pep-0621/#example https://www.python.org/dev/peps/pep-0621/#example - Use pip freeze > requirements.txt as a "lockfile".
- postpawl 5y agoThat probably works for smaller projects without many dependencies, but it’s just going to install the sub-dependency versions that satisfy whatever comes last in the requirements file. The pip docs describe that situation here: https://pip.pypa.io/en/latest/topics/dependency-resolution/ https://pip.pypa.io/en/latest/topics/dependency-resolution/ The pip docs also suggest using pip-tools to create lock files. Pip-tools is only for creating lock files (it’s not trying to fix virtualenvs like poetry is), and it works great.
- kbumsik 5y agoInteresting. So use `pip-compile` instead of `pip freeze > requirements.txt`?
- postpawl 5y agoSort of - you need to make a requirements.in file that specifies only your top level dependencies, then you run pip-compile on that file to generate the requirements.txt.
- kbumsik 5y agoSo another requirements.in file ignoring PEP621? I'm confused... Maybe I will just try first. pip-tools seems to acknowledging PEP621 https://pip-tools.readthedocs.io/en/latest/changelog/?highlight=pyproject#id5 https://pip-tools.readthedocs.io/en/latest/changelog/?highli... https://github.com/jazzband/pip-tools/issues/1047 https://github.com/jazzband/pip-tools/issues/1047
- kodeninja 5y agoWhat are the fundamental limitations to python/pip supporting the installation model of npm, where multiple versions of a dependency can happily co-exist?
- privacyonsec 5y agoHow it is different from : easy_install, conda, poetry, pip, pipenv ....
- framecowbird 5y agoI've never seen PEP582 until now, but it looks concerningly short. It doesn't discuss tradeoffs, alternatives, and is nowhere near comprehensive enough in its examples. It just doesn't seem very thoroughly thought through... It has the feeling of "this is such a great and simple idea! What could possibly go wrong??"
- animal_spirits 5y agoIt's still in a draft phase
- ac130kz 5y agoI'm sorry, but I've had so much trouble setting up IDEs and had 3 major installation/update issues with PDM just from the start - it is unusable for production. Poetry has never failed for me even once, yes, it was slower back in the day, yet it worked like a charm properly resolving dependencies.
- dreyfan 5y agoPython should update their logo to that snake eating itself
- nsonha 5y agoWhy is "doesn't need to create a virtualenv" a good thing? Virtualenv is just PATH enhancement. I always thought of it to be simpler than the npm run-script magic.
- jsilence 5y agoReally wish the Python core developers would focus on this situation and almagate one official solution for installation, env and package management and one click distribution.
- derkoe 5y agoOkay now we have even one more https://xkcd.com/1987/ https://xkcd.com/1987/
- streamofdigits 5y agoMaybe dependency hell is a wicked problem and cannot really be solved? What is the gold standard to aspire to? npm is mentioned in the post but its not clear if that is just for lowering barrier to entry for new users.
- bspammer 5y agoIn the Java world, maven works really well. It’s my personal gold standard, I haven’t found anything in another language which works better.
- yessirwhatever 5y agoSure why not, python needs yet another package manager. It's not confusing enough already! /s
- bayesian_horse 5y agoI think competition in this space is good. Not all package managers will get and keep a big audience or following, but they act as independent proving grounds for new ideas and solutions.
- yessirwhatever 5y agoTell that to a sysadmin who's trying to guarantee their server uptime... I use python almost every day and I find it a cluster fuck anytime i have to install something using a specific version or update some packages. If it wasn't for containerization i would've lost hope very early on in the python packing ecosystem.
- nitred 5y agoYou can use any tool you like, but it triggers me every time some says something misinformed about `conda`. So please if you have anything negative to say about it, please read the official Managing Environments page of `conda` before doing so [1] * Please don't confuse `conda` with Anaconda, Miniconda & Conda-Forge. * You can use `conda` to manage your environment and it can download and install ALL your packages from `pypi.org` and not Anaconda. [1] https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html https://docs.conda.io/projects/conda/en/latest/user-guide/ta...
- bayesian_horse 5y agoPDM may also be a good fit for Blender, because of the per-project approach. Blender doesn't come with a package manager and has a varied relation to the system installed Python interpreters depending on platform and install choices. Scripts, Plugins etc for Blender are currently distributed in a very ad-hoc way, and it is hard to get adoption with plugins that require more elaborate dependencies, especially binary modules.
- 1337shadow 5y agoHardcoding the python version from the start, not for me ...
- solididiot 5y agohttps://xkcd.com/927/ https://xkcd.com/927/
- KolenCh 5y agoFor people finding good package manager for Python, conda is the best. (Mamba is even better and is a drop-in replacement of conda that you can even do conda install mamba and use mamba from there.) It doesn’t replace your setup.py or pyproject.toml as a maintainer, though. For that I uses poetry with pyproject.toml. As a maintainer, you need to release to PyPI first and then release it to anaconda or conda-forge (where I prefer the later.) Releasing on conda-forge seems more troublesome at first, and that process makes you trust conda-forge more than PyPI where it virtually has no barrier to entry. As an end user, most packages you can pip install with can be conda/mamba install with. For those which doesn’t exist, you can use pip install inside a conda environment. (Which is like virtualenv.) dependencies already installed via conda won’t be repeated. Conda is designed to be multi platform and multi language as well. Npm, Julia, ffmpeg, etc can be installed via conda. It is designed for scientific stack where there’s a lot of non pure Python dependencies. In this aspect, it is more like a package manager other than the dIstro provided ones, which should be compared to homebrew, Macports, nix, pkg-src, etc.