14 ms·

How to improve Python packaging

292 points by Kwpolska 4y ago

Spivak 4y ago
Not being able to exclude system packages from resolution when using __pypackages__ is a pretty glaring omission that will force people to continue using venvs until it’s fixed.
deleted 4y ago
[deleted]
matsemann 4y ago
Because of the insanity of python I run everything in Docker (through compose). No more issues with it not working on some dev's computer because of a missing wheel, or that they need to have X and Y c++ toolchain packages installed locally etc. No more trying to fix a broken setup after upgrading python or poetry versions, just "docker compose build" and you're up and running. No spending days getting a freshly cloned project up and running. Then I point Pycharm to use that as my remote interpreter through docker compose. Not watertight, but a hundred times better dev experience.
- DangitBobby 4y ago
  I've tried this a couple of times but just generally not liked the experience and gone back. When you get a shell into the container so you can use your project's python env, the tools you have installed on your machine (bashrc, rg, aliases) aren't available. I ultimately prefer the hassle of managing environments natively over forfeiting the use of my tools or juggling shells.
- LtWorf 4y ago
  .deb packages…
  - groestl 4y ago
    Do you happen to have a link to a python application that ships it's own dependencies and has state-of-the art debian packaging?
    - vbernat 4y ago
      It depends what you mean by state-of-the-art, but it's possible with dh-virtualenv: https://github.com/vincentbernat/pragmatic-debian-packages/tree/master/puppetboard/debian https://github.com/vincentbernat/pragmatic-debian-packages/t... (not something that can become an official package as it goes against Debian policies)
    - LtWorf 4y ago
      > that ships it's own dependencies You mean a link to a malformed .deb packages? It's not difficult to do but it's also the wrong way to do it.
      - groestl 4y ago
        Honestly, there is a lot more I'd like to do with my life than caring about some debian packaging practices when all I want to do is ship a company's internal application with it's exact dependency versions. But it needs to work, and, e.g. "The virtualenv relocate-able flag has been always experimental, and never really worked;" So in practice we're back to shipping whole distributions using Docker (or, like we did, abandon Python as an option for development but keep .deb packages for deployment).
  - u801e 4y ago
    On that note, why are we trying to use language specific packaging tools to build packages rather than just building OS specific packages where dependencies are handled by apt for deb packages or dnf for rpm packages? These are language agnostic and can get the job done.
    - groestl 4y ago
      Because I have dozens of colleagues, each with their own os/distribution, developing an application that needs to lock down dot releases of dependencies to the versions running in production and update them at the same time. There is no way to do that using distribution specific tools without going insane.
      - u801e 4y ago
        >> why are we trying to use language specific packaging tools to build packages rather than just building OS specific packages where dependencies are handled by apt for deb packages or dnf for rpm packages? >> >> These are language agnostic and can get the job done. > Because I have dozens of colleagues, each with their own os/distribution, developing an application that needs to lock down dot releases of dependencies to the versions running in production I assume that the production environment isn't running dozens of os versions and distributions. For development, using a VM or container running the same os version and distro that's used in production and using that OS's package format for packaging the software and installing it in the dev environment (on the container or VM) would work. You're testing to see if the software works in the production environment, not someone's preferred os/distribution. > and update them at the same time I'm not as familiar with apt, but dnf has a version lock feature that would allow you to lock down the dependencies to specific versions. You could update them and test them during development and update the version lock file to pull down the updated version of the dependency when updating production.
        groestl 4y ago
        For every dependency that's missing from the upstream distribution in the exact version we need, we'd need to package that appropriately. We have nothing to gain here, nobody pays for that.
- goodpoint 4y ago
  No way. Docker is not designed for security.
  - matsemann 4y ago
    Well, it's probably more secure than doing `pip install` and letting every dependency run whatever to install itself on your host...
    - goodpoint 4y ago
      Actually not at all assuming you are using a VM for a production workload. Additionally, even if you only rely on a container - still no: other container engines have a lower attack surface.
      - matsemann 4y ago
        I'm talking about local dev here.
- wheelerof4te 4y ago
  When the solution to basic problem like this is "use Docker", you realise how deeply flawed modern software development (in Python, at least) is.
  - jessekv 4y ago
    Not sure I agree with "deeply flawed" verdict. The flaws of python's packaging system (as well as the GIL and other hangups) emerge from the tradeoffs of inventing a language optimised for reusing and connecting as many disparate system binaries as possible. Its not surprising that such a server scripting language is tightly coupled with the server environment that it runs in. Docker is just one way to ensure a reproducible server environment across machines. What do you think it could have done differently?
    - lmm 4y ago
      What could have been done differently: - Make dependency resolution deterministic by default. Unfortunately the whole ecosystem has to buy into it, but the benefits are huge. - Stop building castles of sand by making more layers of tooling that have to run in Python (and will therefore run in some random poorly managed Python). The language install can't manage the build system install - if anything the build system install should manage the language install. Adding pip to the language distribution was such a backwards decision that it marks the point where I gave up on Python ever fixing their stuff. - Ignore Linux system package managers (apt etc.), they have a fundamentally broken model and will infect your language ecosystem with that breakage if you try to cater to them.
  - rtpg 4y ago
    to be fair to Python, almost every ecosystem is in "use Docker" land, even Go binaries will offer docker images!
- groestl 4y ago
  I secretly believe Docker only exists because of how f'up Python's distribution story is.
  - jessekv 4y ago
    I also have often wondered how much Docker owes its success to the shortcomings of python.
optimalsolver 4y ago
Python packaging is a solved problem: https://python-poetry.org/ https://python-poetry.org/
- throwawaaarrgh 4y ago
  as a Poetry user myself, ummmmmmm. no. lol
- revskill 4y ago
  Yes, there should be (one way) to do packaging.
- Kwpolska 4y ago
  Poetry is one of the better options, but its nonstandard pyproject.toml is not ideal. PDM is basically Poetry, with the metadata standard in place, and with support for __pypackages__.
  - frafra 4y ago
    I would like to add that PDM also supports having multiple dependencies groups, and the developer is superfast at fixing issues.
- c7DJTLrn 4y ago
  It's not really solved if you have no choice but to work with codebases that don't use Poetry (quite common). There's 14 tools for people to choose from that aren't going away any time soon.
- wlpu 4y ago
  Poetry is one of the worst modern solutions to python packaging, either pick hatch or PDM, they both have more and better implemented features plus their codebases aren't a complete mess
deleted 4y ago
[deleted]
woodruffw 4y ago
It's not very well documented, but the PyPA tools do provide a unified experience when used correctly. Here's a PyPA project (FD: one I work on) that uses a single pyproject.toml to handle all aspects of packaging (and most non-packaging tool configuration, to boot)[1]. With a single file like that, the only thing you need to do to start a local development environment is: python -m venv env && . env/bin/activate python -m pip install .[dev] (We provide a `make dev` target that does that for you as well.) Similarly, to build distributions, all you need is `build`: python -m build [1]: https://github.com/pypa/pip-audit https://github.com/pypa/pip-audit
- qbasic_forever 4y ago
  Nowhere in pypa documentation is your simple workflow described or mentioned. Instead it's a jumble of links to a myriad of tools including hatchling, flit, pdm, etc. and basically just a shoulder shrug, 'I don't know, figure it all out yourself' message. This article makes a great point that the current pypa 'guidance' is too confusing and vague for actual end users (i.e. people that don't work directly on pypa or haven't been studying python packaging for decades).
  - woodruffw 4y ago
    I agree that it's confusing. That being said, there is an official PyPA tutorial that goes through the exact steps need to produce the commands I suggested, so I don't think it's accurate to say that it's nowhere to be found[1]. Edit: Unfortunately it's easy to confuse the above tutorial with this one[2], which is specifically for setuptools. So I can appreciate end user confusion around the documentation, particularly in that respect! [1]: https://packaging.python.org/en/latest/tutorials/packaging-projects/ https://packaging.python.org/en/latest/tutorials/packaging-p... [2]: https://packaging.python.org/en/latest/guides/distributing-packages-using-setuptools/ https://packaging.python.org/en/latest/guides/distributing-p...
    - qbasic_forever 4y ago
      No I'm looking at link 1 and specifically have an issue with the huge table of options for how to setup your pyproject.toml--it has tabs for hatchling, flit, pdm, etc. but _zero_ description of why I would want to use one of those tools. It's just more confusing, like you're being thrown a bag of parts and no description of what to build or how to do it. To be honest that entire pypa doc should be like two paragraphs long instead of 1000+ words. It should be basically, "ok to package your python app just run <official python tool to init a package>, that's it you're done!". Every decision should be made for me, like it is with npm, cargo, etc. I shouldn't have to think beyond running one command. That's what python end users need. We don't need a million tools and huge docs with seemingly no vision or goal. We need to get shit done and the less time we faff about with packaging and tooling the better.
tempest_ 4y ago
I've pretty much settled on poetry for most things at this point. It still has a ton of rough edges like error messages still closer to a stack trace than something actionable and it's slow as hell resolving but it does the job 90 percent of the time without issue. Also running everything in docker now anyway which side steps some other problems.
- imglorp 4y ago
  That's where we are as well, but yes the problems you didn't sidestep with poetry+docker are still there: the pypi fecosystem out there does not stop quantum fluctuations just because you want it to. If you pin too much you're not getting security fixes and if you don't pin enough, you get bitten every time a tertiary dep changes that you never asked for. Oh yeah and there are now trojans upstream almost daily.
  - tempest_ 4y ago
    Yeah but that trade off is entirely separate. We trade security for speed(or "velocity" if you want to be jargon about it). I just pin everything and go through my projects every couple weeks and bump the deps (unless some really big CVE hits the news).
  - gen220 4y ago
    Does any FOSS package ecosystem have a good solution to this particular set of problems? I know Go has historically done something interesting with selecting the "minimum viable version number" for each dependency, but that's the only relevant idea that comes to mind. With PyPi, at least it's relatively trivial to self-host a pypi instance and self-manage dependency updates. In the Python ecosystem (vs e.g. Node), at least the total set of first+second-order dependencies for most normal projects is quite small (e.g. medium double digits to low triple digits). It doesn't feel too painful to manage that magnitude of upgrades on a monthly or quarterly basis.
    - lmm 4y ago
      In the JVM ecosystem you get the version your immediate dependency was built/released against, transitive dependencies are never silently upgraded (though if you want to upgrade then it's one command). IME that's a much better default; it does mean that when a security issue is found in a low-level library you get everyone doing a bunch of releases that just bump their dependencies, but the consistent behaviour is very much worth it.
dataflow 4y ago
Regarding system package upgrades breaking virtual environments: would hard links address this? If the system package manager removes a file, your venv still has a hard link to it so it doesn't see a difference.
- Denvercoder9 4y ago
  It would also prevent the Python in your venv from receiving any updates when the system-wide Python is installed. At that point it's probably easier to just use your own Python installation, e.g. with pyenv.
  - dataflow 4y ago
    Sure but installing a new copy would defeat the space savings. The goal of this was to get the desired effect without wasting as much space. I wasn't trying to optimize purely for ease of use. Put another way, my suggestion was to migrate from symbolic links to hard links.
arunkant 4y ago
One more thing we need is app distribution for desktop apps written in python. I need to be able to package python app as single binary. There are tools like pyoxidizer https://pyoxidizer.readthedocs.io/en/stable/ https://pyoxidizer.readthedocs.io/en/stable/ . hopefully it become standard in python community
- ctoth 4y ago
  You probably already know about this, but Nuitka[0] is pretty great for building distributable Python apps. [0]: https://nuitka.net https://nuitka.net
rektide 4y ago
Cant resist digging at Node.js even when writing up how infinitely better Node.js is at dealing with packages than python, haha: > Let’s try removing is-odd to demonstrate how badly designed this package is: You literally just deleted is-even 's dependency on is-odd then have the audacity to be shocked that it broke? There's a lot of hatred for the small package philosophy of node.js, but it's also a huge win, stands a good chance of being why javascript has been such a winner, gotten so far: very explicit small things that say what they do on the tin. Rather than repeat yourself by making a copy pasted is-even and then maintaining both, it makes perfect sense to compose functionality, to build off what we have. And it is easier to understand the scope of what a package is & what it can do when it is explicitly limited in scope. This is another place where there is lots of loud vociferous animosity against what is, but it's there for good reason. And with rare rare exception- left-pad rage-quit deliberate breakage, for example, it serves well. With the exception that yes, inventorying your stuff is hard.
- zdragnar 4y ago
  It would be nice if major node packages invested time in re-inventing the wheel, just a little bit. Back when I used it, I really appreciated how little bloat Hapi added to node modules, compared with webpack for example. Obviously there's a world of difference in what problems the two solve, but still...
- kdazzle 4y ago
  I’m biased, but I do like Python’s package management much better than Node. Even just regular old virtualenvs. Can’t tell you how many times deleting the node packages dir and reinstalling fixes a weird issue, but that’s happened very rarely for me on the python side. Also, having to comb through a bunch of node packages of dubious quality to find some sort of standard approach happens way too often. Like take python requests vs axios.
  - revskill 4y ago
    I've tried with venv before, and the console (Iterm) just get broken.
  - 4y ago
samwillis 4y ago
This post is prompted by the survey of Python users, their feedback, and a current thread [0] on the Python forms discussing a way forward. I read the thread the other day, it's long, and there are a lot of opinions. Towards the end of this post there is an interesting observation: > Discourse, the platform that the discussion was held on, shows the number of times a link was clicked. Granted, this count might not be always accurate, but if we assume it is, the link to the results summary was clicked only 14 times (as of 2023-01-14 21:20 UTC). The discussion has 28 participants and 2.2k views. If we believe the link click counter, half of the discussion participants did not even bother reading what the people think. Along with that, my concern is that there are too my cooks in the kitchen. A packaging system is never going to cater for more than 90% of use cases, but there will likely be a higher than average representation from people who would sit in the 10%. There is a danger of them never being able to agree on a solution that solves all problems, and then never launch anything. It would be far better to get something out, have a stake in the ground, and build from there. It also desperately needs to be "Core Python", my understanding is that the PyPA (Python Package Authority) is somewhat disconnected from the core Python development. Ultimately they need one system that is front and centre, that is the "officially endorsed" tool, and very clear messaging on that. 0: https://discuss.python.org/t/python-packaging-strategy-discussion-part-1/22420 https://discuss.python.org/t/python-packaging-strategy-discu...
- forkerenok 4y ago
  Thanks, this is an interesting nugget. Did you intend to append a link reference for [0]?
  - samwillis 4y ago
    Yes, edited. Thanks.
- mkoubaa 4y ago
  I always like to say that's barring divine inspiration, it's impossible to make the right choice. It's far more possible to make a choice and then work hard to make it right
- 4y ago
jasonhansel 4y ago
Over time, I've grown to appreciate "pip-tools." Since it's a dead-simple extension of pip, I wish it could be upstreamed into pip itself; that seems like the most straightforward way of fixing a number of Python's packaging issues.
- pdw 4y ago
  Yes. It usually gets glossed over, but if you're building a Python application it's all you need.
- delijati 4y ago
  I work on mutlirepos and i really dislike gits subtree, subrepo i use https://pypi.org/project/zc.buildout/ https://pypi.org/project/zc.buildout/. Yes i know i can do [1]. But editing on multirepos at the same time i can only do with zc.buildout [2] Still not perfect but it does the job. [1] $ cat req.txt requests git+ssh://git@gitlab.com/foo/bar/amber.git@0.4.0 git+ssh://git@gitlab.com/foo/bar/rebam.git@0.4.0 [2] $ cat buildout.cfg [buildout] extends = versions.cfg extensions = mr.developer auto-checkout = \* develop = . show-picked-versions = true update-versions-file = versions.cfg sources-dir = git-sources parts = py eggs = amber rebam hammer [sources] amber = git git@gitlab.com/foo/bar/amber.git@0.4.0 rebma = git git@gitlab.com/foo/bar/rebma.git@0.4.0 [py] recipe = zc.recipe.egg eggs = ${buildout:eggs} interpreter = py-backend dependent-scripts = true
  - jamesfinlayson 4y ago
    Ah, I haven't used buildout in years (I remember using it a lot when working with Plone). I used it for a personal project but gave up a few years ago as some piece of the puzzle seemed broken and abandoned (something hadn't been updated to use a newer version of TLS or something). I liked buildout though - it was a good system with its bin directory.
- vlovich123 4y ago
  Yup. I like to make it so that each executable has a symlink to a shell script. The shell script checks if there’s a virtual environment set up with the packages in the requirements.txt installed (it takes a snapshot of the file because virtualenv doesn’t have a DB to query cheaply). Once the environment is set up, it dispatches to running from the virtualenv. That way when you update a requirements.in file it recompiles it (if the txt is out of date) and installs any new packages, removes packages that shouldn’t be there anymore and updates ones whose version changed (if there’s any changes found). It also lets you trivially run tools with disparate requirements.in without conflicts because each is siloed behind its own virtualenv. This makes it a trivial experience to use these tools in a shared repo because there’s no worrying about packages / needing to remember to run some command before the right environment is set up. You just modify your code and run it like a regular command-line tool and packages automatically get deployed. It’s also amenable to offline snapshotting / distribution. In fact, I used this to distribute support tooling for factory lines of the Pixel Buds and it worked extremely well.
Lyngbakr 4y ago
It seems most people agree that Python's packaging isn't great, but, conversely, is there a language where most people agree that the approach to packaging is awesome? I mean, what's the gold standard to aspire to?
- nixpulvis 4y ago
  Ruby gems and Cargo are pretty awesome. Also, I find I like Arch linux's Pacman quite a bit, though that's a slightly different use case where versioning isn't resolved.
  - oconnor663 4y ago
    Cargo is a gold standard imho, but there are definitely some simplifying decisions it makes that might not fit Python: - Dependencies and executables can run arbitrary code during build. For example, Cargo knows almost nothing about how to build C code, and the common workflow is to pull in the popular `cc` library for this and call it in your build.rs. - There's mostly no such thing as "installing a library", and each project builds its dependencies from source. This is baked in at the Cargo level (the install command just doesn't work for library crates) and also at the language level (no stable ABI besides "extern C"). - Related to that, the final product of a Cargo workflow is a mostly-statically-linked binary, so there isn't really an equivalent to virtualenv.
    - Kwpolska 4y ago
      > - Dependencies and executables can run arbitrary code during build. For example, Cargo knows almost nothing about how to build C code, and the common workflow is to pull in the popular `cc` library for this and call it in your build.rs. Python packaging involves arbitrary code execution, even today. While it’s less obvious than it was with `setup.py`, if you’re installing from source, the package can specify any build backend it wants, and that build backend can do anything. This could be mitigated by having an allowlist of build backends, but who would be responsible for vetting the tools, and how would it be guaranteed that an allowed tool is not taken over by someone hostile, and that an allowed tool does not have a system to run arbitrary package-provided code?
josteink 4y ago
Every time someone asks me what I think is wrong with Python packaging, I’ll show them this link. Saved. Thanks for sharing!
DangitBobby 4y ago
I have a single, simple script (not a package!) that has dependencies. Actually, I have a few of these, just sitting in /usr/local/bin so I can execute them whenever. How should I be managing environments for these scripts? Do I install dependencies in shared system python? Should I create a shared venv? Where should I store it? Any tools out there that make this decision for you and manage it? Just the fact that homebrew occasionally updates my installed Pythons and breaks everything makes me reluctant to use Python as a scripting language at all. It's pretty reliable when I have a folder dedicated to a project (poetry is good for this) but I'm pretty fed up with how brittle it all is.
- DangitBobby 4y ago
  In an amazing coincidence, I just found another front-page post exploring this same topic! For Python, it seems to suggest using nix-shell. 1. https://dbohdan.com/scripts-with-dependencies https://dbohdan.com/scripts-with-dependencies
  - DangitBobby 4y ago
    My comment with an example of a single file script which can specify its own dependencies: https://news.ycombinator.com/item?id=34393630 https://news.ycombinator.com/item?id=34393630
- wpietri 4y ago
  I don't understand how there still isn't a good answer for this. It seems like such an obvious need, and for years I've heard go being promoted for solving this problem. I get why Python didn't start out with an answer for this, but in an era of 20 TB hard drives, I'm more than willing to burn space for ease of install.
- fleekonpoint 4y ago
  I use a separate directory and venv for each script. To execute the script, I use a shell script to call the venv's python interpreter. This is also how I use python scripts with cron/systemd. #!/bin/bash # myscript.sh venv/bin/python3 myscript.py You could also skip the shell script and use aliases in your .bashrc.
benreesman 4y ago
Trying to build Python under Nix (i.e. solid, not “close enough) is an education in how fucked up the Python packaging ecosystem is. Which is a shame because Python is an amazing scripting language, the de facto numerical computing standard, and probably a better bash for most bash use cases.
- uranusjr 4y ago
  One of the problems with Python packaging is Python Core Devs view “Python packaging” and “packaging Python” as two different things, but most people don’t bother with (or can’t tell) the difference :)
mkoubaa 4y ago
I use sites as composable mini venvs, but there's no tooling for that
charliermarsh 4y ago
It's interesting to note that PEP 582 (i.e. __pypackages__) mode used to be the PDM default -- and was sort of its flagship feature -- before being made opt-in with 2.0 release, the explanation being that the PEP had stalled, and that editor and IDE support for virtualenvs is much better (https://www.pythonbynight.com/blog/using-pdm-for-your-next-python https://www.pythonbynight.com/blog/using-pdm-for-your-next-p...). If you read through the discussion on the Python forum, one of the counterarguments to the PEP is that "If existing tools can already support this, why do we need a PEP for it?" But presumedly a PEP would help push the ecosystem to accommodate __pypackages__, and to solve the aforementioned problems (like broader editor and IDE support). For what it's worth (as someone that builds Python tooling full-time): I'm generally a fan of moving to a node_modules-like model.
- charliermarsh 4y ago
  Oh, also, I think I've read the entire Discussions thread on this PEP, and from what I can tell, the PEP is missing champions who are eager to push it forward. Part of the problem, I'm guessing, is that the PEP kind of sits in a weird space, because it's not up to the PyPA to "approve" it, or whatever -- PEPs are approved by the Python Steering Council. Per Brett Cannon (Steering Council Member): > Either consensus has to be reached (which it hasn’t) and/or someone needs to send this to the SC to make a decision (although it is a weird PEP in that it also impacts packaging, so it isn’t even clear who would get final say). https://discuss.python.org/t/pep-582-python-local-packages-directory/963/239 https://discuss.python.org/t/pep-582-python-local-packages-d...
- Kwpolska 4y ago
  That tooling argument is quite weak. PDM is the only tool I was able to find that has support for __pypackages__, the paths it uses seem to be slightly different to the PEP wording, and it uses a $PYTHONPATH/shell hack (`pdm --pep582` adds a folder with `sitecustomize.py` that does the magic). How do you change the $PYTHONPATH used by an IDE (and its magic features)?
  - charliermarsh 4y ago
    I agree!
baggiponte 4y ago
been using pdm since at least one year and have no regrets: it's the go tool and we should flock to it!
charliermarsh 4y ago
> There are two tools for dealing with packages in the Node world, namely npm and Yarn. Not to mention pnpm. Or Bun. Or even Deno. All of which deal with packages, and make different choices. This isn't to be a pedant, but I think it's a reasonable example of why it's less important that there's a single way to do a thing, or even a single tool vs. multiple alternatives; instead, what I care about is that I don't have to string together tens of tools to do that thing, and that there's some degree of compatibility and standardization in the API between those different alternatives.
- abraxas 4y ago
  Well, no. The JS package ecosystem is a slightly lesser mess than the Python ecosystem but only slightly so. I think it's a bad example to base any redesign off of. At least the author brought up .NET which mostly has its shit together when it comes to builds and packaging.
hkgjjgjfjfjfjf 4y ago
[dead]
sam345 4y ago
The main reason I have stayed away from Python. packaging is a mess!
- c7DJTLrn 4y ago
  And one of the reasons I moved away from it! My blog post on the subject was posted here and it was very unpopular despite having similar points and also making comparisons with Node. There's nothing special about Python that makes it worth enduring this pain. We have other languages with a better developer UX.
  - EdwardDiego 4y ago
    I'd be keen to read your blog post if you have a link?
    - c7DJTLrn 4y ago
      https://cedwards.xyz/breaking-up-with-python/ https://cedwards.xyz/breaking-up-with-python/ Bear in mind this was not written for an audience, let alone a HN audience.
      - EdwardDiego 4y ago
        Thanks, appreciate the link, I'll be back to yell at you shortly :D ...and having read it, the only thing I'd yell is that "I AGREE WITH ALL YOUR PAIN POINTS". Especially around documentation and package management. I do like type annotations for improving the reading experience of Python codebases, even if you're not using a type checker, but trying to get people to use them consistently for new code in an org that has a large pre-existing codebase involves either a lot of carrots, or some CI based sticks, and I don't quite have the organisational influence to just drop that on people... ...yet.
  - abraxas 4y ago
    Well, not for ML we don't. Python killed it all off. If you plan to work on core algorithms or finetuning existing models there is nowhere to run.
noisy_boy 4y ago
conda is a bleeding pain in the ass; installed a ton of stuff without even asking if I needed them. Then when I wanted to install a different version of Python, kept running conflict resolution for hours before I got tired and killed it. Might as well just do the setup oldschool via virtualenv using requirements.txt. Dealing with all this is why I chose to use golang for writing a CLI utility (even though I'm not a big fan of its error handling boilerplate); static typing + produces a single binary that can be run without needing any of the environment setup. I am aware of various Python tools that can produce a binary too but I think they have their own edge cases and it is just nicer when that can be done out of the box without any dependencies. > You can also find deficiencies in the tools for the other languages mentioned. Some people think Maven is terrible because it uses XML and Gradle is the way to go, and others think Gradle’s use of a Groovy-based DSL makes things much harder than they need to be and prefer Maven instead. Yeah but I have never had the installer crap out like pip or its ilk do when installing a package with maven. At worst, it can't find the package if the repositories are not configured properly.
smitty1e 4y ago
Time for GvR to saddle up the BDFL pony for a (last?) ride?
synergy20 4y ago
So we should all use PDM then as it's the best so far among all others? I read it somewhere once in the past but never used it. I just use `venv + pip` all the time so far, worked well for my case though.
reducesuffering 4y ago
https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#how-is-node-better-than-python https://chriswarrick.com/blog/2023/01/15/how-to-improve-pyth... Is quite damning on how much of a better experience Node/JS/TS NPM is than Python's clusterfuck. Working with any other JS/TS open source is easy to get running. Even if you use Poetry in Python, you will run into everyone else using pip-tools, pipenv, pdm, conda, etc.
nezirus 4y ago
Reading this is really depressing. I just want Cargo for Python. Poetry it is for now, but it has quirks, and it is dog slow...
- uranusjr 4y ago
  A lot of people said they want Cargo for Python but immediately backed out when you require them to `cargo run` their program. They want `python myscript.py` to still magically work, but that’s exactly where a lot of the magic comes from. Running `python` directly is like manually invoking `rustc` (or maybe slightly more automated like a make script); it works, but is on an entirely different layer of abstraction and doesn’t fit well with the Cargo one. I’m not trying to imply you’re one of those people, but this may provide some insight why “Cargo for Python” is not a more widely adopted workflow.
  - nezirus 4y ago
    That is funny since poetry modus operandi is 'poetry run python ...' On topic, "Cargo for Python" should be official, singular (no competing standards), opinionated and fast (tm).
  - int_19h 4y ago
    Is there any reason why `python foo.py` couldn't do the equivalent of what `cargo run` does, if that's the expectation of most users?
    - uranusjr 4y ago
      Most of those workflow tools support at least two execution contexts, one for development and one for deployment (`cargo run` vs executing the compiled binary directly, `npm run` vs `node foo.js`, etc.). There are various reasons for the separation including hidden build steps or to set up the runtime environment, generally you want certain things automated during development but not when the program is deployed. A more accurate way to say this `python foo.py` can’t replace `cargo run` while still doing what it currently does; obviously it could be changed to run the program in the development context. The problem is it’s currently used in multiple contexts, and nobody really wants their workflow to break.
egberts1 4y ago
What is clearly still missing on python.org website is the obvious edit-code-run-debug step-by-step HOWTO for a typical tight but circular Python development iterations, complete with standardized packaging approach.
korijn 4y ago
I get the impression the author has not been exposed to enough pain doing package maintenance on a webpack codebase.
tiziano88 4y ago
Interesting article though it's missing at least a mention of Nix, which IMO solves a lot of the pain points described.
TigerTeamX 4y ago
I must admit I just gave up on all these tools. Instead I just do pip install -r requirements.txt -t .lib and I have import sys import os sys.path.append(f"{os.getcwd()}/.lib") in the top of my script. Some will tell me this is silly, but it just works. Rememer to at .lib to your .gitignore. Else you'll have lot of fun.
- TigerTeamX 4y ago
  This is similar to how node is doing. I just basically copied that idea.
- thinker5555 4y ago
  I tinkered with this a little, and it's interesting. But what about using pip list or pip freeze? They don't show the packages in the .lib directory. Do you just add them by hand to the requirements.txt?
  - TigerTeamX 4y ago
    I add them by hand to requirements.txt and I try to keep it as minimum as possible. Yes it makes it not 100% reproducible, but it forces me to think about each dependency and I can see in my git log why it was added.
    - thinker5555 4y ago
      It's funny how "pip help list" or "pip help freeze" works. :-) I found I can add --path .lib to list/freeze those. It's an extra step, but not too terrible.
- theacodes 4y ago
  As the person who implemented the -t flag and knows the horrendously sharp edge cases that it fails on, all I can say is bless your heart.
  - TigerTeamX 4y ago
kelnage 4y ago
So, I have recently returned to Python development after several years out. In my first project [1], I was building on an existing library that was already using Poetry - so obviously I went with that for my work. Although it was a bit of a learning curve, I quickly got accustomed to it, but still wondered why it had come about given my recollections of the other tools (virtualenv et al.) being “good enough”. Then, more recently, I had to run a different project that lacked any documentation as to how I was to run it, had a setup.py file, a Pipfile, and more in it. In trying to get this to run, I managed to make a real pigs ear such that (no doubt thanks to my lack of experience with those tools) I eventually had to delete all my virtual environments, as none of them worked anymore… So yes, I am 100% in the “one tool to rule them all” camp these days - and although PDM does look promising, right now it isn’t offering me anything above Poetry that I care that strongly about. As Cato might say, PyPA delenda est. 1. http://github.com/grafana/pySigma-backend-loki http://github.com/grafana/pySigma-backend-loki
vonseel 4y ago
Today I was preparing a minimal project for something I needed to share with someone else. I wasn’t sure what tools they would already have installed, so I checked out Python’s official packaging docs and browsed around. One page defaulted to giving instructions by default using “hatch”, while another page says the official recommendation is setuptools with a setup.cfg and only dynamic things declared in setup.py. Meanwhile, pyproject.toml support is being pushed elsewhere and in beta support for a lot of setup tools features. There’s way too many tools and too much confusion around which ones to use and which ones are the best to choose going forward. Why can’t we just be like Rust and have one tool that builds and formats and runs tests and everything else? Pipenv is crap but it’s somehow gained support of the PyCQA, meanwhile I’m not sure if poetry is even mentioned by the Python docs or CQA but it’s the one I’ve been using and it seems great, but I don’t even know if I’ve made the right choice anymore by using that. All of this fragmentation just leads to developer confusion, newbies and people who have been using the language 10+ years alike.
bageljr 4y ago
I think it's crazy how they don't want to change cause it'd disrespect the work of maintainers of existing projects. Python packaging is screwed if maintaining all packaging software at the cost of functionality is the goal
- NullInvictus 4y ago
  At a previous company I had a chance to sit down and talk with python developers across a number of groups, and a fun question was asked: Imagine Python 4 is another large breaking change on the scale of 2 to 3. What feature makes it worth it to you? The most common and enthusiastic answer was: Packaging and distribution. I think to say that "large numbers of python developers think packaging is bad and should be fixed" undersells the current sentiment. I think there are large numbers of python developers with boots on the ground who are _utterly sick to death_ of the problem and willing to accept drastic measures resulting inconsiderable suffering if just means it means the issue goes away. People are in other words, getting rather desperate and discontent. Upstream needs to do a little soul searching and come to an understanding that whatever ideological rock they're chained to that is preventing some solution from emerging has a high chance of not being moored in the needs of reality.
  - bageljr 4y ago
    Well who is upstream? Pypa? Python steering council? Python itself won't do much without pypa approval. Pypa won't endorse an existing or new tool that solves most mainstream cases well.
whatever1 4y ago
Please allow me to vent on python packaging. I was working to debug a very weird error that was happening in my code only when my app was raising a warning through my `warnings.py`. My code was importing numpy, which in turn was trying to import its own `warnings.py`, but because of the python environment precedence, it was loading my own `warnings.py` which unfortunately had the same name. How on earth was this something that passed the design of the import / package system of python? Why there are not unique id’s for the imported modules? TLDR: An imported package took dependency on my own code because of module name clashing and lead to unexpected behavior.
- greazy 4y ago
  How does 'warnings.py' work with the rest of the code? Intrigued as I've put myself into a corner with the way warnings/errors are handled in my project. By the way I think there is a convention to handle these issues but it appears numpy isn't using it. If I remember correctly, any module with __ infront will be called {package}.__{module}.
actinium226 4y ago
Glad to see this getting discussion. I hate virtualenvs for several of the reasons mentioned in the article - extra state to keep track of, and potentially installing packages to the wrong location if I failed to keep track of it. I always just force my user packages to work. I usually only use numpy/scipy/matplotlib and occasionally a few others so its not that hard, but some sort of npm like experience would be welcome. I know that many are struggling with these environments.
- randomluck040 4y ago
  As someone who has next to no experience with npm, which benefits of npm would you like to see in venv or package management in Python?
  - actinium226 4y ago
    I think the article describes it pretty well, it actually highlighted a few features of npm I wasn't familiar with. But to speak to my own experience, it's convenient that npm installs all the packages you need into a project-local node_modules folder. This eliminates the need for venv. Mostly I just want to see venv eliminated. I really don't like the workflow it requires.
rpep 4y ago
My view is that it is not that hard (with any tool) to build a wheel. You can basically follow a guide, you have to supply the same information, it’s mostly just the format you do it in. The problem is that compiled extensions are not a minor use case for Python, they’re hugely widely used for packages even if users don’t use them that much. Building a package with no dependencies is still not that hard. But there is no good way to distribute the compiled packages you depend on as their own thing - Intel release MKL as a “Python” package for e.g. but it is just the C library [1]. But of course, if you’re a package maintainer, the chance that the package maintainer of your dependency has both uploaded it to a package manager in a language they don’t use and that also being the version you need is pretty slim. So the issue becomes “how do I bundle every single possible dependency I might need inside a Wheel”, because source installs are not something most Python users understand. Conda and Spack try and fix this issue by being general package managers that can distribute packages in any language, allowing you to depend on FFTW or Eigen or SUNDIALS or whatever you need in your C extensions, and in all of the discussion I don’t think any proposal really tackles this gap. [1] https://pypi.org/project/mkl/ https://pypi.org/project/mkl/
pharmakom 4y ago
Glad to see dotnet CLI get a shout out. That tool really makes cross compilation a breeze. In Python I usually give up and resort to Docker.
NelsonMinar 4y ago
Note this post is 8700 words long and it doesn't really delve into how bad it is to build packages on Python; mostly it's about how bad it is to use them. It's an excellent writeup and the length is necessary. My guess is a post about creating packages would need to be 2x as long for the same level of detail.
- daneah 4y ago
  I don't agree with all the points from the article, but I do agree there is a depth of learning to be had about creating packages (and doing it repeatably/scalably). I wrote a book about creating Python packages that just came out: https://pypackages.com https://pypackages.com Even this book doesn't cover all options in each area, and it skips almost wholly over conda because I have no personal experience using it. conda and the work in the scientific community adds complexity both to the creation and the consumption side of packaging, and that's one area I'm not sure this post covers all the nuance of when considering how a "one size fits all" solution might work in practice.
penciltwirler 4y ago
Just do what Go did.
BerislavLopac 4y ago
Despite being misguided in a few points, the best bit about this article is the detailed table comparing the functionalities of different Python packaging tools. One missing detail is the ability of PDM to support different build backends, which allows for some interesting capabilities: for example, using hatchling as the backend it is possible to utilise hatch's support for dynamic versioning, whcih does not exist in PDM proper. I haven't tried, but wouldn't be surprised if that could allow PDM to support C-extensions by using setuptools as the backend...
- BerislavLopac 4y ago
  One of the misguided points is this: > It is also notable that PEP 20, the Zen of Python, states this: > There should be one-- and preferably only one --obvious way to do it. > Python packaging definitely does not follow it. There are 14 ways, and none of them is obvious or the only good one. All in all, this is an unsalvageable mess. Why can’t Python pick one tool? So a few comments on that: 1. There is a reason PEP 20 is called "Zen of Python" and not "Zen of Python Packaging". 1a. Even if it applied to the ecosystem and not just the language itself - PEP 20 is a guide, not a set of divine laws. 2. There is one obvious way, right there in the Python documentation. [0] Yes, it lists several different tools, but "a way to do it" and "a tool to do it with" are two very different topics. 3. "Why can’t Python pick one tool?" I never understood this fixation on silver bullets and "one tool to rule them all"... As long as common principles are well defined - which has been true for Python for quite some time even before PEP 517, with things like PyPI and pip - what is the harm in having multiple competing solutions? [0] https://packaging.python.org/en/latest/tutorials/packaging-projects/ https://packaging.python.org/en/latest/tutorials/packaging-p...
  - vonseel 4y ago
    > Why can’t Python pick one tool? Sometimes I wonder if developers are just making new tools out of self-interest and they want their name on some well-known project, and that's why they don't just work with one of the existing projects to implement their ideas and move the overall community towards a common working model. > what is the harm in having multiple competing solutions? Developer confusion and community fragmentation.
damnever 4y ago
I am the author of pigar[1], and I am using Go a lot, Go has its problems too, but I am a fan of `import "url"` style import statement, developers can write code first, and sync the dependency later with `go mod tidy`. To fix problems in Python's world, Python's community should simplify the tools and cultivate a habit to declare the dependency first(maybe this should be mandatory). [1]: https://github.com/damnever/pigar https://github.com/damnever/pigar

292 points by Kwpolska 4y ago

Not being able to exclude system packages from resolution when using __pypackages__ is a pretty glaring omission that will force people to continue using venvs until it’s fixed.

deleted 4y ago

[deleted]

matsemann 4y ago

Because of the insanity of python I run everything in Docker (through compose). No more issues with it not working on some dev's computer because of a missing wheel, or that they need to have X and Y c++ toolchain packages installed locally etc. No more trying to fix a broken setup after upgrading python or poetry versions, just "docker compose build" and you're up and running. No spending days getting a freshly cloned project up and running. Then I point Pycharm to use that as my remote interpreter through docker compose. Not watertight, but a hundred times better dev experience.

DangitBobby 4y ago
I've tried this a couple of times but just generally not liked the experience and gone back. When you get a shell into the container so you can use your project's python env, the tools you have installed on your machine (bashrc, rg, aliases) aren't available. I ultimately prefer the hassle of managing environments natively over forfeiting the use of my tools or juggling shells.
LtWorf 4y ago
.deb packages…
- groestl 4y ago
  Do you happen to have a link to a python application that ships it's own dependencies and has state-of-the art debian packaging?
  - vbernat 4y ago
    It depends what you mean by state-of-the-art, but it's possible with dh-virtualenv: https://github.com/vincentbernat/pragmatic-debian-packages/tree/master/puppetboard/debian https://github.com/vincentbernat/pragmatic-debian-packages/t... (not something that can become an official package as it goes against Debian policies)
  - LtWorf 4y ago
    > that ships it's own dependencies You mean a link to a malformed .deb packages? It's not difficult to do but it's also the wrong way to do it.
    - groestl 4y ago
      Honestly, there is a lot more I'd like to do with my life than caring about some debian packaging practices when all I want to do is ship a company's internal application with it's exact dependency versions. But it needs to work, and, e.g. "The virtualenv relocate-able flag has been always experimental, and never really worked;" So in practice we're back to shipping whole distributions using Docker (or, like we did, abandon Python as an option for development but keep .deb packages for deployment).
- u801e 4y ago
  On that note, why are we trying to use language specific packaging tools to build packages rather than just building OS specific packages where dependencies are handled by apt for deb packages or dnf for rpm packages? These are language agnostic and can get the job done.
  - groestl 4y ago
    Because I have dozens of colleagues, each with their own os/distribution, developing an application that needs to lock down dot releases of dependencies to the versions running in production and update them at the same time. There is no way to do that using distribution specific tools without going insane.
    - u801e 4y ago
      >> why are we trying to use language specific packaging tools to build packages rather than just building OS specific packages where dependencies are handled by apt for deb packages or dnf for rpm packages? >> >> These are language agnostic and can get the job done. > Because I have dozens of colleagues, each with their own os/distribution, developing an application that needs to lock down dot releases of dependencies to the versions running in production I assume that the production environment isn't running dozens of os versions and distributions. For development, using a VM or container running the same os version and distro that's used in production and using that OS's package format for packaging the software and installing it in the dev environment (on the container or VM) would work. You're testing to see if the software works in the production environment, not someone's preferred os/distribution. > and update them at the same time I'm not as familiar with apt, but dnf has a version lock feature that would allow you to lock down the dependencies to specific versions. You could update them and test them during development and update the version lock file to pull down the updated version of the dependency when updating production.
      - groestl 4y ago
        For every dependency that's missing from the upstream distribution in the exact version we need, we'd need to package that appropriately. We have nothing to gain here, nobody pays for that.
goodpoint 4y ago
No way. Docker is not designed for security.
- matsemann 4y ago
  Well, it's probably more secure than doing `pip install` and letting every dependency run whatever to install itself on your host...
  - goodpoint 4y ago
    Actually not at all assuming you are using a VM for a production workload. Additionally, even if you only rely on a container - still no: other container engines have a lower attack surface.
    - matsemann 4y ago
      I'm talking about local dev here.
wheelerof4te 4y ago
When the solution to basic problem like this is "use Docker", you realise how deeply flawed modern software development (in Python, at least) is.
- jessekv 4y ago
  Not sure I agree with "deeply flawed" verdict. The flaws of python's packaging system (as well as the GIL and other hangups) emerge from the tradeoffs of inventing a language optimised for reusing and connecting as many disparate system binaries as possible. Its not surprising that such a server scripting language is tightly coupled with the server environment that it runs in. Docker is just one way to ensure a reproducible server environment across machines. What do you think it could have done differently?
  - lmm 4y ago
    What could have been done differently: - Make dependency resolution deterministic by default. Unfortunately the whole ecosystem has to buy into it, but the benefits are huge. - Stop building castles of sand by making more layers of tooling that have to run in Python (and will therefore run in some random poorly managed Python). The language install can't manage the build system install - if anything the build system install should manage the language install. Adding pip to the language distribution was such a backwards decision that it marks the point where I gave up on Python ever fixing their stuff. - Ignore Linux system package managers (apt etc.), they have a fundamentally broken model and will infect your language ecosystem with that breakage if you try to cater to them.
- rtpg 4y ago
  to be fair to Python, almost every ecosystem is in "use Docker" land, even Go binaries will offer docker images!
groestl 4y ago
I secretly believe Docker only exists because of how f'up Python's distribution story is.
- jessekv 4y ago
  I also have often wondered how much Docker owes its success to the shortcomings of python.

optimalsolver 4y ago

Python packaging is a solved problem: https://python-poetry.org/ https://python-poetry.org/

throwawaaarrgh 4y ago
as a Poetry user myself, ummmmmmm. no. lol
revskill 4y ago
Yes, there should be (one way) to do packaging.
Kwpolska 4y ago
Poetry is one of the better options, but its nonstandard pyproject.toml is not ideal. PDM is basically Poetry, with the metadata standard in place, and with support for __pypackages__.
- frafra 4y ago
  I would like to add that PDM also supports having multiple dependencies groups, and the developer is superfast at fixing issues.
c7DJTLrn 4y ago
It's not really solved if you have no choice but to work with codebases that don't use Poetry (quite common). There's 14 tools for people to choose from that aren't going away any time soon.
wlpu 4y ago
Poetry is one of the worst modern solutions to python packaging, either pick hatch or PDM, they both have more and better implemented features plus their codebases aren't a complete mess

deleted 4y ago

[deleted]

woodruffw 4y ago

It's not very well documented, but the PyPA tools do provide a unified experience when used correctly. Here's a PyPA project (FD: one I work on) that uses a single pyproject.toml to handle all aspects of packaging (and most non-packaging tool configuration, to boot)[1]. With a single file like that, the only thing you need to do to start a local development environment is: python -m venv env && . env/bin/activate python -m pip install .[dev] (We provide a `make dev` target that does that for you as well.) Similarly, to build distributions, all you need is `build`: python -m build [1]: https://github.com/pypa/pip-audit https://github.com/pypa/pip-audit

qbasic_forever 4y ago
Nowhere in pypa documentation is your simple workflow described or mentioned. Instead it's a jumble of links to a myriad of tools including hatchling, flit, pdm, etc. and basically just a shoulder shrug, 'I don't know, figure it all out yourself' message. This article makes a great point that the current pypa 'guidance' is too confusing and vague for actual end users (i.e. people that don't work directly on pypa or haven't been studying python packaging for decades).
- woodruffw 4y ago
  I agree that it's confusing. That being said, there is an official PyPA tutorial that goes through the exact steps need to produce the commands I suggested, so I don't think it's accurate to say that it's nowhere to be found[1]. Edit: Unfortunately it's easy to confuse the above tutorial with this one[2], which is specifically for setuptools. So I can appreciate end user confusion around the documentation, particularly in that respect! [1]: https://packaging.python.org/en/latest/tutorials/packaging-projects/ https://packaging.python.org/en/latest/tutorials/packaging-p... [2]: https://packaging.python.org/en/latest/guides/distributing-packages-using-setuptools/ https://packaging.python.org/en/latest/guides/distributing-p...
  - qbasic_forever 4y ago
    No I'm looking at link 1 and specifically have an issue with the huge table of options for how to setup your pyproject.toml--it has tabs for hatchling, flit, pdm, etc. but _zero_ description of why I would want to use one of those tools. It's just more confusing, like you're being thrown a bag of parts and no description of what to build or how to do it. To be honest that entire pypa doc should be like two paragraphs long instead of 1000+ words. It should be basically, "ok to package your python app just run <official python tool to init a package>, that's it you're done!". Every decision should be made for me, like it is with npm, cargo, etc. I shouldn't have to think beyond running one command. That's what python end users need. We don't need a million tools and huge docs with seemingly no vision or goal. We need to get shit done and the less time we faff about with packaging and tooling the better.

tempest_ 4y ago

I've pretty much settled on poetry for most things at this point. It still has a ton of rough edges like error messages still closer to a stack trace than something actionable and it's slow as hell resolving but it does the job 90 percent of the time without issue. Also running everything in docker now anyway which side steps some other problems.

imglorp 4y ago
That's where we are as well, but yes the problems you didn't sidestep with poetry+docker are still there: the pypi fecosystem out there does not stop quantum fluctuations just because you want it to. If you pin too much you're not getting security fixes and if you don't pin enough, you get bitten every time a tertiary dep changes that you never asked for. Oh yeah and there are now trojans upstream almost daily.
- tempest_ 4y ago
  Yeah but that trade off is entirely separate. We trade security for speed(or "velocity" if you want to be jargon about it). I just pin everything and go through my projects every couple weeks and bump the deps (unless some really big CVE hits the news).
- gen220 4y ago
  Does any FOSS package ecosystem have a good solution to this particular set of problems? I know Go has historically done something interesting with selecting the "minimum viable version number" for each dependency, but that's the only relevant idea that comes to mind. With PyPi, at least it's relatively trivial to self-host a pypi instance and self-manage dependency updates. In the Python ecosystem (vs e.g. Node), at least the total set of first+second-order dependencies for most normal projects is quite small (e.g. medium double digits to low triple digits). It doesn't feel too painful to manage that magnitude of upgrades on a monthly or quarterly basis.
  - lmm 4y ago
    In the JVM ecosystem you get the version your immediate dependency was built/released against, transitive dependencies are never silently upgraded (though if you want to upgrade then it's one command). IME that's a much better default; it does mean that when a security issue is found in a low-level library you get everyone doing a bunch of releases that just bump their dependencies, but the consistent behaviour is very much worth it.

dataflow 4y ago

Regarding system package upgrades breaking virtual environments: would hard links address this? If the system package manager removes a file, your venv still has a hard link to it so it doesn't see a difference.

Denvercoder9 4y ago
It would also prevent the Python in your venv from receiving any updates when the system-wide Python is installed. At that point it's probably easier to just use your own Python installation, e.g. with pyenv.
- dataflow 4y ago
  Sure but installing a new copy would defeat the space savings. The goal of this was to get the desired effect without wasting as much space. I wasn't trying to optimize purely for ease of use. Put another way, my suggestion was to migrate from symbolic links to hard links.

arunkant 4y ago

One more thing we need is app distribution for desktop apps written in python. I need to be able to package python app as single binary. There are tools like pyoxidizer https://pyoxidizer.readthedocs.io/en/stable/ https://pyoxidizer.readthedocs.io/en/stable/ . hopefully it become standard in python community

ctoth 4y ago
You probably already know about this, but Nuitka[0] is pretty great for building distributable Python apps. [0]: https://nuitka.net https://nuitka.net

rektide 4y ago

Cant resist digging at Node.js even when writing up how infinitely better Node.js is at dealing with packages than python, haha: > Let’s try removing is-odd to demonstrate how badly designed this package is: You literally just deleted is-even 's dependency on is-odd then have the audacity to be shocked that it broke? There's a lot of hatred for the small package philosophy of node.js, but it's also a huge win, stands a good chance of being why javascript has been such a winner, gotten so far: very explicit small things that say what they do on the tin. Rather than repeat yourself by making a copy pasted is-even and then maintaining both, it makes perfect sense to compose functionality, to build off what we have. And it is easier to understand the scope of what a package is & what it can do when it is explicitly limited in scope. This is another place where there is lots of loud vociferous animosity against what is, but it's there for good reason. And with rare rare exception- left-pad rage-quit deliberate breakage, for example, it serves well. With the exception that yes, inventorying your stuff is hard.

zdragnar 4y ago
It would be nice if major node packages invested time in re-inventing the wheel, just a little bit. Back when I used it, I really appreciated how little bloat Hapi added to node modules, compared with webpack for example. Obviously there's a world of difference in what problems the two solve, but still...
kdazzle 4y ago
I’m biased, but I do like Python’s package management much better than Node. Even just regular old virtualenvs. Can’t tell you how many times deleting the node packages dir and reinstalling fixes a weird issue, but that’s happened very rarely for me on the python side. Also, having to comb through a bunch of node packages of dubious quality to find some sort of standard approach happens way too often. Like take python requests vs axios.
- revskill 4y ago
  I've tried with venv before, and the console (Iterm) just get broken.
- 4y ago

samwillis 4y ago

This post is prompted by the survey of Python users, their feedback, and a current thread [0] on the Python forms discussing a way forward. I read the thread the other day, it's long, and there are a lot of opinions. Towards the end of this post there is an interesting observation: > Discourse, the platform that the discussion was held on, shows the number of times a link was clicked. Granted, this count might not be always accurate, but if we assume it is, the link to the results summary was clicked only 14 times (as of 2023-01-14 21:20 UTC). The discussion has 28 participants and 2.2k views. If we believe the link click counter, half of the discussion participants did not even bother reading what the people think. Along with that, my concern is that there are too my cooks in the kitchen. A packaging system is never going to cater for more than 90% of use cases, but there will likely be a higher than average representation from people who would sit in the 10%. There is a danger of them never being able to agree on a solution that solves all problems, and then never launch anything. It would be far better to get something out, have a stake in the ground, and build from there. It also desperately needs to be "Core Python", my understanding is that the PyPA (Python Package Authority) is somewhat disconnected from the core Python development. Ultimately they need one system that is front and centre, that is the "officially endorsed" tool, and very clear messaging on that. 0: https://discuss.python.org/t/python-packaging-strategy-discussion-part-1/22420 https://discuss.python.org/t/python-packaging-strategy-discu...

forkerenok 4y ago
Thanks, this is an interesting nugget. Did you intend to append a link reference for [0]?
- samwillis 4y ago
  Yes, edited. Thanks.
mkoubaa 4y ago
I always like to say that's barring divine inspiration, it's impossible to make the right choice. It's far more possible to make a choice and then work hard to make it right
4y ago

jasonhansel 4y ago

Over time, I've grown to appreciate "pip-tools." Since it's a dead-simple extension of pip, I wish it could be upstreamed into pip itself; that seems like the most straightforward way of fixing a number of Python's packaging issues.

pdw 4y ago
Yes. It usually gets glossed over, but if you're building a Python application it's all you need.
delijati 4y ago
I work on mutlirepos and i really dislike gits subtree, subrepo i use https://pypi.org/project/zc.buildout/ https://pypi.org/project/zc.buildout/. Yes i know i can do [1]. But editing on multirepos at the same time i can only do with zc.buildout [2] Still not perfect but it does the job. [1] $ cat req.txt requests git+ssh://git@gitlab.com/foo/bar/amber.git@0.4.0 git+ssh://git@gitlab.com/foo/bar/rebam.git@0.4.0 [2] $ cat buildout.cfg [buildout] extends = versions.cfg extensions = mr.developer auto-checkout = \* develop = . show-picked-versions = true update-versions-file = versions.cfg sources-dir = git-sources parts = py eggs = amber rebam hammer [sources] amber = git git@gitlab.com/foo/bar/amber.git@0.4.0 rebma = git git@gitlab.com/foo/bar/rebma.git@0.4.0 [py] recipe = zc.recipe.egg eggs = ${buildout:eggs} interpreter = py-backend dependent-scripts = true
- jamesfinlayson 4y ago
  Ah, I haven't used buildout in years (I remember using it a lot when working with Plone). I used it for a personal project but gave up a few years ago as some piece of the puzzle seemed broken and abandoned (something hadn't been updated to use a newer version of TLS or something). I liked buildout though - it was a good system with its bin directory.
vlovich123 4y ago
Yup. I like to make it so that each executable has a symlink to a shell script. The shell script checks if there’s a virtual environment set up with the packages in the requirements.txt installed (it takes a snapshot of the file because virtualenv doesn’t have a DB to query cheaply). Once the environment is set up, it dispatches to running from the virtualenv. That way when you update a requirements.in file it recompiles it (if the txt is out of date) and installs any new packages, removes packages that shouldn’t be there anymore and updates ones whose version changed (if there’s any changes found). It also lets you trivially run tools with disparate requirements.in without conflicts because each is siloed behind its own virtualenv. This makes it a trivial experience to use these tools in a shared repo because there’s no worrying about packages / needing to remember to run some command before the right environment is set up. You just modify your code and run it like a regular command-line tool and packages automatically get deployed. It’s also amenable to offline snapshotting / distribution. In fact, I used this to distribute support tooling for factory lines of the Pixel Buds and it worked extremely well.

Lyngbakr 4y ago

It seems most people agree that Python's packaging isn't great, but, conversely, is there a language where most people agree that the approach to packaging is awesome? I mean, what's the gold standard to aspire to?

nixpulvis 4y ago
Ruby gems and Cargo are pretty awesome. Also, I find I like Arch linux's Pacman quite a bit, though that's a slightly different use case where versioning isn't resolved.
- oconnor663 4y ago
  Cargo is a gold standard imho, but there are definitely some simplifying decisions it makes that might not fit Python: - Dependencies and executables can run arbitrary code during build. For example, Cargo knows almost nothing about how to build C code, and the common workflow is to pull in the popular `cc` library for this and call it in your build.rs. - There's mostly no such thing as "installing a library", and each project builds its dependencies from source. This is baked in at the Cargo level (the install command just doesn't work for library crates) and also at the language level (no stable ABI besides "extern C"). - Related to that, the final product of a Cargo workflow is a mostly-statically-linked binary, so there isn't really an equivalent to virtualenv.
  - Kwpolska 4y ago
    > - Dependencies and executables can run arbitrary code during build. For example, Cargo knows almost nothing about how to build C code, and the common workflow is to pull in the popular `cc` library for this and call it in your build.rs. Python packaging involves arbitrary code execution, even today. While it’s less obvious than it was with `setup.py`, if you’re installing from source, the package can specify any build backend it wants, and that build backend can do anything. This could be mitigated by having an allowlist of build backends, but who would be responsible for vetting the tools, and how would it be guaranteed that an allowed tool is not taken over by someone hostile, and that an allowed tool does not have a system to run arbitrary package-provided code?

josteink 4y ago

Every time someone asks me what I think is wrong with Python packaging, I’ll show them this link. Saved. Thanks for sharing!

DangitBobby 4y ago

I have a single, simple script (not a package!) that has dependencies. Actually, I have a few of these, just sitting in /usr/local/bin so I can execute them whenever. How should I be managing environments for these scripts? Do I install dependencies in shared system python? Should I create a shared venv? Where should I store it? Any tools out there that make this decision for you and manage it? Just the fact that homebrew occasionally updates my installed Pythons and breaks everything makes me reluctant to use Python as a scripting language at all. It's pretty reliable when I have a folder dedicated to a project (poetry is good for this) but I'm pretty fed up with how brittle it all is.

DangitBobby 4y ago
In an amazing coincidence, I just found another front-page post exploring this same topic! For Python, it seems to suggest using nix-shell. 1. https://dbohdan.com/scripts-with-dependencies https://dbohdan.com/scripts-with-dependencies
- DangitBobby 4y ago
  My comment with an example of a single file script which can specify its own dependencies: https://news.ycombinator.com/item?id=34393630 https://news.ycombinator.com/item?id=34393630
wpietri 4y ago
I don't understand how there still isn't a good answer for this. It seems like such an obvious need, and for years I've heard go being promoted for solving this problem. I get why Python didn't start out with an answer for this, but in an era of 20 TB hard drives, I'm more than willing to burn space for ease of install.
fleekonpoint 4y ago
I use a separate directory and venv for each script. To execute the script, I use a shell script to call the venv's python interpreter. This is also how I use python scripts with cron/systemd. #!/bin/bash # myscript.sh venv/bin/python3 myscript.py You could also skip the shell script and use aliases in your .bashrc.

benreesman 4y ago

Trying to build Python under Nix (i.e. solid, not “close enough) is an education in how fucked up the Python packaging ecosystem is. Which is a shame because Python is an amazing scripting language, the de facto numerical computing standard, and probably a better bash for most bash use cases.

uranusjr 4y ago
One of the problems with Python packaging is Python Core Devs view “Python packaging” and “packaging Python” as two different things, but most people don’t bother with (or can’t tell) the difference :)

mkoubaa 4y ago

I use sites as composable mini venvs, but there's no tooling for that

charliermarsh 4y ago

It's interesting to note that PEP 582 (i.e. __pypackages__) mode used to be the PDM default -- and was sort of its flagship feature -- before being made opt-in with 2.0 release, the explanation being that the PEP had stalled, and that editor and IDE support for virtualenvs is much better (https://www.pythonbynight.com/blog/using-pdm-for-your-next-python https://www.pythonbynight.com/blog/using-pdm-for-your-next-p...). If you read through the discussion on the Python forum, one of the counterarguments to the PEP is that "If existing tools can already support this, why do we need a PEP for it?" But presumedly a PEP would help push the ecosystem to accommodate __pypackages__, and to solve the aforementioned problems (like broader editor and IDE support). For what it's worth (as someone that builds Python tooling full-time): I'm generally a fan of moving to a node_modules-like model.

charliermarsh 4y ago
Oh, also, I think I've read the entire Discussions thread on this PEP, and from what I can tell, the PEP is missing champions who are eager to push it forward. Part of the problem, I'm guessing, is that the PEP kind of sits in a weird space, because it's not up to the PyPA to "approve" it, or whatever -- PEPs are approved by the Python Steering Council. Per Brett Cannon (Steering Council Member): > Either consensus has to be reached (which it hasn’t) and/or someone needs to send this to the SC to make a decision (although it is a weird PEP in that it also impacts packaging, so it isn’t even clear who would get final say). https://discuss.python.org/t/pep-582-python-local-packages-directory/963/239 https://discuss.python.org/t/pep-582-python-local-packages-d...
Kwpolska 4y ago
That tooling argument is quite weak. PDM is the only tool I was able to find that has support for __pypackages__, the paths it uses seem to be slightly different to the PEP wording, and it uses a $PYTHONPATH/shell hack (`pdm --pep582` adds a folder with `sitecustomize.py` that does the magic). How do you change the $PYTHONPATH used by an IDE (and its magic features)?
- charliermarsh 4y ago
  I agree!

baggiponte 4y ago

been using pdm since at least one year and have no regrets: it's the go tool and we should flock to it!

charliermarsh 4y ago

> There are two tools for dealing with packages in the Node world, namely npm and Yarn. Not to mention pnpm. Or Bun. Or even Deno. All of which deal with packages, and make different choices. This isn't to be a pedant, but I think it's a reasonable example of why it's less important that there's a single way to do a thing, or even a single tool vs. multiple alternatives; instead, what I care about is that I don't have to string together tens of tools to do that thing, and that there's some degree of compatibility and standardization in the API between those different alternatives.

abraxas 4y ago
Well, no. The JS package ecosystem is a slightly lesser mess than the Python ecosystem but only slightly so. I think it's a bad example to base any redesign off of. At least the author brought up .NET which mostly has its shit together when it comes to builds and packaging.

hkgjjgjfjfjfjf 4y ago

[dead]

sam345 4y ago

The main reason I have stayed away from Python. packaging is a mess!

c7DJTLrn 4y ago
And one of the reasons I moved away from it! My blog post on the subject was posted here and it was very unpopular despite having similar points and also making comparisons with Node. There's nothing special about Python that makes it worth enduring this pain. We have other languages with a better developer UX.
- EdwardDiego 4y ago
  I'd be keen to read your blog post if you have a link?
  - c7DJTLrn 4y ago
    https://cedwards.xyz/breaking-up-with-python/ https://cedwards.xyz/breaking-up-with-python/ Bear in mind this was not written for an audience, let alone a HN audience.
    - EdwardDiego 4y ago
      Thanks, appreciate the link, I'll be back to yell at you shortly :D ...and having read it, the only thing I'd yell is that "I AGREE WITH ALL YOUR PAIN POINTS". Especially around documentation and package management. I do like type annotations for improving the reading experience of Python codebases, even if you're not using a type checker, but trying to get people to use them consistently for new code in an org that has a large pre-existing codebase involves either a lot of carrots, or some CI based sticks, and I don't quite have the organisational influence to just drop that on people... ...yet.
- abraxas 4y ago
  Well, not for ML we don't. Python killed it all off. If you plan to work on core algorithms or finetuning existing models there is nowhere to run.

noisy_boy 4y ago

conda is a bleeding pain in the ass; installed a ton of stuff without even asking if I needed them. Then when I wanted to install a different version of Python, kept running conflict resolution for hours before I got tired and killed it. Might as well just do the setup oldschool via virtualenv using requirements.txt. Dealing with all this is why I chose to use golang for writing a CLI utility (even though I'm not a big fan of its error handling boilerplate); static typing + produces a single binary that can be run without needing any of the environment setup. I am aware of various Python tools that can produce a binary too but I think they have their own edge cases and it is just nicer when that can be done out of the box without any dependencies. > You can also find deficiencies in the tools for the other languages mentioned. Some people think Maven is terrible because it uses XML and Gradle is the way to go, and others think Gradle’s use of a Groovy-based DSL makes things much harder than they need to be and prefer Maven instead. Yeah but I have never had the installer crap out like pip or its ilk do when installing a package with maven. At worst, it can't find the package if the repositories are not configured properly.

smitty1e 4y ago

Time for GvR to saddle up the BDFL pony for a (last?) ride?

synergy20 4y ago

So we should all use PDM then as it's the best so far among all others? I read it somewhere once in the past but never used it. I just use `venv + pip` all the time so far, worked well for my case though.

reducesuffering 4y ago

https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#how-is-node-better-than-python https://chriswarrick.com/blog/2023/01/15/how-to-improve-pyth... Is quite damning on how much of a better experience Node/JS/TS NPM is than Python's clusterfuck. Working with any other JS/TS open source is easy to get running. Even if you use Poetry in Python, you will run into everyone else using pip-tools, pipenv, pdm, conda, etc.

nezirus 4y ago

Reading this is really depressing. I just want Cargo for Python. Poetry it is for now, but it has quirks, and it is dog slow...

uranusjr 4y ago
A lot of people said they want Cargo for Python but immediately backed out when you require them to `cargo run` their program. They want `python myscript.py` to still magically work, but that’s exactly where a lot of the magic comes from. Running `python` directly is like manually invoking `rustc` (or maybe slightly more automated like a make script); it works, but is on an entirely different layer of abstraction and doesn’t fit well with the Cargo one. I’m not trying to imply you’re one of those people, but this may provide some insight why “Cargo for Python” is not a more widely adopted workflow.
- nezirus 4y ago
  That is funny since poetry modus operandi is 'poetry run python ...' On topic, "Cargo for Python" should be official, singular (no competing standards), opinionated and fast (tm).
- int_19h 4y ago
  Is there any reason why `python foo.py` couldn't do the equivalent of what `cargo run` does, if that's the expectation of most users?
  - uranusjr 4y ago
    Most of those workflow tools support at least two execution contexts, one for development and one for deployment (`cargo run` vs executing the compiled binary directly, `npm run` vs `node foo.js`, etc.). There are various reasons for the separation including hidden build steps or to set up the runtime environment, generally you want certain things automated during development but not when the program is deployed. A more accurate way to say this `python foo.py` can’t replace `cargo run` while still doing what it currently does; obviously it could be changed to run the program in the development context. The problem is it’s currently used in multiple contexts, and nobody really wants their workflow to break.

egberts1 4y ago

What is clearly still missing on python.org website is the obvious edit-code-run-debug step-by-step HOWTO for a typical tight but circular Python development iterations, complete with standardized packaging approach.

korijn 4y ago

I get the impression the author has not been exposed to enough pain doing package maintenance on a webpack codebase.

tiziano88 4y ago

Interesting article though it's missing at least a mention of Nix, which IMO solves a lot of the pain points described.

TigerTeamX 4y ago

I must admit I just gave up on all these tools. Instead I just do pip install -r requirements.txt -t .lib and I have import sys import os sys.path.append(f"{os.getcwd()}/.lib") in the top of my script. Some will tell me this is silly, but it just works. Rememer to at .lib to your .gitignore. Else you'll have lot of fun.

TigerTeamX 4y ago
This is similar to how node is doing. I just basically copied that idea.
thinker5555 4y ago
I tinkered with this a little, and it's interesting. But what about using pip list or pip freeze? They don't show the packages in the .lib directory. Do you just add them by hand to the requirements.txt?
- TigerTeamX 4y ago
  I add them by hand to requirements.txt and I try to keep it as minimum as possible. Yes it makes it not 100% reproducible, but it forces me to think about each dependency and I can see in my git log why it was added.
  - thinker5555 4y ago
    It's funny how "pip help list" or "pip help freeze" works. :-) I found I can add --path .lib to list/freeze those. It's an extra step, but not too terrible.
theacodes 4y ago
As the person who implemented the -t flag and knows the horrendously sharp edge cases that it fails on, all I can say is bless your heart.
- TigerTeamX 4y ago

kelnage 4y ago

So, I have recently returned to Python development after several years out. In my first project [1], I was building on an existing library that was already using Poetry - so obviously I went with that for my work. Although it was a bit of a learning curve, I quickly got accustomed to it, but still wondered why it had come about given my recollections of the other tools (virtualenv et al.) being “good enough”. Then, more recently, I had to run a different project that lacked any documentation as to how I was to run it, had a setup.py file, a Pipfile, and more in it. In trying to get this to run, I managed to make a real pigs ear such that (no doubt thanks to my lack of experience with those tools) I eventually had to delete all my virtual environments, as none of them worked anymore… So yes, I am 100% in the “one tool to rule them all” camp these days - and although PDM does look promising, right now it isn’t offering me anything above Poetry that I care that strongly about. As Cato might say, PyPA delenda est. 1. http://github.com/grafana/pySigma-backend-loki http://github.com/grafana/pySigma-backend-loki

vonseel 4y ago

Today I was preparing a minimal project for something I needed to share with someone else. I wasn’t sure what tools they would already have installed, so I checked out Python’s official packaging docs and browsed around. One page defaulted to giving instructions by default using “hatch”, while another page says the official recommendation is setuptools with a setup.cfg and only dynamic things declared in setup.py. Meanwhile, pyproject.toml support is being pushed elsewhere and in beta support for a lot of setup tools features. There’s way too many tools and too much confusion around which ones to use and which ones are the best to choose going forward. Why can’t we just be like Rust and have one tool that builds and formats and runs tests and everything else? Pipenv is crap but it’s somehow gained support of the PyCQA, meanwhile I’m not sure if poetry is even mentioned by the Python docs or CQA but it’s the one I’ve been using and it seems great, but I don’t even know if I’ve made the right choice anymore by using that. All of this fragmentation just leads to developer confusion, newbies and people who have been using the language 10+ years alike.

bageljr 4y ago

I think it's crazy how they don't want to change cause it'd disrespect the work of maintainers of existing projects. Python packaging is screwed if maintaining all packaging software at the cost of functionality is the goal

NullInvictus 4y ago
At a previous company I had a chance to sit down and talk with python developers across a number of groups, and a fun question was asked: Imagine Python 4 is another large breaking change on the scale of 2 to 3. What feature makes it worth it to you? The most common and enthusiastic answer was: Packaging and distribution. I think to say that "large numbers of python developers think packaging is bad and should be fixed" undersells the current sentiment. I think there are large numbers of python developers with boots on the ground who are _utterly sick to death_ of the problem and willing to accept drastic measures resulting inconsiderable suffering if just means it means the issue goes away. People are in other words, getting rather desperate and discontent. Upstream needs to do a little soul searching and come to an understanding that whatever ideological rock they're chained to that is preventing some solution from emerging has a high chance of not being moored in the needs of reality.
- bageljr 4y ago
  Well who is upstream? Pypa? Python steering council? Python itself won't do much without pypa approval. Pypa won't endorse an existing or new tool that solves most mainstream cases well.

whatever1 4y ago

Please allow me to vent on python packaging. I was working to debug a very weird error that was happening in my code only when my app was raising a warning through my `warnings.py`. My code was importing numpy, which in turn was trying to import its own `warnings.py`, but because of the python environment precedence, it was loading my own `warnings.py` which unfortunately had the same name. How on earth was this something that passed the design of the import / package system of python? Why there are not unique id’s for the imported modules? TLDR: An imported package took dependency on my own code because of module name clashing and lead to unexpected behavior.

greazy 4y ago
How does 'warnings.py' work with the rest of the code? Intrigued as I've put myself into a corner with the way warnings/errors are handled in my project. By the way I think there is a convention to handle these issues but it appears numpy isn't using it. If I remember correctly, any module with __ infront will be called {package}.__{module}.

actinium226 4y ago

Glad to see this getting discussion. I hate virtualenvs for several of the reasons mentioned in the article - extra state to keep track of, and potentially installing packages to the wrong location if I failed to keep track of it. I always just force my user packages to work. I usually only use numpy/scipy/matplotlib and occasionally a few others so its not that hard, but some sort of npm like experience would be welcome. I know that many are struggling with these environments.

randomluck040 4y ago
As someone who has next to no experience with npm, which benefits of npm would you like to see in venv or package management in Python?
- actinium226 4y ago
  I think the article describes it pretty well, it actually highlighted a few features of npm I wasn't familiar with. But to speak to my own experience, it's convenient that npm installs all the packages you need into a project-local node_modules folder. This eliminates the need for venv. Mostly I just want to see venv eliminated. I really don't like the workflow it requires.

rpep 4y ago

My view is that it is not that hard (with any tool) to build a wheel. You can basically follow a guide, you have to supply the same information, it’s mostly just the format you do it in. The problem is that compiled extensions are not a minor use case for Python, they’re hugely widely used for packages even if users don’t use them that much. Building a package with no dependencies is still not that hard. But there is no good way to distribute the compiled packages you depend on as their own thing - Intel release MKL as a “Python” package for e.g. but it is just the C library [1]. But of course, if you’re a package maintainer, the chance that the package maintainer of your dependency has both uploaded it to a package manager in a language they don’t use and that also being the version you need is pretty slim. So the issue becomes “how do I bundle every single possible dependency I might need inside a Wheel”, because source installs are not something most Python users understand. Conda and Spack try and fix this issue by being general package managers that can distribute packages in any language, allowing you to depend on FFTW or Eigen or SUNDIALS or whatever you need in your C extensions, and in all of the discussion I don’t think any proposal really tackles this gap. [1] https://pypi.org/project/mkl/ https://pypi.org/project/mkl/

pharmakom 4y ago

Glad to see dotnet CLI get a shout out. That tool really makes cross compilation a breeze. In Python I usually give up and resort to Docker.

NelsonMinar 4y ago

Note this post is 8700 words long and it doesn't really delve into how bad it is to build packages on Python; mostly it's about how bad it is to use them. It's an excellent writeup and the length is necessary. My guess is a post about creating packages would need to be 2x as long for the same level of detail.

daneah 4y ago
I don't agree with all the points from the article, but I do agree there is a depth of learning to be had about creating packages (and doing it repeatably/scalably). I wrote a book about creating Python packages that just came out: https://pypackages.com https://pypackages.com Even this book doesn't cover all options in each area, and it skips almost wholly over conda because I have no personal experience using it. conda and the work in the scientific community adds complexity both to the creation and the consumption side of packaging, and that's one area I'm not sure this post covers all the nuance of when considering how a "one size fits all" solution might work in practice.

penciltwirler 4y ago

Just do what Go did.

BerislavLopac 4y ago

Despite being misguided in a few points, the best bit about this article is the detailed table comparing the functionalities of different Python packaging tools. One missing detail is the ability of PDM to support different build backends, which allows for some interesting capabilities: for example, using hatchling as the backend it is possible to utilise hatch's support for dynamic versioning, whcih does not exist in PDM proper. I haven't tried, but wouldn't be surprised if that could allow PDM to support C-extensions by using setuptools as the backend...

BerislavLopac 4y ago
One of the misguided points is this: > It is also notable that PEP 20, the Zen of Python, states this: > There should be one-- and preferably only one --obvious way to do it. > Python packaging definitely does not follow it. There are 14 ways, and none of them is obvious or the only good one. All in all, this is an unsalvageable mess. Why can’t Python pick one tool? So a few comments on that: 1. There is a reason PEP 20 is called "Zen of Python" and not "Zen of Python Packaging". 1a. Even if it applied to the ecosystem and not just the language itself - PEP 20 is a guide, not a set of divine laws. 2. There is one obvious way, right there in the Python documentation. [0] Yes, it lists several different tools, but "a way to do it" and "a tool to do it with" are two very different topics. 3. "Why can’t Python pick one tool?" I never understood this fixation on silver bullets and "one tool to rule them all"... As long as common principles are well defined - which has been true for Python for quite some time even before PEP 517, with things like PyPI and pip - what is the harm in having multiple competing solutions? [0] https://packaging.python.org/en/latest/tutorials/packaging-projects/ https://packaging.python.org/en/latest/tutorials/packaging-p...
- vonseel 4y ago
  > Why can’t Python pick one tool? Sometimes I wonder if developers are just making new tools out of self-interest and they want their name on some well-known project, and that's why they don't just work with one of the existing projects to implement their ideas and move the overall community towards a common working model. > what is the harm in having multiple competing solutions? Developer confusion and community fragmentation.

damnever 4y ago

I am the author of pigar[1], and I am using Go a lot, Go has its problems too, but I am a fan of `import "url"` style import statement, developers can write code first, and sync the dependency later with `go mod tidy`. To fix problems in Python's world, Python's community should simplify the tools and cultivate a habit to declare the dependency first(maybe this should be mandatory). [1]: https://github.com/damnever/pigar https://github.com/damnever/pigar