6 ms·
Python Packaging, One Year Later: A Look Back at 2023 in Python Packaging
- bspammer 3y agoI really enjoyed this too (linked in the article) https://gregoryszorc.com/blog/2023/10/30/my-user-experience-porting-off-setup.py/ https://gregoryszorc.com/blog/2023/10/30/my-user-experience-...
- bvrmn 3y agoYes, it's a great read. It validated my frustration with python packaging.
- infecto 3y agoI always struggle with these discussions. I have been using Python a long time and while there ahve definitely been a handful of gotchas over the years, it was never an experience that held me up long enought to think twice about. I am sure there is a better way but what holds me up is how often it impacts other people but not myself. Perhaps my usage is not advanced enough but it always leaves me wondering. Edit: Just as an additional thought, is one of the main issues when distributing to client machines to what is essentially python executables and the clients could be using different OS? My mind jumps to probably not using Python at that point unless there was a specific dependency/library to using it.
- bbojan 3y agoSame here. I always see people complaining about Python packaging, but rarely run into issues myself. Maybe it's an OS issue? I use Linux. Some of the projects at work use gigabytes of computer vision dependencies and again I don't remember the last time I had any issues. Could some of the people who do have issues give an example or two of the problems they run into?
- oblio 3y agoYou don't try to distribute Python applications to multiple OSes, I assume?
- epage 3y agoPart of the problem is the frustration is the curse of choice, the need to research what tool to use. Before the new crowd of tools, like poetry, the problem was also having to figure out how to chain all the scattered tools together. I likely still have some bias from this experience. I stopped doing python before getting to adopt any of these tools but when I researched Poetry for a company to use, a fatal flaw is that they copied too much from Rust/Cargo, Cargo was born with its community and they help shaped each other. The rust community has a strong adherance to semver. In Python, you have at mess of versioning schemes (e.g. CalVer) and low quality version requirements. You need a way to override version requirements but they refuse. Doing cross platform development? The tools that do locking today via `requirements.txt` generate platform-specific lockfiles.
- throwup238 3y ago> CUDA version on your system does not match the CUDA version torch was compiled with Bane of my existence.
- gipp 3y agoI mostly agree with the GP comment, but CUDA support has definitely been the one exception that was just an absolutely infuriating experience.
- Shish2k 3y agoYes this - I’ve been using python for decades, and been mostly fine (dropping down to ~zero packaging & deployment issues since I started putting all my services into docker images). But AI stuff? Somehow that manages to be a massive pain in the ass every single time
- marbs 3y agoHere are the two main packaging issues I run into, specifically when using Poetry: 1) Lack of support for building extension modules (as mentioned by the article). There is a workaround using an undocumented feature [0], which I've tried, but ultimately decided it was not the right approach. I still use Poetry, but build the extension as a separate step in CI, rather than kludging it into Poetry. 2) Lack of support for offline installs [1], e.g. being able to download the dependencies, copy them to another machine, and perform the install from the downloaded dependencies (similar to using "pip --no-index --find-links=."). Again, you can work around this (by using "poetry export --with-credentials" and "pip download" for fetching the dependencies, then firing up pypiserver [2] to run a local PyPI server on the offline machine), but ideally this would all be a first class feature of Poetry, similar to how it is in pip. I don't have the capacity to create Pull Requests for addressing these issues with Poetry, and I'm very grateful for the maintainers and those who do contribute. Instead, on the linked issues I share my notes on the matter, in the hope that it may at least help others and potentially get us closer to a solution. Regardless, I'm sticking with Poetry for now. Though to be fair, the only other Python packaging tools I've used extensively are Pipenv and pip/setuptools. It's time consuming to thoroughly try out these other packaging tools, and is generally lower priority than developing features/fixing bugs, so it's helpful to read about the author's experience with these other tools, such as PDM and Hatch. [0] https://github.com/python-poetry/poetry/issues/2740 https://github.com/python-poetry/poetry/issues/2740 [1] https://github.com/python-poetry/poetry/issues/2184 https://github.com/python-poetry/poetry/issues/2184 [2] https://pypi.org/project/pypiserver/ https://pypi.org/project/pypiserver/
- jonawesomegreen 3y agoI've encountered frustration with Python packaging in two main areas: 1) Installing applications within Docker containers. While wheels have improved this situation, I was surprised that building a package for easy copying into a container and running without the need for installing build tools and extensions in the final container image was not straightforward, especially coming from other languages. 2) Distributing Python utilities to end users across various platforms in an easily installable manner without requiring them to follow lengthy instructions to set up all the dependencies has been another challenge. We use Poetry and have largely "solved" #1 with a somewhat complicated Docker build. It works well now, so no one has to think about it much. That made deploying Python server-side fairly easy. However, #2 has been much more of a challenge, and I wonder if that is where other folks in this thread are feeling the most pain.
- kingkongjaffa 3y agoPyinstaller has been robust enough for us to deploy Python written windows executables without any issues.
- guappa 3y agoAll of those problems can be easily solved by distribution packages.
- ogrisel 3y agoDistribution packages are nice but: - they require different instructions for each platform: one for each distribution: it's not possible to give a one liner that will work for all supported OS/architecture combinations, - distributions are not all updated at the speed of upstream. Sometimes users do not want to wait several years to use a recently released feature.
- coldtea 3y ago>it was never an experience that held me up long enought to think twice about The pain is when you're doing more complex stuff than creating a local virtualenv with some basic deps though.
- infecto 3y agoThat is why I was wondering. I have produced a lot of software using python, deployed it on different platforms, using containers, not using containers, bare metal, cloud, using complex dependecies, etc. I have not run into a lot of issues and most of them for me were mostly depedency verion related where different versions were required for subdependencies but even then it was not super complicate. I realize more complex scenarios exist but it surprises me that its such a hot topic always with Python.
- volker48 3y agoThe only time I've ever experienced a memorable amount of pain was when I was working on a project that used one of the newer options like Poetry. Using a virtualenv, a requirements.txt file, and pip I have not had serious issues in 15 years of using Python.
- cassianoleal 3y agoPython packaging is really bad. These days I just use virtualenv since at least it's simple to use and reason about. It's bad, just not nearly as bad as the others I've tried.
- hasty_pudding 3y agopoetry is awesome. i recommend it
- jzombie 3y agoIt won't install tensorfow on a Mac Mx series without hacking a custom config. I gave up on Poetry after that and try to stick with pip now.
- Hackbraten 3y agoYou might like Relieving your Python packaging pain [1]. The article seems to come to a similar conclusion as you do. [1]: https://www.bitecode.dev/p/relieving-your-python-packaging-pain https://www.bitecode.dev/p/relieving-your-python-packaging-p...
- cassianoleal 3y agoIndeed. I'm not too fussed about points 1 and 2. I will install whatever version works for me, usually via Homebrew. Otherwise, sound advice.
- 3y ago
- hasty_pudding 3y agoUse Poetry. Python packaging problems solved.
- apple4ever 3y agoThat's what I found. It just works. I never have to worry about anything. Python should just adopt it.
- drcongo 3y agoI don't know why the PyPA seems to be stubbornly ignoring Poetry, most Python devs I know have switched to it and are very unlikely to switch again to something else. It does what you need it to do, it's stable and reliable these days, and the ergonomics are good.
- Hackbraten 3y agoI wish Poetry were PEP-621 compliant though. [1] Currently, it uses a proprietary configuration group (or "tool section", as they seem to call it in `pyproject.toml` speech). [1]: https://github.com/python-poetry/roadmap/issues/3 https://github.com/python-poetry/roadmap/issues/3
- drcongo 3y agoI kinda wish PEP-621 had gone more down the pyproject route that poetry took, their version looks so much cleaner and more readable at a glance than the PEP-621 version.
- nerdponx 3y agoHatch seems to be the preferred tool by PyPA, probably because it's more standards-compliant and it claims standards-compliance as a design goal.
- deleted 3y ago[deleted]
- bbojan 3y ago
- smitty1e 3y agoSeeing the Poetry support on the thread, maybe just 1. pick the best of breed, 2. have the PSF sponsor a Packaging kickstarter for those of us who'd toss in some money but lack time/skill to do more than whine, and 3. get this mess fixed up for 3.13 this Fall or 3.14 (pi-thon?) in 2025? I get it that the feature Venn across the scene has many disjoint parts. The management seems a greater challenge than the code itself.
- jjgreen 3y agoget this mess fixed up for 3.13 this Fall or 3.14 (pi-thon?) in 2025? I admire your optimism but don't share it. Crap packaging will be present in the Python experience for the lifetimes of multiple dogs. 2075 at the earliest.
- smitty1e 3y agoI submit that the existing packaging mosaic covers the requirements (almost) totally. Just need to incentivi$e the proper ninja(s) to merge the functionality into One True package. Whether or not that is a native python, or C-port of all this Rust goodness, or the Rust binary will be a sticky wicket.
- ggregoire 3y agoI use pip inside docker and it just works.
- coldtea 3y agoIf by "just works" you mean "now I also need to have and maintain a container image for isolation", yes.
- ggregoire 3y agoThere is nothing to maintain. I bump python in the dockerfile if I need to bump it for some reasons, otherwise there is no reason to change anything and it indeed just works. Takes like 2 min of work once a year. I can rebuild an image from 5 years ago and deploy it on multiple OSes and it will probably still work.
- coldtea 3y ago>There is nothing to maintain. Yeah, just install a root-owning Docker service, run everything in a virtualized container, write the dockerfile -- "nothing to maintain".
- boxed 3y agoThe loss of Guido as the BDFL was not good for this. And honestly, even before that he was too much hands off to resolve this situation sadly.
- nerdponx 3y agoI'm less pessimistic about packaging than most. No language that I know of has ever attempted a standardization effort like this. I think it will pay off in the long term. But it's taken years to get here and it will take more years to wrap up the project. > An attempt at a specification was rejected due to “lukewarm reception”, even though there exist at least four implementations which are achieving roughly the same goals, and other ecosystems also went through this before. Python has been weird lately. We got structural pattern matching, which is entirely new syntax that is essentially just syntactic sugar and only kind-of solves a problem that relatively few users ever had. But then we reject __pypackages__ and null-coalescing/safe-navigation operators, which solve problems that everyone has and are unambiguous improvements and modernizations of the language. Even if the PEPs had problems and needed to be rewritten from scratch, there is now approximately zero chance that will ever happen. I love Python but sometimes I feel like the decisions are arbitrary and do not reflect what is best for the language. It drives me nuts because I'm a captive user. There's too much momentum for data science and machine learning to switch and be taken seriously in a non-solo workplace. Also I just like Python.
- ngrilly 3y ago> No language that I know of has ever attempted a standardization effort like this. What about Ruby, Rust, or Go, for example?
- oblio 3y agoOr, you know, Java, the language that solved packaging around 2004.
- nerdponx 3y agoI don't know how Java packaging works. Got a short summary?
- oblio 3y agoWell, I'll start from the original sin for JS/Python/Ruby/PHP/Perl: 0. Java was designed to be fast enough to be comparable with C/C++. Think 2x slower or less, not 10-100x slower. This leads to: 1. Most Java libraries are in Java, they're not native. And they're portable as-is across platforms. If they do have platform specific code, it's their job to make sure they bundle and load everything needed for the specific problem. 2. In 2004 Maven 1 was launched, then in 2005 Maven 2 came with a repository format update. Packages are zip files called jars (Java ARchive), there are also wars (Web ARchive) which are zip-of-zips meant to be unpacked by the application server before the first launch, or ear (Enterprise ARchive), which are also zip-of-zips but I forget the exact details for using these. 3. The Maven repo format is the same, locally and remotely. When you work with Maven (or Gradle, or any Java build tool, since they're ALL compatible with Maven, otherwise nobody would use them), you get a local cache/mirror/proxy of the remote repos. 4. Because Java has CLASSPATH (sort of like PYTHONPATH but probably better, and it probably inspired PYTHONPATH because I'm sure the Java CLASSPATH predates it), packages are not copied over to the local folder when developing. Maven & co just assemble the correct CLASSPATH and everything is referred to directly from the local repo. You don't have venvs or node_modules because those are just silly hacks that aren't needed here. 5. If you need to package Java stuff, the standard approaches are: - cross platform jars for libraries (these are usually published to the Maven repo and can be used natively by any Java package manager) - zips for desktop apps; shell scripts or executable launchers inside the zips to launch the apps - wars for web applications - ears for huge, enterprise applications It's obviously very deep and complex when you want to look at everything, but that's it. Because Java is really close to "Write Once, Run Anywhere" in practice, for most platforms yeah, you just copy over the jar/zip. The hardest part that people complain is installing the JRE, which is SUPER silly since for a technical person it should be trivial to do. For non technical person, since at least 10 years ago, you can just bundle the JRE with your app. These days (at least 5+ years), I'm fairly sure you can AOT compile the app, even. Python, by comparison, is a horror show. And it all comes from 0. -> Python is slow, it was meant to be used with C libraries, so it carries all the baggage from that ancient and creaky ecosystem. Packaging a Python (or Ruby, or...) app to deploy on Windows, Linux (multiple distros), MacOS is such a horror show that they invented an entire layer with Docker, to just put the whole thing into an almost literal shipping container and not bother with the craziness inside.
- lysecret 3y agoNowadays I am usually using micromamba inside Docker. I honestly never had any issues. I need docker anyway for deployment. And mamba/conda is almost needed for scientific stuff (working with some super niche weather stuff which is only on conda). IMO the most important thing is to stay flexible, all of them have their tradeoffs pick one that works and move to a different one if it becomes painful. Docker is very very useful though. Can you imagine the joy I felt when I spun up a 6 month old very complex project and it immediately worked.
- t43562 3y agoI'm more worried about commercial entities gaining power - by simply employing enough people to work on something that the alternatives cannot compete. People who want "one" anything - I can sympathise with the desire for simplicity (as long as it's a kind that suits you) but I hope it never happens.
- oblio 3y agoThis is a niche concern. There are no commercial package managers that I know of, in any mainstream ecosystem. Even Gradle, which probably comes closest, is perfectly usable without the commercial add-ons.
- nerdponx 3y agoArguably, Conda is a commercial product that exists to serve the needs of Anaconda first and others second. Fortunately Anaconda has been a very benevolent overlord.
- t43562 3y agoI'm not thinking of commercial anything - just changes financed by companies that want them. Example: PyPy getting almost no funding but Microsoft getting it's JIT into cpython. Big money will be deciding what happens. We're already all dependent on Github - and they've managed to use our code to train their AI. Like a vampire offering poor travellers a bed in it's castle and then sucking their blood :-D
- dannyz 3y agoFrom a user point of view Python packaging has probably never been better, but it is still a huge mess from anyone having to maintain these projects which might explain some of the dichotomy in the comments. In my experience most of these packaging tools work fine for pure Python packages, it is when you try and bundle extensions in a cross-platform way that things get really messy. For better or worse I think third party package managers somewhat outside the Python ecosystem, i.e. conda + conda-forge, are the only tools that get this right.
- bvrmn 3y agoI have a dozens of packages created since 2009 and never had issues with distutils, setuptools also worked fine for me. Even C extensions weren't a problem. Recently I've tried setup.py-less approach. Oh boy. What a mess "modern" python packaging is. Multiple backends. PEP-517. pip<23 could not install your package. `build` tool includes explicitly excluded files. At least you can include markdown as a README for the package.
- nerdponx 3y agoSetuptools is supposed to work with PEP 517 just fine with no changes, other than adding the build-backend declaration in pyproject.toml. Old versions of Pip will find your setup.py and use it directly, and new versions will find it via pyproject.toml and invoke it via `build`. Maybe you hit a bug? Or were doing something unusual and complicated. I never had a problem with the transition in a Setuptools pronect.