21 ms·

My User Experience Porting Off Setup.py

150 points by markdog12 3y ago

stop50 3y ago
Its more funny to migrate the old setup.py process to the pyproject.toml and an rpm specfile. with an setup.py you can say something like "python3 setup.py install -root=$builddir"
- cbarrick 3y ago
  With project.toml, the new strat is to build the wheel with `python3 -m build` and then install it with `pip install --root=$DESTDIR` plus a handful of flags to tell pip not to touch the network or the local cache. It's not great, but it's also not terrible. E.g. https://github.com/cbarrick/efiboot/blob/2ca46a7c27c837adf23ad1e9bcc73a2398f45b9d/Makefile#L162 https://github.com/cbarrick/efiboot/blob/2ca46a7c27c837adf23...
  - rpier001 3y ago
    Out of context (and with limited knowledge) comment 3 seems terrifying.
    - cbarrick 3y ago
      I assume you mean this part: > We will clobber any previously installed version of this package, even if it breaks whatever else is installed. It's the user's job to make sure that is all sorted out ahead of time. But this is a `make install` recipe, and I think this is generally expected behavior when running that command. Typically, this would be run in a chroot when building a package.
  - stop50 3y ago
    I know that now, but at the time i migrated i didn't and i had the same problems as the post. I still have my problems with using python in the company but it isn't from python or its ecosystem itself. Its from redhat splitting their python packages into the different supported python versions(package a is only available for 3.6 and package b only for 3.11)
  - jjgreen 3y ago
    Don't try running that on Debian: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1035546 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1035546
    - cbarrick 3y ago
      Oh wow, this is a pretty gnarly issue for the Python packaging story, considering the looming deprecation of setup.py. I traced the discussion to [1], which is an interesting read, but it seems that progress on this died out in April, at least on the Python side. [1]: https://discuss.python.org/t/linux-distro-patches-to-sysconfig-are-changing-pip-install-prefix-outside-virtual-environments/18240/34 https://discuss.python.org/t/linux-distro-patches-to-sysconf...
      - jjgreen 3y ago
        Grim. Python packaging is a skip (dumpster) full of dead rats.
Taikonerd 3y ago
Packages have always been Python's Achilles heel. Python's philosophy for code is, "there should be one, and preferably only one, obvious way to do it." But for packaging systems, their philosophy is more like Perl: "there's more than one way to do it (and all of them have their own pitfalls)." I don't understand why the Python leadership hasn't shown stronger... leadership... in which tools they recommend.
- MrJohz 3y ago
  My impression is that there's lots of different niches that have developed, each with their own needs, and trying to unify them is a mess. If you're putting something like a web app or something else that will be bundled and distributed as a unit, then you're probably best off with something like Poetry, PDM, or pip-tools - you have a lock file for deterministic dependencies, most of your dependencies will be pure python wheels, and you only really need to test things once. On the other hand, if you're developing a library, you'll need to test against multiple versions of Python, and ideally multiple versions of some of your larger dependencies, to ensure that your library will work for as many users as possible. You'll also need to be able to build and package wheels. Alternatively, you're working in data science, and your main concern is probably making sure you can install the packages you need in the environments you're going to use them - specifically, so that they work with the GPUs and other hardware you have available. And there's still the group of people writing mainly scripts for server maintenance or other tasks, who want to be able to easily install dependencies and keep those dependencies up to date for security reasons, with the minimum number of breaking changes. Right now, there are different tools, packaging systems, etc catering to each of these groups, and so building the One Ring of Python package management is going to involve (a) solving all of these problems, and (b) convincing all these groups of people that your general solution is better than their niche-specific solution. That's certainly not easy, I don't even know if it's all that possible. I do think that working from the ground up (i.e building the individual components like the package metadata file, or the pypackages folder experiment) seems to be working well, in that tools seem to be coalescing around these options and finding the best ways to use them, which is all work that might hopefully feed into new official tooling. But we'll see.
_dain_ 3y ago
A friend of mine had a critique of videogames that has stuck with me. He called them "antiknowledge". Your brain's learning faculties are repurposed into "learning" something that isn't actually a thing. You get really good at some artificial skinner box treadmill system ... and then they change the meta and you have to keep up with the new stats and strategies and so on. Yet after sinking in hundreds of hours, you don't come out the other end with any tangible real-life skill, that would be useful outside of the game. That's what Python packaging feels like. At least videogames are fun.
- Pannoniae 3y ago
  Not all of them! Many of those skinnerbox type games actually have a pretty mediocre or outright unfun game loop - it's the skins, battle passes and "rewards" which keep people playing.
- totallywrong 3y ago
  That's nonsense. The same could be said of most hobbies. The motivation isn't learning real life skills, it's entertainment.
  - smitty1e 3y ago
    I'm +infinity on "antiknowledge", but I like your "entertainment" point. To harmonize these ideas, I argue to the kids that the "entertainment" choices are like spice on the food. We want enough to enjoy the flavor in question, but not so much that the usage becomes expensive or overpowers the dish.
  - atoav 3y ago
    It can be. But isn't it a question of balance? If you play some useless¹ game for the majority of your days, you can have multiple reasons for this. Maybe it keeps you occupied and thinking of the problems in your life? But after a certain degree it certainly does not "entertain" you anymore: You are doing it because doing anything else that you can think of feels worse. And a few years after not a lot will be left of it, unless you played with your friends or there was something in it for you beyond just pushing the time forward. If I am playing my instrument for hours, I have at least improved at expressing my feelings with my instrument while having a generally good and relaxing time. And after years of doing it I can do it well enough to play on concerts without feeling afraid. I know to many people who are so afraid of their own thoughts, they will obsessively "entertain" themselves dueinf all wake hours, and the majority of the time they are not enjoying it. This is bad. We only have one life. You don't get a price for having some the most, but at least do something that is meaningful. ¹ a game that you yourself don't even love that much
_8j50 3y ago
I just wanna say I really hate the new poetry+toml crap, it's just more complicated and yet another set of crap I gotta deal with arbitrarily by arbitrary projects. Why not just improve upon pip? I don't know, just have pip use toml and give it different flags or auto detect things? Was a whole new ecosystem of tools needed? I look at decisions like this and I know a Python 4.0 is on the way, just like the 2->3 jump because why not right? Any language that updates it's syntax or toolset in a way that is impossible to have backwards compatibility is an irresponsible toy language, as awesome and powerful as it may be, the developers are still toying and tinkering with it with no regard to real world impact of their decisions. Why can't python3.12 have a flag like --std=2.7.18 for example, like gcc? If the devs don't have enough resources, I would live to see a donate page I can help out with. We are at a point where to deploy python you can't use python but you need shell scripts to figure out versions, venvs, pyev, pipx, poetry, etc... and reliably fail of course and every time you have to troubleshoot the problem. This is a failure in software engineering, new grads should be taught of python and similar languages and the lack of planning and organization and resulting cascading chaos as examples of how not to design the user experience of any piece of software. Sorry if I exaggerated a bit anywhere, it's difficult to pretend all the frustrations and crying out "why???" when using python don't exist. But at the same time, it is still my #1 go to language for most use cases because the language itself is just fabulous!
- piyh 3y ago
  I deal with aggressive vulnerability management at work, and we have a unified pipeline. Figuring out what our servers will be running versus what we are doing locally versus what the pipeline expects us to do for dependencies, plus vuln management is so much work. God forbid you have to upgrade a base python version and reopen the dusty tomes of python dependency hell.
  - _8j50 3y ago
    I work in infosec as well and I have yet to even look into malicious pip packages (although I've seen malicious nuget packages), with the last curl vuln it was chaotic, telling people a lot of things actually use libcurl. Can you imagine if something like the requests or urllib package became compromised, absolutley no real way to manage the patching, projects using old versions of it will be forced to upgrade and every package that claims it needs a specific version will break, pure chaos!
__mharrison__ 3y ago
Ouch. I stopped reading 15% of the way through because I noticed that I was only 15% of the way through and I felt like it should be the end. It seems like with a tenth of the effort of this blog rant the author could have written a flowchart or best practices. Luckily for 99% of Python users, we only need to install libraries and not package them...
- acemarke 3y ago
  I wrote a very similar post a couple months ago about trying to modernize the Redux JS packages to support ESM and CJS module formats: https://blog.isquaredsoftware.com/2023/08/esm-modernization-lessons/ https://blog.isquaredsoftware.com/2023/08/esm-modernization-... I _liked_ Gregory's post. I _felt_ all of the frustration, and the confusion over lack of good docs, and the plethora of competing tools and formats. It resonated deeply for me. I haven't touched Python in years, but I could see exactly how all these pain points were happening thanks to the writing style and the explanations. The goal is to share "here's what I tried", "here's what seems wrong or broken", "here's the pitfalls I ran into", and "here's the _frustration_ I'm feeling at how messed up this all is". And it succeeded. (And fwiw, I've had a lot of other folks in the JS ecosystem express similar thoughts as thanks for my post. It's a limited target audience, most people won't care, but the folks who _do_ have to deal with these kinds of problems understand and appreciate the details and the effort involved.)
  - golergka 3y ago
    Thank you for this post! I've recently read it and then decided to abandon any attempt at such a migration.
- bjornasm 3y ago
  How can he write that when seemingly there is no best practices from Python?
  - jjgreen 3y ago
    "Rewrite in Pascal"?
akubera 3y ago
I've had a similarly frustrating time trying to understand and wrangle the pyproject.toml builder system, (egg-layer? wheel-roller? cheese-monger?) One thing the author might want to try is writing their own "build-backend". You can specify your own script (even use setup.py) and that will be the target of python -m build or pip wheel or presumably whatever build-frontend you use. # pyproject.toml [build-system] requires = ["setuptools"] build-backend = "setup" # import setup.py as the build-module backend-path = ["."] Then in setup.py you should write two functions: def build_sdist(sdist_directory, config_settings): ... def build_wheel(wheel_directory, config_settings, metadata_directory): ... Where config_settings is a dictionary of the command line "--config-settings" options passed to the builder. (sys.argv does not have access to the actual invocation, I suppose to ensure frontend standardization) example: $ python -m build --config-setting=foo=bar --config-setting=can-spam # will call >>> build_sdist("the/dist/dir", {"foo": "bar", "can": "spam"}) Of course, you can extend the default setuptools build meta so you only have to do the pre-compilation or whatever your custom build step requires: from setuptools.build_meta import build_sdist as setuptools_build_sdist def build_sdist(sdist_directory, config_settings): # ... code-gen and copy files to source ... # this will call setup.py::setup, to make things extra confusing return setuptools_build_sdist(sdist_directory, config_settings) I had to create a temporary MANIFEST.in file to make sure that the setuptools build_sdist saw the generated files. Maybe there's a better way? I think the wheel "just" packages whatever the sdist produces, though that might be more difficult if you're compiling .so files or whatnot. Still overall pretty fiddly/under-documented and a shame there seems to be a push for more dependencies rather than encouraging users to build their own solutions. More info in PEP 517: https://peps.python.org/pep-0517/ https://peps.python.org/pep-0517/
deleted 3y ago
[deleted]
ahgamut 3y ago
Thank you Gregory for writing this post. There have been a bunch of announcements about "setup.py has been deprecated", but few have clearly outlined how to move away from setup.py, and more importantly, fewer have outlined what a struggle it is to move away from setup.py. I was sad to see setuptools officially deprecated, because it looks like another way in which Python packaging is being red-taped away for a non-expert. If someone like the OP (who has 10+ years programming Python) had to do so much for what appears to be a zstd CFFI/Rust wrapper, where does that leave the rest of us? Here's a python package of mine that uses setup.py: https://github.com/ahgamut/cliquematch/blob/master/setup.py https://github.com/ahgamut/cliquematch/blob/master/setup.py which I have not upgraded to the new tool(s) yet. I think I will need to upgrade it soon. If anyone has suggestions for a tool that will _fully replace_ setup.py, I would like to see tutorials with the following examples: 1. How would I build a package that has pure-Python files and data files? With setuptools I would use maybe MANIFEST.in or package_dir. 2. How would I build a package that has a CPython extension accessed via cffi? (this post points to the answer) 3. How would I build a package that has a CPython extension _without_ cffi, that just wraps some small C code I wrote with CPython's API? What about an extension that uses PyBind11? What about an extension that uses Rust? 4. How would I build a package that requires a "system" package like libblas-dev? Can something like numpy be built optimally without ever writing setup.py? What would a config for that look like? Last I remember numpy used their own patch of distutils to build, I wonder what it is now.
- ketozhang 3y ago
  Here's a packaging guide that answers most of your questions. https://learn.scientific-python.org/development/guides/packaging-compiled/ https://learn.scientific-python.org/development/guides/packa... As a TLDR, you have many options 3rd party build tool (aka build backends). Each build tool have different *static* ways to specify compile options that is native to the language or generic (e.g., CMakeList,s Cargo.toml, 3rd party YAML. When it comes to dynamically specifying your extensions, setuptools is still the only option.
JodieBenitez 3y ago
> No offence meant to the Poetry project here but I don't perceive my project as needing whatever features Poetry provides: I'm just trying to publish a simple library package I don't understand... Poetry solves this too.
- phanimahesh 3y ago
  It is not obvious it does.
  - JodieBenitez 3y ago
    It's literally written on the home page: SHARE YOUR WORK Publish Make your work known by publishing it to PyPI You can also publish on private repositories
anacrolix 3y ago
from* ffs
PennRobotics 3y ago
I open https://blog.ganssle.io/articles/2021/10/setup-py-deprecated.html in my browser and see a 4,000+ word blog post. Oof. Do I really want/need to read this? *proceeds to write an 8,000 word blog post about it* ----- Kidding aside, good content and a great reminder to think before blindly upgrading—at least until the kinks and details are worked through
wodenokoto 3y ago
I’m quite surprised to see that setup.py has been deprecated for years. I set out to setup my first package earlier this year and after spending way too much time trying to figure out what is recommended and why I actually settled on setup.py I knew it was “the old” way, but didn’t realize it was abandoned. Getting your code packaged seems way harder than developing your code.
- ketozhang 3y ago
  It is not deprecated. Calling setup.py directly is (i.e., `python setup.py sdist`) Modern way of building a package is to use a build frontend. Most notably is the PyPA's build package. python -m build . Most project managers like poetry has this built-in with it's `poetry build` command.
stabbles 3y ago
About "eager deprecations" let me give you another absolute gem: ******************************************************************************** The license_file parameter is deprecated, use license_files instead. By 2023-Oct-30, you need to update your project and remove deprecated calls or your builds will no longer be supported. ******************************************************************************** Yes, please go ahead and break people's builds at an arbitrary date because the technical challenges of supporting both `license_file` and `license_files` are insurmountable.
- fmajid 3y ago
  Or packages' need to support more than one version of Python.
- xnorswap 3y ago
  To be fair, that seems to have been a 2.5 year warning: https://github.com/pypa/setuptools/commit/3544de73b3662a27fac14d8eb9f5c841668d66de https://github.com/pypa/setuptools/commit/3544de73b3662a27fa...
  - stabbles 3y ago
    Just as an example: CMake ensures roughly a decade of backwards compatibility. This `license_file` change is the epitome of unnecessary breakage.
mekoka 3y ago
As someone who just yesterday also went through the exercise of (finally) understanding how to package with pyproject.toml, I empathize with the author and agree with a few painpoints he mentioned. Namely the confusion caused when opening some pages in https://packaging.python.org/en/latest/ https://packaging.python.org/en/latest/ and seeing references to a number of soon to be deprecated tools and approaches to packaging, as though they still have a place in the horizon. It's especially frustrating because the website is versioned, so you would expect a deliberate use of deprecation warnings and clear recommendations to migrate to new approaches and tools. I opened that website with the understanding that pyproject.toml is the future and setup.py is out. Instead, I still saw pages where the two are treated as though they will coexist for a while. Having said that, the author also sounds like he's ranting a bit. He seems to insist in finding specifically how to work the way setup.py used to, but without setup.py, instead of just learning how to use pyproject.toml. While learning the new way of doing something, how it replaces the old way is usually self-evident. The (official) tutorial he eventually lands on (https://packaging.python.org/en/latest/tutorials/packaging-projects/ https://packaging.python.org/en/latest/tutorials/packaging-p...) is actually a pretty good primer. Without previously knowing what hatchling, build, twine, or even pyproject.toml was, I was able to quickly understand their purpose. From clicking a few other links on the side bar, I understood that packaging is done with tools that present a frontend and interact with a backend. Sometimes a tooling set provides both. Hatch seems to be the frontend of one such project, while Hatchling is the backend.
- eesmith 3y ago
  > The (official) tutorial he eventually lands on ... is actually a pretty good primer. That tutorial you linked to does not describe how to handle the issue the author faced in building a Python binary extension. It doesn't even describe how to build a C/C++/whatever extension. At the bottom of the page the text "Read about Packaging binary extensions" points to https://packaging.python.org/en/latest/guides/packaging-binary-extensions/ https://packaging.python.org/en/latest/guides/packaging-bina... which starts "Page Status: Incomplete" and "Last Reviewed: 2013-12-08", and contains a number of "FIXME" sections. (Parts have been updated during the decade, but the contents of the page do not instill confidence that it's up-to-date.) That documentation still doesn't show how to configure the extension, instead linking to https://docs.python.org/3/extending/extending.html https://docs.python.org/3/extending/extending.html . That in turn links to https://docs.python.org/3/extending/building.html https://docs.python.org/3/extending/building.html which links to https://setuptools.pypa.io/en/latest/setuptools.html https://setuptools.pypa.io/en/latest/setuptools.html which takes you to https://setuptools.pypa.io/en/latest/userguide/ext_modules.html https://setuptools.pypa.io/en/latest/userguide/ext_modules.h... showing how to configure your setup.py. However, unlike using setup.py directly, where you can hack setup.py's argv, that documentation does not show any way to specify compile-time parameters, "like --system-zstd and --rust-backend as a way to influence the build". Which is what the author wants to do.
ri0t 3y ago
I've decided (a few years ago) that if i'll ever have to upgrade the packaging for my stuff, i'm going to do it with nix (and only that - sorry other os users, you'll have to install nix - actually: not sorry!). I was in (distro) packaging way too deep and decided that my limited time in this world doesn't allow for that kind of crap, anymore. The blatant mess that ensued in the meantime (i.e. last few years) proves me right, imho.
fastasucan 3y ago
I think the documentation (or lack of documentation) can be very illuminating in the focus and though process of those behind a project. That there is concise "This is how you did it before and this is how it is done now" shows that there seemingly haven't been any though process in building that bridge. And in general it seems like the process of packaging projects that is not going to be published on pypi, but are going to be used internally is a dark spot as well. Having the mess of different blog posts and documentation sources saying different stuff is far from ideal though. If you can't sum up the process in a clear concise way you are far from done.
- guappa 3y ago
  Python removed distutils… basically now there is no official way to install things.
  - eesmith 3y ago
    The Python documentation says "pip is the preferred installer program. Starting with Python 3.4, it is included by default with the Python binary installers." - https://docs.python.org/3/installing/index.html#installing-index https://docs.python.org/3/installing/index.html#installing-i...
    - guappa 3y ago
      Except that pip doesn't actually do the install part, you need some other module for that.
      - eesmith 3y ago
        I do not understand your comment. What module are you talking about? What prevents the pip from a normal CPython installation from doing an install? You can even bootstrap pip via the ensurepip module documented at https://docs.python.org/3/library/ensurepip.html https://docs.python.org/3/library/ensurepip.html which notes "This module does not access the internet. All of the components needed to bootstrap pip are included as internal parts of the package."
paradox242 3y ago
It is as if someone has taken the time to describe my typical day. I'm not a package maintainer or anything, just a guy trying to keep multiple CI/CD systems working. I did not get halfway through this before I could start to feel the hairs of my neck start to stand on end. The numerous blind alleys. The promising leads that aren't. The official documentation that contains links to other documents that contradict the first and so I have to try and also piece together some temporal state to make any sense of any of it. Some days it seems I am actually some kind of an information archeologist, piecing together the detritus of overlapping civilizations.
- ketozhang 3y ago
  In this particular case. OP had a lot of knowledge which served as a crutch in this journey. The less you are aware of how setuptools used to work, the better.
Nimitz14 3y ago
Thank you for sharing this journey! Very helpful.
deleted 3y ago
[deleted]

150 points by markdog12 3y ago

Its more funny to migrate the old setup.py process to the pyproject.toml and an rpm specfile. with an setup.py you can say something like "python3 setup.py install -root=$builddir"

cbarrick 3y ago
With project.toml, the new strat is to build the wheel with `python3 -m build` and then install it with `pip install --root=$DESTDIR` plus a handful of flags to tell pip not to touch the network or the local cache. It's not great, but it's also not terrible. E.g. https://github.com/cbarrick/efiboot/blob/2ca46a7c27c837adf23ad1e9bcc73a2398f45b9d/Makefile#L162 https://github.com/cbarrick/efiboot/blob/2ca46a7c27c837adf23...
- rpier001 3y ago
  Out of context (and with limited knowledge) comment 3 seems terrifying.
  - cbarrick 3y ago
    I assume you mean this part: > We will clobber any previously installed version of this package, even if it breaks whatever else is installed. It's the user's job to make sure that is all sorted out ahead of time. But this is a `make install` recipe, and I think this is generally expected behavior when running that command. Typically, this would be run in a chroot when building a package.
- stop50 3y ago
  I know that now, but at the time i migrated i didn't and i had the same problems as the post. I still have my problems with using python in the company but it isn't from python or its ecosystem itself. Its from redhat splitting their python packages into the different supported python versions(package a is only available for 3.6 and package b only for 3.11)
- jjgreen 3y ago
  Don't try running that on Debian: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1035546 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1035546
  - cbarrick 3y ago
    Oh wow, this is a pretty gnarly issue for the Python packaging story, considering the looming deprecation of setup.py. I traced the discussion to [1], which is an interesting read, but it seems that progress on this died out in April, at least on the Python side. [1]: https://discuss.python.org/t/linux-distro-patches-to-sysconfig-are-changing-pip-install-prefix-outside-virtual-environments/18240/34 https://discuss.python.org/t/linux-distro-patches-to-sysconf...
    - jjgreen 3y ago
      Grim. Python packaging is a skip (dumpster) full of dead rats.

Taikonerd 3y ago

Packages have always been Python's Achilles heel. Python's philosophy for code is, "there should be one, and preferably only one, obvious way to do it." But for packaging systems, their philosophy is more like Perl: "there's more than one way to do it (and all of them have their own pitfalls)." I don't understand why the Python leadership hasn't shown stronger... leadership... in which tools they recommend.

MrJohz 3y ago
My impression is that there's lots of different niches that have developed, each with their own needs, and trying to unify them is a mess. If you're putting something like a web app or something else that will be bundled and distributed as a unit, then you're probably best off with something like Poetry, PDM, or pip-tools - you have a lock file for deterministic dependencies, most of your dependencies will be pure python wheels, and you only really need to test things once. On the other hand, if you're developing a library, you'll need to test against multiple versions of Python, and ideally multiple versions of some of your larger dependencies, to ensure that your library will work for as many users as possible. You'll also need to be able to build and package wheels. Alternatively, you're working in data science, and your main concern is probably making sure you can install the packages you need in the environments you're going to use them - specifically, so that they work with the GPUs and other hardware you have available. And there's still the group of people writing mainly scripts for server maintenance or other tasks, who want to be able to easily install dependencies and keep those dependencies up to date for security reasons, with the minimum number of breaking changes. Right now, there are different tools, packaging systems, etc catering to each of these groups, and so building the One Ring of Python package management is going to involve (a) solving all of these problems, and (b) convincing all these groups of people that your general solution is better than their niche-specific solution. That's certainly not easy, I don't even know if it's all that possible. I do think that working from the ground up (i.e building the individual components like the package metadata file, or the pypackages folder experiment) seems to be working well, in that tools seem to be coalescing around these options and finding the best ways to use them, which is all work that might hopefully feed into new official tooling. But we'll see.

_dain_ 3y ago

A friend of mine had a critique of videogames that has stuck with me. He called them "antiknowledge". Your brain's learning faculties are repurposed into "learning" something that isn't actually a thing. You get really good at some artificial skinner box treadmill system ... and then they change the meta and you have to keep up with the new stats and strategies and so on. Yet after sinking in hundreds of hours, you don't come out the other end with any tangible real-life skill, that would be useful outside of the game. That's what Python packaging feels like. At least videogames are fun.

Pannoniae 3y ago
Not all of them! Many of those skinnerbox type games actually have a pretty mediocre or outright unfun game loop - it's the skins, battle passes and "rewards" which keep people playing.
totallywrong 3y ago
That's nonsense. The same could be said of most hobbies. The motivation isn't learning real life skills, it's entertainment.
- smitty1e 3y ago
  I'm +infinity on "antiknowledge", but I like your "entertainment" point. To harmonize these ideas, I argue to the kids that the "entertainment" choices are like spice on the food. We want enough to enjoy the flavor in question, but not so much that the usage becomes expensive or overpowers the dish.
- atoav 3y ago
  It can be. But isn't it a question of balance? If you play some useless¹ game for the majority of your days, you can have multiple reasons for this. Maybe it keeps you occupied and thinking of the problems in your life? But after a certain degree it certainly does not "entertain" you anymore: You are doing it because doing anything else that you can think of feels worse. And a few years after not a lot will be left of it, unless you played with your friends or there was something in it for you beyond just pushing the time forward. If I am playing my instrument for hours, I have at least improved at expressing my feelings with my instrument while having a generally good and relaxing time. And after years of doing it I can do it well enough to play on concerts without feeling afraid. I know to many people who are so afraid of their own thoughts, they will obsessively "entertain" themselves dueinf all wake hours, and the majority of the time they are not enjoying it. This is bad. We only have one life. You don't get a price for having some the most, but at least do something that is meaningful. ¹ a game that you yourself don't even love that much

_8j50 3y ago

I just wanna say I really hate the new poetry+toml crap, it's just more complicated and yet another set of crap I gotta deal with arbitrarily by arbitrary projects. Why not just improve upon pip? I don't know, just have pip use toml and give it different flags or auto detect things? Was a whole new ecosystem of tools needed? I look at decisions like this and I know a Python 4.0 is on the way, just like the 2->3 jump because why not right? Any language that updates it's syntax or toolset in a way that is impossible to have backwards compatibility is an irresponsible toy language, as awesome and powerful as it may be, the developers are still toying and tinkering with it with no regard to real world impact of their decisions. Why can't python3.12 have a flag like --std=2.7.18 for example, like gcc? If the devs don't have enough resources, I would live to see a donate page I can help out with. We are at a point where to deploy python you can't use python but you need shell scripts to figure out versions, venvs, pyev, pipx, poetry, etc... and reliably fail of course and every time you have to troubleshoot the problem. This is a failure in software engineering, new grads should be taught of python and similar languages and the lack of planning and organization and resulting cascading chaos as examples of how not to design the user experience of any piece of software. Sorry if I exaggerated a bit anywhere, it's difficult to pretend all the frustrations and crying out "why???" when using python don't exist. But at the same time, it is still my #1 go to language for most use cases because the language itself is just fabulous!

piyh 3y ago
I deal with aggressive vulnerability management at work, and we have a unified pipeline. Figuring out what our servers will be running versus what we are doing locally versus what the pipeline expects us to do for dependencies, plus vuln management is so much work. God forbid you have to upgrade a base python version and reopen the dusty tomes of python dependency hell.
- _8j50 3y ago
  I work in infosec as well and I have yet to even look into malicious pip packages (although I've seen malicious nuget packages), with the last curl vuln it was chaotic, telling people a lot of things actually use libcurl. Can you imagine if something like the requests or urllib package became compromised, absolutley no real way to manage the patching, projects using old versions of it will be forced to upgrade and every package that claims it needs a specific version will break, pure chaos!

__mharrison__ 3y ago

Ouch. I stopped reading 15% of the way through because I noticed that I was only 15% of the way through and I felt like it should be the end. It seems like with a tenth of the effort of this blog rant the author could have written a flowchart or best practices. Luckily for 99% of Python users, we only need to install libraries and not package them...

acemarke 3y ago
I wrote a very similar post a couple months ago about trying to modernize the Redux JS packages to support ESM and CJS module formats: https://blog.isquaredsoftware.com/2023/08/esm-modernization-lessons/ https://blog.isquaredsoftware.com/2023/08/esm-modernization-... I _liked_ Gregory's post. I _felt_ all of the frustration, and the confusion over lack of good docs, and the plethora of competing tools and formats. It resonated deeply for me. I haven't touched Python in years, but I could see exactly how all these pain points were happening thanks to the writing style and the explanations. The goal is to share "here's what I tried", "here's what seems wrong or broken", "here's the pitfalls I ran into", and "here's the _frustration_ I'm feeling at how messed up this all is". And it succeeded. (And fwiw, I've had a lot of other folks in the JS ecosystem express similar thoughts as thanks for my post. It's a limited target audience, most people won't care, but the folks who _do_ have to deal with these kinds of problems understand and appreciate the details and the effort involved.)
- golergka 3y ago
  Thank you for this post! I've recently read it and then decided to abandon any attempt at such a migration.
bjornasm 3y ago
How can he write that when seemingly there is no best practices from Python?
- jjgreen 3y ago
  "Rewrite in Pascal"?

akubera 3y ago

I've had a similarly frustrating time trying to understand and wrangle the pyproject.toml builder system, (egg-layer? wheel-roller? cheese-monger?) One thing the author might want to try is writing their own "build-backend". You can specify your own script (even use setup.py) and that will be the target of python -m build or pip wheel or presumably whatever build-frontend you use. # pyproject.toml [build-system] requires = ["setuptools"] build-backend = "setup" # import setup.py as the build-module backend-path = ["."] Then in setup.py you should write two functions: def build_sdist(sdist_directory, config_settings): ... def build_wheel(wheel_directory, config_settings, metadata_directory): ... Where config_settings is a dictionary of the command line "--config-settings" options passed to the builder. (sys.argv does not have access to the actual invocation, I suppose to ensure frontend standardization) example: $ python -m build --config-setting=foo=bar --config-setting=can-spam # will call >>> build_sdist("the/dist/dir", {"foo": "bar", "can": "spam"}) Of course, you can extend the default setuptools build meta so you only have to do the pre-compilation or whatever your custom build step requires: from setuptools.build_meta import build_sdist as setuptools_build_sdist def build_sdist(sdist_directory, config_settings): # ... code-gen and copy files to source ... # this will call setup.py::setup, to make things extra confusing return setuptools_build_sdist(sdist_directory, config_settings) I had to create a temporary MANIFEST.in file to make sure that the setuptools build_sdist saw the generated files. Maybe there's a better way? I think the wheel "just" packages whatever the sdist produces, though that might be more difficult if you're compiling .so files or whatnot. Still overall pretty fiddly/under-documented and a shame there seems to be a push for more dependencies rather than encouraging users to build their own solutions. More info in PEP 517: https://peps.python.org/pep-0517/ https://peps.python.org/pep-0517/

deleted 3y ago

[deleted]

ahgamut 3y ago

Thank you Gregory for writing this post. There have been a bunch of announcements about "setup.py has been deprecated", but few have clearly outlined how to move away from setup.py, and more importantly, fewer have outlined what a struggle it is to move away from setup.py. I was sad to see setuptools officially deprecated, because it looks like another way in which Python packaging is being red-taped away for a non-expert. If someone like the OP (who has 10+ years programming Python) had to do so much for what appears to be a zstd CFFI/Rust wrapper, where does that leave the rest of us? Here's a python package of mine that uses setup.py: https://github.com/ahgamut/cliquematch/blob/master/setup.py https://github.com/ahgamut/cliquematch/blob/master/setup.py which I have not upgraded to the new tool(s) yet. I think I will need to upgrade it soon. If anyone has suggestions for a tool that will _fully replace_ setup.py, I would like to see tutorials with the following examples: 1. How would I build a package that has pure-Python files and data files? With setuptools I would use maybe MANIFEST.in or package_dir. 2. How would I build a package that has a CPython extension accessed via cffi? (this post points to the answer) 3. How would I build a package that has a CPython extension _without_ cffi, that just wraps some small C code I wrote with CPython's API? What about an extension that uses PyBind11? What about an extension that uses Rust? 4. How would I build a package that requires a "system" package like libblas-dev? Can something like numpy be built optimally without ever writing setup.py? What would a config for that look like? Last I remember numpy used their own patch of distutils to build, I wonder what it is now.

ketozhang 3y ago
Here's a packaging guide that answers most of your questions. https://learn.scientific-python.org/development/guides/packaging-compiled/ https://learn.scientific-python.org/development/guides/packa... As a TLDR, you have many options 3rd party build tool (aka build backends). Each build tool have different *static* ways to specify compile options that is native to the language or generic (e.g., CMakeList,s Cargo.toml, 3rd party YAML. When it comes to dynamically specifying your extensions, setuptools is still the only option.

JodieBenitez 3y ago

> No offence meant to the Poetry project here but I don't perceive my project as needing whatever features Poetry provides: I'm just trying to publish a simple library package I don't understand... Poetry solves this too.

phanimahesh 3y ago
It is not obvious it does.
- JodieBenitez 3y ago
  It's literally written on the home page: SHARE YOUR WORK Publish Make your work known by publishing it to PyPI You can also publish on private repositories

anacrolix 3y ago

from* ffs

PennRobotics 3y ago

I open https://blog.ganssle.io/articles/2021/10/setup-py-deprecated.html in my browser and see a 4,000+ word blog post. Oof. Do I really want/need to read this? *proceeds to write an 8,000 word blog post about it* ----- Kidding aside, good content and a great reminder to think before blindly upgrading—at least until the kinks and details are worked through

wodenokoto 3y ago

I’m quite surprised to see that setup.py has been deprecated for years. I set out to setup my first package earlier this year and after spending way too much time trying to figure out what is recommended and why I actually settled on setup.py I knew it was “the old” way, but didn’t realize it was abandoned. Getting your code packaged seems way harder than developing your code.

ketozhang 3y ago
It is not deprecated. Calling setup.py directly is (i.e., `python setup.py sdist`) Modern way of building a package is to use a build frontend. Most notably is the PyPA's build package. python -m build . Most project managers like poetry has this built-in with it's `poetry build` command.

stabbles 3y ago

About "eager deprecations" let me give you another absolute gem: ******************************************************************************** The license_file parameter is deprecated, use license_files instead. By 2023-Oct-30, you need to update your project and remove deprecated calls or your builds will no longer be supported. ******************************************************************************** Yes, please go ahead and break people's builds at an arbitrary date because the technical challenges of supporting both `license_file` and `license_files` are insurmountable.

fmajid 3y ago
Or packages' need to support more than one version of Python.
xnorswap 3y ago
To be fair, that seems to have been a 2.5 year warning: https://github.com/pypa/setuptools/commit/3544de73b3662a27fac14d8eb9f5c841668d66de https://github.com/pypa/setuptools/commit/3544de73b3662a27fa...
- stabbles 3y ago
  Just as an example: CMake ensures roughly a decade of backwards compatibility. This `license_file` change is the epitome of unnecessary breakage.

mekoka 3y ago

As someone who just yesterday also went through the exercise of (finally) understanding how to package with pyproject.toml, I empathize with the author and agree with a few painpoints he mentioned. Namely the confusion caused when opening some pages in https://packaging.python.org/en/latest/ https://packaging.python.org/en/latest/ and seeing references to a number of soon to be deprecated tools and approaches to packaging, as though they still have a place in the horizon. It's especially frustrating because the website is versioned, so you would expect a deliberate use of deprecation warnings and clear recommendations to migrate to new approaches and tools. I opened that website with the understanding that pyproject.toml is the future and setup.py is out. Instead, I still saw pages where the two are treated as though they will coexist for a while. Having said that, the author also sounds like he's ranting a bit. He seems to insist in finding specifically how to work the way setup.py used to, but without setup.py, instead of just learning how to use pyproject.toml. While learning the new way of doing something, how it replaces the old way is usually self-evident. The (official) tutorial he eventually lands on (https://packaging.python.org/en/latest/tutorials/packaging-projects/ https://packaging.python.org/en/latest/tutorials/packaging-p...) is actually a pretty good primer. Without previously knowing what hatchling, build, twine, or even pyproject.toml was, I was able to quickly understand their purpose. From clicking a few other links on the side bar, I understood that packaging is done with tools that present a frontend and interact with a backend. Sometimes a tooling set provides both. Hatch seems to be the frontend of one such project, while Hatchling is the backend.

eesmith 3y ago
> The (official) tutorial he eventually lands on ... is actually a pretty good primer. That tutorial you linked to does not describe how to handle the issue the author faced in building a Python binary extension. It doesn't even describe how to build a C/C++/whatever extension. At the bottom of the page the text "Read about Packaging binary extensions" points to https://packaging.python.org/en/latest/guides/packaging-binary-extensions/ https://packaging.python.org/en/latest/guides/packaging-bina... which starts "Page Status: Incomplete" and "Last Reviewed: 2013-12-08", and contains a number of "FIXME" sections. (Parts have been updated during the decade, but the contents of the page do not instill confidence that it's up-to-date.) That documentation still doesn't show how to configure the extension, instead linking to https://docs.python.org/3/extending/extending.html https://docs.python.org/3/extending/extending.html . That in turn links to https://docs.python.org/3/extending/building.html https://docs.python.org/3/extending/building.html which links to https://setuptools.pypa.io/en/latest/setuptools.html https://setuptools.pypa.io/en/latest/setuptools.html which takes you to https://setuptools.pypa.io/en/latest/userguide/ext_modules.html https://setuptools.pypa.io/en/latest/userguide/ext_modules.h... showing how to configure your setup.py. However, unlike using setup.py directly, where you can hack setup.py's argv, that documentation does not show any way to specify compile-time parameters, "like --system-zstd and --rust-backend as a way to influence the build". Which is what the author wants to do.

ri0t 3y ago

I've decided (a few years ago) that if i'll ever have to upgrade the packaging for my stuff, i'm going to do it with nix (and only that - sorry other os users, you'll have to install nix - actually: not sorry!). I was in (distro) packaging way too deep and decided that my limited time in this world doesn't allow for that kind of crap, anymore. The blatant mess that ensued in the meantime (i.e. last few years) proves me right, imho.

fastasucan 3y ago

I think the documentation (or lack of documentation) can be very illuminating in the focus and though process of those behind a project. That there is concise "This is how you did it before and this is how it is done now" shows that there seemingly haven't been any though process in building that bridge. And in general it seems like the process of packaging projects that is not going to be published on pypi, but are going to be used internally is a dark spot as well. Having the mess of different blog posts and documentation sources saying different stuff is far from ideal though. If you can't sum up the process in a clear concise way you are far from done.

guappa 3y ago
Python removed distutils… basically now there is no official way to install things.
- eesmith 3y ago
  The Python documentation says "pip is the preferred installer program. Starting with Python 3.4, it is included by default with the Python binary installers." - https://docs.python.org/3/installing/index.html#installing-index https://docs.python.org/3/installing/index.html#installing-i...
  - guappa 3y ago
    Except that pip doesn't actually do the install part, you need some other module for that.
    - eesmith 3y ago
      I do not understand your comment. What module are you talking about? What prevents the pip from a normal CPython installation from doing an install? You can even bootstrap pip via the ensurepip module documented at https://docs.python.org/3/library/ensurepip.html https://docs.python.org/3/library/ensurepip.html which notes "This module does not access the internet. All of the components needed to bootstrap pip are included as internal parts of the package."

paradox242 3y ago

It is as if someone has taken the time to describe my typical day. I'm not a package maintainer or anything, just a guy trying to keep multiple CI/CD systems working. I did not get halfway through this before I could start to feel the hairs of my neck start to stand on end. The numerous blind alleys. The promising leads that aren't. The official documentation that contains links to other documents that contradict the first and so I have to try and also piece together some temporal state to make any sense of any of it. Some days it seems I am actually some kind of an information archeologist, piecing together the detritus of overlapping civilizations.

ketozhang 3y ago
In this particular case. OP had a lot of knowledge which served as a crutch in this journey. The less you are aware of how setuptools used to work, the better.

Nimitz14 3y ago

Thank you for sharing this journey! Very helpful.

deleted 3y ago

[deleted]