4 ms·

There's also scikit-build (https://scikit-build.readthedocs.io/en/latest/ https://scikit-build.readthedocs.io/en/latest/), a wrapper around CMake, like how meso

There's also scikit-build (https://scikit-build.readthedocs.io/en/latest/ https://scikit-build.readthedocs.io/en/latest/), a wrapper around CMake, like how meson-python is a wrapper around meson, and enscons (https://github.com/dholth/enscons https://github.com/dholth/enscons, which is by the original wheel creator), which is a wrapper around SCons (I'm not sure how much maintenance it gets though). It's interesting that the original post (https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/ https://chriswarrick.com/blog/2023/01/15/how-to-improve-pyth...) complained about there being too many options (whilst missing the "scientific python" and rust backends, and including wheel (which is part of setuptools now) and venv (which is primarily a library)), when much of the original push to standardise packaging came from the scientific python community (see e.g. https://pypackaging-native.github.io/ https://pypackaging-native.github.io/), and the need to get away from a distutils/setuptools monoculture (given previous attempts at improving the situation e.g. bento).

eesmith 3y ago
I didn't look at scikit-build because it requires an existing CMake installation and I didn't want my customers to deal with that. (I sell a source code license.) ... oh, wait! TIL "The easiest way to get CMake is to add it to the pyproject.toml file. With pip 10 or later, this will cause the CMake Python package to be downloaded and installed when your project is built." 99 of the backends use scikit-build-core. Only 11 of the backends analyzed used enscons. There are too many options. "There should be one-- and preferably only one --obvious way to do it." is at odds with the "let a hundred flowers bloom" of the current build backend philosophy. The obvious way to do it is to start with the Python documentation. https://docs.python.org/3/extending/extending.html https://docs.python.org/3/extending/extending.html only mentions modifying Modules/Setup.local from a Python source distribution(!). https://docs.python.org/3/extending/building.html#building https://docs.python.org/3/extending/building.html#building says: "Python 3.12 and newer no longer come with distutils. Please refer to the setuptools documentation at https://setuptools.readthedocs.io/en/latest/setuptools.html https://setuptools.readthedocs.io/en/latest/setuptools.html" My code works with setuptools because I started this project 15 years ago, and it's been incremental changes to my setup.py ever since. I find it hard to figure out which option I should use, because so few people use these alternatives and there's very little guidance - either direct advice from PyPA or hits from DDG - about how to handle the issues I've already resolved in setuptools. Pypackaging-native even describes the special things I need, and which I've solved with setuptools: - how do I have per-file compilation options so I can optimize for different architectures? These are also compiler specific. - how do I enable/disable compile-time support for OpenMP? And specify the right link flags? And provide binary wheels for macOS with OpenMP? - how do I add a code generation step? - how do I have the source distribution include the Cython-generated C source so my clients don't need to have Cython installed? (My experience is there's about a 20% chance my customers have an incompatible Cython installed in their environment. Distributing C source means they can build without issues.) The fact that NumPy successfully switched to Meson tells me Meson will likely handle what I want, I just have to figure how to do it. Which I do not look forward to. That I now know CMake is another option makes the paradox of choice rear its ugly head.
- aragilar 3y ago
  Just FYI, but you do not need to switch away from setuptools if you do not want to. `setup.py` isn't going away, rather it is no longer the interface that "build frontends" (i.e. pip, but also pypa/build) use to build the project (there is a well defined interface made up of a number of task specific functions).
  - eesmith 3y ago
    Thanks. I do understand that, and even if PyPA and the setuptools maintainers wanted to get rid of it, there's such a large installed base that any transition would take at least a decade. Plus, as I learned yesterday, the cmake-based backend also depends on setuptools. Thing is, my setup.py is full of hacks and is a nuisance to use, so I would prefer a better solution. What kind of hacks? In the "python setup.py install" days I processed sys.argv to handle extra compile-time parameters, like "don't compile for OpenMP". When direct use of setup.py became unsupported, I switched to environment variables to pass in the configuration. While it works, I don't know if it's a long-term viable solution. I read about the push towards isolated environments and reproducible builds, which is not something you can get if your build depends on ephemeral environment variables which are not captured in the pyproject.toml or dependency hashes. (Remember, I distribute a source package, not a precompiled wheel.) The longest component in my distribution takes a minute to compile. Setuptools does not handle dependency tracking well, so even if I touch C code which only takes a second to compile, it still rebuilds the entire component. I have a hacked-up Makefile for when I really need to work on that part of the code and want faster turn-arounds, but I would prefer to have a unified build system that really understands build dependencies. I subclass build_ext to handle the per-file compilation args, per-compiler link args, and to add an '$ORIGIN' rpath for non-Darwin builds as I generate shared libraries which depend on other shared libraries. To make this work I had to monkeypatch setuptools implementation internals. I know how fragile it is, and would like something better, which I could depend on being around for another 15 years. But when I dip into the Packaging world, I feel like my needs are in a forgotten corner, with the main impetuous being the needs of enterprise systems with engineering staff - a far cry from the packaging world I started with 20 years ago where building C extensions was one of the primary concerns.

eesmith 3y ago

I didn't look at scikit-build because it requires an existing CMake installation and I didn't want my customers to deal with that. (I sell a source code license.) ... oh, wait! TIL "The easiest way to get CMake is to add it to the pyproject.toml file. With pip 10 or later, this will cause the CMake Python package to be downloaded and installed when your project is built." 99 of the backends use scikit-build-core. Only 11 of the backends analyzed used enscons. There are too many options. "There should be one-- and preferably only one --obvious way to do it." is at odds with the "let a hundred flowers bloom" of the current build backend philosophy. The obvious way to do it is to start with the Python documentation. https://docs.python.org/3/extending/extending.html https://docs.python.org/3/extending/extending.html only mentions modifying Modules/Setup.local from a Python source distribution(!). https://docs.python.org/3/extending/building.html#building https://docs.python.org/3/extending/building.html#building says: "Python 3.12 and newer no longer come with distutils. Please refer to the setuptools documentation at https://setuptools.readthedocs.io/en/latest/setuptools.html https://setuptools.readthedocs.io/en/latest/setuptools.html" My code works with setuptools because I started this project 15 years ago, and it's been incremental changes to my setup.py ever since. I find it hard to figure out which option I should use, because so few people use these alternatives and there's very little guidance - either direct advice from PyPA or hits from DDG - about how to handle the issues I've already resolved in setuptools. Pypackaging-native even describes the special things I need, and which I've solved with setuptools: - how do I have per-file compilation options so I can optimize for different architectures? These are also compiler specific. - how do I enable/disable compile-time support for OpenMP? And specify the right link flags? And provide binary wheels for macOS with OpenMP? - how do I add a code generation step? - how do I have the source distribution include the Cython-generated C source so my clients don't need to have Cython installed? (My experience is there's about a 20% chance my customers have an incompatible Cython installed in their environment. Distributing C source means they can build without issues.) The fact that NumPy successfully switched to Meson tells me Meson will likely handle what I want, I just have to figure how to do it. Which I do not look forward to. That I now know CMake is another option makes the paradox of choice rear its ugly head.

aragilar 3y ago
Just FYI, but you do not need to switch away from setuptools if you do not want to. `setup.py` isn't going away, rather it is no longer the interface that "build frontends" (i.e. pip, but also pypa/build) use to build the project (there is a well defined interface made up of a number of task specific functions).
- eesmith 3y ago
  Thanks. I do understand that, and even if PyPA and the setuptools maintainers wanted to get rid of it, there's such a large installed base that any transition would take at least a decade. Plus, as I learned yesterday, the cmake-based backend also depends on setuptools. Thing is, my setup.py is full of hacks and is a nuisance to use, so I would prefer a better solution. What kind of hacks? In the "python setup.py install" days I processed sys.argv to handle extra compile-time parameters, like "don't compile for OpenMP". When direct use of setup.py became unsupported, I switched to environment variables to pass in the configuration. While it works, I don't know if it's a long-term viable solution. I read about the push towards isolated environments and reproducible builds, which is not something you can get if your build depends on ephemeral environment variables which are not captured in the pyproject.toml or dependency hashes. (Remember, I distribute a source package, not a precompiled wheel.) The longest component in my distribution takes a minute to compile. Setuptools does not handle dependency tracking well, so even if I touch C code which only takes a second to compile, it still rebuilds the entire component. I have a hacked-up Makefile for when I really need to work on that part of the code and want faster turn-arounds, but I would prefer to have a unified build system that really understands build dependencies. I subclass build_ext to handle the per-file compilation args, per-compiler link args, and to add an '$ORIGIN' rpath for non-Darwin builds as I generate shared libraries which depend on other shared libraries. To make this work I had to monkeypatch setuptools implementation internals. I know how fragile it is, and would like something better, which I could depend on being around for another 15 years. But when I dip into the Packaging world, I feel like my needs are in a forgotten corner, with the main impetuous being the needs of enterprise systems with engineering staff - a far cry from the packaging world I started with 20 years ago where building C extensions was one of the primary concerns.