4 ms·

Python Packaging, One Year Later: A Look Back at 2023 in Python Packaging

56 points by Kwpolska 3y ago

earthboundkid 3y ago
I regret that I have but one up vote to give. I’m not optimistic for “Python Packaging Two Years Later” in 2025.
eesmith 3y ago
TIL about meson-python, which appears to be the most popular method after setuptools to build C extensions. There are 80 projects out of 107,757 analyzed pyproject.toml files, and SciPy and NumPy migrated to Meson. I use setup.py legacy because of my semi-complicated requirements for C and Cython-based extensions (including a code generation step, and per-file compilation options, and compiler-based compilation and link flags). Knowing that SciPy and NumPy managed to migrate means Meson might be the right option for me. The PyPA docs about "Packaging binary extensions" at https://packaging.python.org/en/latest/guides/packaging-binary-extensions/ https://packaging.python.org/en/latest/guides/packaging-bina... still says "Last Reviewed 2013-12-08".
- aragilar 3y ago
  There's also scikit-build (https://scikit-build.readthedocs.io/en/latest/ https://scikit-build.readthedocs.io/en/latest/), a wrapper around CMake, like how meson-python is a wrapper around meson, and enscons (https://github.com/dholth/enscons https://github.com/dholth/enscons, which is by the original wheel creator), which is a wrapper around SCons (I'm not sure how much maintenance it gets though). It's interesting that the original post (https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/ https://chriswarrick.com/blog/2023/01/15/how-to-improve-pyth...) complained about there being too many options (whilst missing the "scientific python" and rust backends, and including wheel (which is part of setuptools now) and venv (which is primarily a library)), when much of the original push to standardise packaging came from the scientific python community (see e.g. https://pypackaging-native.github.io/ https://pypackaging-native.github.io/), and the need to get away from a distutils/setuptools monoculture (given previous attempts at improving the situation e.g. bento).
  - eesmith 3y ago
    I didn't look at scikit-build because it requires an existing CMake installation and I didn't want my customers to deal with that. (I sell a source code license.) ... oh, wait! TIL "The easiest way to get CMake is to add it to the pyproject.toml file. With pip 10 or later, this will cause the CMake Python package to be downloaded and installed when your project is built." 99 of the backends use scikit-build-core. Only 11 of the backends analyzed used enscons. There are too many options. "There should be one-- and preferably only one --obvious way to do it." is at odds with the "let a hundred flowers bloom" of the current build backend philosophy. The obvious way to do it is to start with the Python documentation. https://docs.python.org/3/extending/extending.html https://docs.python.org/3/extending/extending.html only mentions modifying Modules/Setup.local from a Python source distribution(!). https://docs.python.org/3/extending/building.html#building https://docs.python.org/3/extending/building.html#building says: "Python 3.12 and newer no longer come with distutils. Please refer to the setuptools documentation at https://setuptools.readthedocs.io/en/latest/setuptools.html https://setuptools.readthedocs.io/en/latest/setuptools.html" My code works with setuptools because I started this project 15 years ago, and it's been incremental changes to my setup.py ever since. I find it hard to figure out which option I should use, because so few people use these alternatives and there's very little guidance - either direct advice from PyPA or hits from DDG - about how to handle the issues I've already resolved in setuptools. Pypackaging-native even describes the special things I need, and which I've solved with setuptools: - how do I have per-file compilation options so I can optimize for different architectures? These are also compiler specific. - how do I enable/disable compile-time support for OpenMP? And specify the right link flags? And provide binary wheels for macOS with OpenMP? - how do I add a code generation step? - how do I have the source distribution include the Cython-generated C source so my clients don't need to have Cython installed? (My experience is there's about a 20% chance my customers have an incompatible Cython installed in their environment. Distributing C source means they can build without issues.) The fact that NumPy successfully switched to Meson tells me Meson will likely handle what I want, I just have to figure how to do it. Which I do not look forward to. That I now know CMake is another option makes the paradox of choice rear its ugly head.
    - aragilar 3y ago
qtieb 3y ago
Quality research and a good read, thanks for that! See you next year.
Saphyel 3y ago
blows my mind that in 2023 PIP still exists as it is, having in other languages good examples like: npm, composer, maver, cargo, yarn, etc.. which works like a charm since several years and most of the community are happy with them (No one is happy with PIP let's be honest)
- ogarten 3y ago
  Haven't had much troble with pip to be honest. I hear a lot of complaints from all kinds of people but I have never had an issue. Issues only arise for me when people are deep in the poetry, pdm and whatever tools.
tdhopper 3y ago
> Fast is good, I guess, but what does this say about Python? I don't buy this complaint (?) about Ruff. The broader point is that Python is _so good_ that people are willing to learn Rust to make the Python experience even better.

56 points by Kwpolska 3y ago

I regret that I have but one up vote to give. I’m not optimistic for “Python Packaging Two Years Later” in 2025.

TIL about meson-python, which appears to be the most popular method after setuptools to build C extensions. There are 80 projects out of 107,757 analyzed pyproject.toml files, and SciPy and NumPy migrated to Meson. I use setup.py legacy because of my semi-complicated requirements for C and Cython-based extensions (including a code generation step, and per-file compilation options, and compiler-based compilation and link flags). Knowing that SciPy and NumPy managed to migrate means Meson might be the right option for me. The PyPA docs about "Packaging binary extensions" at https://packaging.python.org/en/latest/guides/packaging-binary-extensions/ https://packaging.python.org/en/latest/guides/packaging-bina... still says "Last Reviewed 2013-12-08".

aragilar 3y ago
There's also scikit-build (https://scikit-build.readthedocs.io/en/latest/ https://scikit-build.readthedocs.io/en/latest/), a wrapper around CMake, like how meson-python is a wrapper around meson, and enscons (https://github.com/dholth/enscons https://github.com/dholth/enscons, which is by the original wheel creator), which is a wrapper around SCons (I'm not sure how much maintenance it gets though). It's interesting that the original post (https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/ https://chriswarrick.com/blog/2023/01/15/how-to-improve-pyth...) complained about there being too many options (whilst missing the "scientific python" and rust backends, and including wheel (which is part of setuptools now) and venv (which is primarily a library)), when much of the original push to standardise packaging came from the scientific python community (see e.g. https://pypackaging-native.github.io/ https://pypackaging-native.github.io/), and the need to get away from a distutils/setuptools monoculture (given previous attempts at improving the situation e.g. bento).
- eesmith 3y ago
  I didn't look at scikit-build because it requires an existing CMake installation and I didn't want my customers to deal with that. (I sell a source code license.) ... oh, wait! TIL "The easiest way to get CMake is to add it to the pyproject.toml file. With pip 10 or later, this will cause the CMake Python package to be downloaded and installed when your project is built." 99 of the backends use scikit-build-core. Only 11 of the backends analyzed used enscons. There are too many options. "There should be one-- and preferably only one --obvious way to do it." is at odds with the "let a hundred flowers bloom" of the current build backend philosophy. The obvious way to do it is to start with the Python documentation. https://docs.python.org/3/extending/extending.html https://docs.python.org/3/extending/extending.html only mentions modifying Modules/Setup.local from a Python source distribution(!). https://docs.python.org/3/extending/building.html#building https://docs.python.org/3/extending/building.html#building says: "Python 3.12 and newer no longer come with distutils. Please refer to the setuptools documentation at https://setuptools.readthedocs.io/en/latest/setuptools.html https://setuptools.readthedocs.io/en/latest/setuptools.html" My code works with setuptools because I started this project 15 years ago, and it's been incremental changes to my setup.py ever since. I find it hard to figure out which option I should use, because so few people use these alternatives and there's very little guidance - either direct advice from PyPA or hits from DDG - about how to handle the issues I've already resolved in setuptools. Pypackaging-native even describes the special things I need, and which I've solved with setuptools: - how do I have per-file compilation options so I can optimize for different architectures? These are also compiler specific. - how do I enable/disable compile-time support for OpenMP? And specify the right link flags? And provide binary wheels for macOS with OpenMP? - how do I add a code generation step? - how do I have the source distribution include the Cython-generated C source so my clients don't need to have Cython installed? (My experience is there's about a 20% chance my customers have an incompatible Cython installed in their environment. Distributing C source means they can build without issues.) The fact that NumPy successfully switched to Meson tells me Meson will likely handle what I want, I just have to figure how to do it. Which I do not look forward to. That I now know CMake is another option makes the paradox of choice rear its ugly head.
  - aragilar 3y ago

qtieb 3y ago

Quality research and a good read, thanks for that! See you next year.

Saphyel 3y ago

blows my mind that in 2023 PIP still exists as it is, having in other languages good examples like: npm, composer, maver, cargo, yarn, etc.. which works like a charm since several years and most of the community are happy with them (No one is happy with PIP let's be honest)

ogarten 3y ago
Haven't had much troble with pip to be honest. I hear a lot of complaints from all kinds of people but I have never had an issue. Issues only arise for me when people are deep in the poetry, pdm and whatever tools.

tdhopper 3y ago

> Fast is good, I guess, but what does this say about Python? I don't buy this complaint (?) about Ruff. The broader point is that Python is _so good_ that people are willing to learn Rust to make the Python experience even better.