3 ms·

As I see it, there is a hierarchy of packaging needs, the base levels have been solved over and over with new, better, shiny all you need tools -- while the mos

As I see it, there is a hierarchy of packaging needs, the base levels have been solved over and over with new, better, shiny all you need tools -- while the most tricky, complicated part has been solved over and over separately with each project. * Pure python -- Easy, use one of the declarative ones. * Python + Standalone C -- not too bad, use the build tool. * Python + external (potentially distro supplied) C libraries -- Using setup.py, customized, and different for each project. That last one is where Pillow, the ML space, scipy, and others live, and it's painful. Pillow has a 1000 line setup.py file to find all the optional (and 2 required) dependencies and headers on it's platforms. We've also got code to build the dependencies if necessary for packaging. To port this to some standard, we'd effectively need rpm or dpkg style build infra from PyPa, to work on all the supported platforms.

MrJohz 3y ago
I think that's also a valid view of the problem (i.e. the further away from pure Python you get, the more complex and unclear things are). But I also think that's just the view from the "library developers" perspective — if you aren't publishing a library, but using Python for some other purpose, you are going to run into issues even at points that, from your perspective, are already fairly solved. For example, for application developers, even if they just stick to pure Python dependencies, there's still no standard lockfile format that standard Python tools can just emit and ingest. At best, you've got `pip freeze`, but you'll need to use custom tooling to update and maintain that, or switch to pip-compile or another, more full-featured package manager. To me, lockfiles really are table stakes here, but they're not at all easy to get working in Python.
eesmith 3y ago
> Python + Standalone C -- not too bad, use the build tool Doesn't that generally end up using setup.py as well? From my understanding, build ends up calling the build-backend, which defaults to setuptools.build_meta:__legacy__, which is setup.py. I know there are other backends, but they seem very specialized to a certain project's needs. I think there's a cmake backend too, but I don't like requiring my customers to install cmake first, and that dependency can't be expressed in pyproject.toml. I had hoped that redo (https://redo.readthedocs.io/en/latest/ https://redo.readthedocs.io/en/latest/) would become popular, as a small, simple, pure-Python Makefile replacement, and that there would be a back-end using it, but neither happened. My specific needs for a backend is to support and configure a code-generation step when building my C extension. The full code generation is >10MB, which handles all 3x24 or so different specialized implementations of the core algorithm. This takes a while to compile, so during development I use a slower, general-purpose implementation.

MrJohz 3y ago

I think that's also a valid view of the problem (i.e. the further away from pure Python you get, the more complex and unclear things are). But I also think that's just the view from the "library developers" perspective — if you aren't publishing a library, but using Python for some other purpose, you are going to run into issues even at points that, from your perspective, are already fairly solved. For example, for application developers, even if they just stick to pure Python dependencies, there's still no standard lockfile format that standard Python tools can just emit and ingest. At best, you've got `pip freeze`, but you'll need to use custom tooling to update and maintain that, or switch to pip-compile or another, more full-featured package manager. To me, lockfiles really are table stakes here, but they're not at all easy to get working in Python.

eesmith 3y ago

> Python + Standalone C -- not too bad, use the build tool Doesn't that generally end up using setup.py as well? From my understanding, build ends up calling the build-backend, which defaults to setuptools.build_meta:__legacy__, which is setup.py. I know there are other backends, but they seem very specialized to a certain project's needs. I think there's a cmake backend too, but I don't like requiring my customers to install cmake first, and that dependency can't be expressed in pyproject.toml. I had hoped that redo (https://redo.readthedocs.io/en/latest/ https://redo.readthedocs.io/en/latest/) would become popular, as a small, simple, pure-Python Makefile replacement, and that there would be a back-end using it, but neither happened. My specific needs for a backend is to support and configure a code-generation step when building my C extension. The full code generation is >10MB, which handles all 3x24 or so different specialized implementations of the core algorithm. This takes a while to compile, so during development I use a slower, general-purpose implementation.