4 ms·

The business about mapping from PyPI to system dependencies is an important one, and (having not read the entire thread) I do hope that gets some attention— it'

The business about mapping from PyPI to system dependencies is an important one, and (having not read the entire thread) I do hope that gets some attention— it's particularly curious that it's been this long and it hasn't, given Python's often-role as a glue language. Another example of an ecosystem maintaining mappings out to system packages is ROS and rosdep: https://github.com/ros/rosdistro/blob/master/rosdep/base.yaml https://github.com/ros/rosdistro/blob/master/rosdep/base.yam... Now it's interesting because ROS is primarily concerned with supplying a sane build-from-source story, so much of what's in the rosdep "database" is the xxxx-dev packages, but in the case of wheels, it would be more about binary dependencies, and those are auto-discoverable with ldd, shlibdeps, and the like. In Debian (and I assume other distros), the binary so packages are literally the library soname + abi versions, so if you have ldd output, you have the list of exactly what to install. Maybe one interesting "social" piece to including this kind of functionality would be what the behaviour of pip would become. Like, would it a) go back to needing to be invoked as root and call through to the system package manager, b) emit the command needed for the user to install the required system packages, or c) download the system packages itself, extract the libraries, and insert them into its non-root-owned virtualenv or whatever other environment?

deleted 5y ago
[deleted]
ihnorton 5y ago
Julia did that for binary dependencies for a few years, with adapters for several linux distros, homebrew, and for cross-compiled RPMs for Windows. It worked, to a degree -- less well on Windows -- but the combinatorial complexity led to many hiccups and significant maintenance effort. Each Julia package had to account for the peculiarities of each dependency across a range of dependency versions and packaging practices (linkage policies, bundling policies, naming variations, distro versions) -- and this is easier in Julia than in (C)Python because shared libraries are accessed via locally-JIT'd FFI, so there is no need to eg compile extensions for 4 different CPython ABIs (Julia also has syntactic macros which can be helpful here). To provide a better experience for both package authors and users, as well as reducing the maintenance burden, the community has developed and migrated to a unified system called BinaryBuilder (https://binarybuilder.org https://binarybuilder.org) over the past 2-3 years. BinaryBuilder allows targeting all supported platforms with a single build script and also "audits" build products for common compatibility and linkage snafus (similar to some of the conda-build tooling and auditwheel). One downside of "cross-compilation everywhere" is that it's not always well-supported by upstream libraries, so the initial effort to build a library may be higher if patches need to be developed, but that effort is arguably higher-leverage than a per-distro packaging approach (eg https://twitter.com/Blosc2/status/1395425736597585920 https://twitter.com/Blosc2/status/1395425736597585920). All that to say: "make binaries a distro packager's problem" sounds like a simplifying step, but there are some big caveats. It has been tried before, both in other languages and in Python: the fact that conda and manylinux don't use system packages was not borne out of inexperience. One additional issue is that distro packages are only available with sysadmin consent in shared unix-like environments, which can be very limiting for end-users.

deleted 5y ago

[deleted]

ihnorton 5y ago

Julia did that for binary dependencies for a few years, with adapters for several linux distros, homebrew, and for cross-compiled RPMs for Windows. It worked, to a degree -- less well on Windows -- but the combinatorial complexity led to many hiccups and significant maintenance effort. Each Julia package had to account for the peculiarities of each dependency across a range of dependency versions and packaging practices (linkage policies, bundling policies, naming variations, distro versions) -- and this is easier in Julia than in (C)Python because shared libraries are accessed via locally-JIT'd FFI, so there is no need to eg compile extensions for 4 different CPython ABIs (Julia also has syntactic macros which can be helpful here). To provide a better experience for both package authors and users, as well as reducing the maintenance burden, the community has developed and migrated to a unified system called BinaryBuilder (https://binarybuilder.org https://binarybuilder.org) over the past 2-3 years. BinaryBuilder allows targeting all supported platforms with a single build script and also "audits" build products for common compatibility and linkage snafus (similar to some of the conda-build tooling and auditwheel). One downside of "cross-compilation everywhere" is that it's not always well-supported by upstream libraries, so the initial effort to build a library may be higher if patches need to be developed, but that effort is arguably higher-leverage than a per-distro packaging approach (eg https://twitter.com/Blosc2/status/1395425736597585920 https://twitter.com/Blosc2/status/1395425736597585920). All that to say: "make binaries a distro packager's problem" sounds like a simplifying step, but there are some big caveats. It has been tried before, both in other languages and in Python: the fact that conda and manylinux don't use system packages was not borne out of inexperience. One additional issue is that distro packages are only available with sysadmin consent in shared unix-like environments, which can be very limiting for end-users.