4 ms·

Wouldn't something along the lines of Guix be suitable? It's already used in HPC for pipelines and deals with all dependencies.

• andyonthewings 8y ago
  It's kind of like suggesting to use system package manager for development package management. It will work, as long as the development material (e.g. a Python library) is available to the package repo, or the package manager is able to use the language-specific repo (e.g. Pypi). But most system package managers are platform specific, which is not really suitable to be adopted officially by cross-platform languages like Python. I mean, Python can't adopt Guix as "the" Python package manager since Guix has no Windows support.
  • chriswarbo 8y ago
    > It's kind of like suggesting to use system package manager for development package management. Yes. I have always found it odd to distinguish between "package managers" and "build systems", let alone the (seemingly more recent?) distinction between "system package manager" and "language-specific package manager". These are all doing essentially the same job: run some commands, based on dependencies, to ensure some desired files are in the desired place. I think the problem stems from Make being so baroque and unmodular/composable: it became widespread (probably via Worse Is Better) and was just about usable for individual applications (where the Makefile and source code are maintained by the same people). Yet it was hopeless for building an OS/distro, since: - The distro's Makefiles would be maintained by different people to the applications', causing fragility - Users want a lot of flexibility from their chosen distro, in terms of what gets installed, what versions, etc. I think of Nix/Guix as "Make done right": their foundation is simple, logical, consistent, modular and composable, which makes them usable for defining everything from a small project's build script, up to whole OS configurations or cloud deployments. > as long as the development material (e.g. a Python library) is available to the package repo It depends what you mean by "the package repo". Nix and Guix can import definitions from anywhere, e.g. from files included with a project, or downloaded from a URL, or cloned from a particular revision of a git repo, etc. There just-so-happen to be some quite large repos out there (e.g. "nixpkgs") which define a whole load of packages, as well as utility functions for packaging or overriding certain languages, etc. > or the package manager is able to use the language-specific repo (e.g. Pypi) Yes, Nix and Guix can fetch data from arbitrary places (like Pypi) and run arbitrary code (like setup.py) as part of the build. The big package repos already contain reusable functions which do this, e.g. 'fetchPypi', 'buildPythonPackage', etc. https://github.com/NixOS/nixpkgs/blob/master/doc/languages-frameworks/python.section.md https://github.com/NixOS/nixpkgs/blob/master/doc/languages-f... > But most system package managers are platform specific, which is not really suitable to be adopted officially by cross-platform languages like Python Nix and Guix are cross-platform. They have been used to define whole OSes, but they also work standalone on any Linux distro and macOS. > Guix has no Windows support I've been following discussions about this, at least on the Nix side. I've not used Windows in decades, but apparently: - Nix works in "Windows Subsystem for Linux" ( https://github.com/NixOS/nixpkgs/issues/30391 https://github.com/NixOS/nixpkgs/issues/30391 ) - Nix works on Cygwin (not sure what the latest info is, but I came across https://ternaris.com/lab/nix-on-windows.html https://ternaris.com/lab/nix-on-windows.html ) - There's been some work on building/running with mingw, but it seems to be painful due to things like Windows' limitations on path names ( https://github.com/NixOS/nix/issues/1320 https://github.com/NixOS/nix/issues/1320 ) - Nix and Guix can be used to cross-compile standalone packages, e.g. developing on Linux or Cygwin but building artefacts which will run on Windows without external DLLs, etc.
• kristjansson 8y ago
  Yes, but the effort to package the sort of software that's causing issues in python for the Guix/Nix/Spack/Easybuild class of systems is herculean[0] and idiosyncratic. The groups maintaining and promulgating 'bad' packages would have to relinquish control over some of the dependencies they're currently venturing in or statically linking, or packagers have to reverse engineer their build systems, do that work themselves, and maintain it going forward. [0]: https://www.youtube.com/watch?v=NSemlYagjIU https://www.youtube.com/watch?v=NSemlYagjIU