4 ms·

The caching trick for libsolv here could be a game changer for performance in a lot of Linux distro package managers, where libsolv's completeness comes at a su

by pxc 4y ago

The caching trick for libsolv here could be a game changer for performance in a lot of Linux distro package managers, where libsolv's completeness comes at a substantial performance cost. I've never looked too deeply into how conda works because I'm not that into Python, but I guess I should. It seems there are some valuable general lessons here— I wish that more package managers, and in particular, Nix, maintained more version combinations in their repos. Searching through Nixpkgs commits to find compatible versions of things because you have to rely on the implicit versioning of the monorepo sucks. Adding dependency resolution to Nix as it stands would normally add a big performance cost. Something like this would help with that problem a lot, and Nix already usually runs with a persistent daemon. Maybe a similar technique could fit in well here.

droelf 4y ago
Yes, we definitely believe and rely a lot on proper "SAT" solving in the conda ecosystem (old versions are also never deleted, for reproducibility reasons, which makes this problem actually quite interesting and large). We are also quite curious to learn more about the Nix approach. Building packages is still a bit too painful / annoying in the conda ecosystem, which is something we really want to fix, and I think we can learn a lesson or two in the Nix ecosystem. And by the way, the conda ecosystem is not at all Python specific. It's a common misconception, but they are all general binary packages and can be used for Ruby, R, Rust, C++, Zig, ... whatever – just like Nix. Rattler, mamba and conda all handle them the same nice way.
- pxc 4y ago
  I knew that conda's connection to Python was mostly historical rather than technological, since it handled native deps and so it had to be able to work with C as well. I didn't realize so much work had been done packaging all of those other language ecosystems, though!
chriswarbo 4y ago
I'm not sure how 'dependency resolution' would work in a system like Nix; since Nix has no notion of "package", "dependency", "version", etc. > Searching through Nixpkgs commits to find compatible versions of things because you have to rely on the implicit versioning of the monorepo sucks. I get what you're saying, but I gave up on thinking of Nix/Nixpkgs in terms of "package management". It's much closer to a build system, like GNU Make; rather than, say, APT/Yum. When it comes to versions of things in Nixpkgs: most things are defined as a function of other things; i.e. the arguments to that function are (roughly) its "dependencies". If we want some different version, we can call those functions with different arguments (or use the various `override` helpers included in their return values). Once we've overridden the things we care about, we can usually take the fixed-point/closure of that package set (whether it's the whole of Nixpkgs, or just e.g. python3Packages, or whatever); that way, our changes get propagated through all of the function calls to ensure everything's referencing our versions. Admittedly, this can be quite tedious!
- pxc 4y ago
  I'm aware of how Nixpkgs currently works. :) It's a huge limitation for many reasonable ways that people want to use Nix. It's a shortcut that Nix has gotten away with, not a superior approach. Look at the machinery mach-nix implements with a cached database of PyPi and how it has to scan across different Nixpkgs revisions just to automatically choose compatible versions of deps when it does its code generation for a clear illustration. There are lots of source-based package management systems which rely on a mix of implicit versioning based on what happens to be exist under an unversioned name in the monorepo and explicit versioning, Gentoo Portage for example. There's no reason that we couldn't have either a. a complement to Nixpkgs which serves as a record of which build recipes have successfully built what versions, and what versions their deps were at b. richer metadata in Nixpkgs about versions that can be used to support multiple versions of same-named packages in the repository, and a smarter callPackage that uses those version constraints to choose, when necessary, which versions of dependencies to put into scope when calling a package (yes, I know there is ongoing work to remove callPackage, or at least explicit invocations of it) and both would be really useful for tools that autogenerate Nix code for packaging purposes. Hacking things together with nix-index just to get a little automagical code generation where users can specify package versions rather than Nixpkgs commits is a kludge. It's cumbersome and rightly turns some developers off from using Nix. Nixpkgs should be enriched or complemented in order to obviate that approach, and the Nix ecosystem will need tooling that has an actual dependency resolver to work with it— even if the only maintained combination of versions for distribution is pretty much singular, and implicitly encoded in Nixpkgs just as it is today.
  - RaitoBezarius 4y ago
    Flox has some solution for this. Other than that, this is being worked on :).
  - tgamblin 4y ago
    Have you looked at Spack (https://github.com/spack/spack https://github.com/spack/spack)? Disclaimer: I lead the project. Spack essentially is nix with dependency resolution. spack packages can have conditional (nearly) anything: versions, options, compiler choices, dependencies (including virtual deps). The resolver can make choices about all of those. See here for more details: https://arxiv.org/abs/2210.08404 https://arxiv.org/abs/2210.08404
dralley 4y ago
.solv cache files are a libsolv thing that has existed for a long time, the only "trick" here is using them. The package managers you refer to already do as far as I'm aware, the slowness is for other reasons.
- pxc 4y ago
  Doesn't the persistent daemon/server that keeps them pre-loaded in RAM make a difference here over just repeatedly loading something that's cached on disk?

by pxc 4y ago

droelf 4y ago

Yes, we definitely believe and rely a lot on proper "SAT" solving in the conda ecosystem (old versions are also never deleted, for reproducibility reasons, which makes this problem actually quite interesting and large). We are also quite curious to learn more about the Nix approach. Building packages is still a bit too painful / annoying in the conda ecosystem, which is something we really want to fix, and I think we can learn a lesson or two in the Nix ecosystem. And by the way, the conda ecosystem is not at all Python specific. It's a common misconception, but they are all general binary packages and can be used for Ruby, R, Rust, C++, Zig, ... whatever – just like Nix. Rattler, mamba and conda all handle them the same nice way.

pxc 4y ago
I knew that conda's connection to Python was mostly historical rather than technological, since it handled native deps and so it had to be able to work with C as well. I didn't realize so much work had been done packaging all of those other language ecosystems, though!

chriswarbo 4y ago

I'm not sure how 'dependency resolution' would work in a system like Nix; since Nix has no notion of "package", "dependency", "version", etc. > Searching through Nixpkgs commits to find compatible versions of things because you have to rely on the implicit versioning of the monorepo sucks. I get what you're saying, but I gave up on thinking of Nix/Nixpkgs in terms of "package management". It's much closer to a build system, like GNU Make; rather than, say, APT/Yum. When it comes to versions of things in Nixpkgs: most things are defined as a function of other things; i.e. the arguments to that function are (roughly) its "dependencies". If we want some different version, we can call those functions with different arguments (or use the various `override` helpers included in their return values). Once we've overridden the things we care about, we can usually take the fixed-point/closure of that package set (whether it's the whole of Nixpkgs, or just e.g. python3Packages, or whatever); that way, our changes get propagated through all of the function calls to ensure everything's referencing our versions. Admittedly, this can be quite tedious!

pxc 4y ago
I'm aware of how Nixpkgs currently works. :) It's a huge limitation for many reasonable ways that people want to use Nix. It's a shortcut that Nix has gotten away with, not a superior approach. Look at the machinery mach-nix implements with a cached database of PyPi and how it has to scan across different Nixpkgs revisions just to automatically choose compatible versions of deps when it does its code generation for a clear illustration. There are lots of source-based package management systems which rely on a mix of implicit versioning based on what happens to be exist under an unversioned name in the monorepo and explicit versioning, Gentoo Portage for example. There's no reason that we couldn't have either a. a complement to Nixpkgs which serves as a record of which build recipes have successfully built what versions, and what versions their deps were at b. richer metadata in Nixpkgs about versions that can be used to support multiple versions of same-named packages in the repository, and a smarter callPackage that uses those version constraints to choose, when necessary, which versions of dependencies to put into scope when calling a package (yes, I know there is ongoing work to remove callPackage, or at least explicit invocations of it) and both would be really useful for tools that autogenerate Nix code for packaging purposes. Hacking things together with nix-index just to get a little automagical code generation where users can specify package versions rather than Nixpkgs commits is a kludge. It's cumbersome and rightly turns some developers off from using Nix. Nixpkgs should be enriched or complemented in order to obviate that approach, and the Nix ecosystem will need tooling that has an actual dependency resolver to work with it— even if the only maintained combination of versions for distribution is pretty much singular, and implicitly encoded in Nixpkgs just as it is today.
- RaitoBezarius 4y ago
  Flox has some solution for this. Other than that, this is being worked on :).
- tgamblin 4y ago
  Have you looked at Spack (https://github.com/spack/spack https://github.com/spack/spack)? Disclaimer: I lead the project. Spack essentially is nix with dependency resolution. spack packages can have conditional (nearly) anything: versions, options, compiler choices, dependencies (including virtual deps). The resolver can make choices about all of those. See here for more details: https://arxiv.org/abs/2210.08404 https://arxiv.org/abs/2210.08404

dralley 4y ago

.solv cache files are a libsolv thing that has existed for a long time, the only "trick" here is using them. The package managers you refer to already do as far as I'm aware, the slowness is for other reasons.

pxc 4y ago
Doesn't the persistent daemon/server that keeps them pre-loaded in RAM make a difference here over just repeatedly loading something that's cached on disk?